Q2 2022
Progress is going slow as I'm on the pathless path wandering.
A completely valid criticism of this project could be: Adama is an over-engineered bespoke piece of infrastructure in search of a problem. Fair. But after a conversation with someone who pointed out what I was actually sitting on, I started to see the super power differently. I solved the problem of turning infrastructure into user-generated content. If the browser is a mutable surface to drive user experiences, then Adama is a mutable backend to connect people. Every side project I've ever done feeds into this vertical foundation.
The quarter started with me reading "How Tech Loses Out" and recognizing that I'm an anti-salesman and negative marketer. Technical people recommend that others don't use the product because they know how it was built. I know exactly how the sausage is made, and I'm too embarrassed about the current state to properly evangelize. But I'm also not bound by the prison of expectations -- the service is in early access, and nobody is paying me yet.
Since Adama was built around reactivity, the language provides a hybrid engine that could sit under the love child of Access or Excel. The dimensions of growth are plentiful, and the number of domains possible with low-latency compute-centric storage feel infinite. But there are hard walls. Adama documents are isolated -- no disk access, no network access. This is fantastic for privacy (it's a closed box, almost like a database) but disappointing for the dream of a complete backend-as-a-service.
Meanwhile, I spent time thinking about what a decade in distributed systems actually taught me. The insight: every distributed system that contends with state either has a single point of failure, has a broken protocol, or sits on top of paxos/raft. The big services (SQS, S3, Dynamo) are essentially data structures using replicated state machines and transaction logs. Having separate services for various data structures is exceptionally wasteful. You build your bespoke storage engine, and then you need a high-throughput protocol, encryption, host selection, shard management, health checking, metrics, capacity management, rebalancing, a public protocol, authentication, and more. The engineering is well defined but expensive. For most businesses, the opportunity cost is radically high.
Why can't we have a virtual machine target a replicated state machine? Ship your novel data structure, describe how to mutate or query it, and get everything else for free. If I were to go back to academics, this would be my research area. Adama is an existence proof -- a differential state machine that can feed a raft cluster. Differentiability gives you efficient replication and upgradability. The proof: to produce a difference, you read current state, execute logic, emit a difference. Since logic is stateless, it can change between deployments.
On the practical side, I deployed Caravan (the new storage engine) to production with a single point of failure. Obviously problematic, but common enough for environments running a database. I designed a system where documents are backed up to S3 periodically, and when a document is closed it gets archived, unbound from the directory, and deleted locally. This means documents can follow users between regions -- fly from LA to New York and your document zips to the cloud and rehydrates on the east coast. Storage mirrors memory capacity, so scaling dimensions drop from (cpu, memory, storage) to just (cpu, memory).
Capacity management became the hard problem. Specifically, I needed to figure out host selection and detecting the need for new capacity. I sketched out an interface where an Adama host detects heat, picks a space to load shed, and brings more capacity online. It's a knapsack problem (NP-complete), so serious science is needed. But for now: if a host hits 70% memory or 80% CPU, wake up every 10 minutes and make a decision. Better than nothing.
The biggest development of the quarter was RxHTML. I've thought much about UIs from a low level, and for whatever mysterious reason I love building my own frameworks. In retrospect, this is an immature approach to a common problem. Web development sucks tremendously. The JavaScript ecosystem is a dumpster fire of npm dependencies, supply chain attacks, broken shit, and abandonware. Generating complaints is like shooting fish in a barrel with a bazooka.
So I asked: where did we go wrong? If bandwidth had been what it is now during the birth of the internet, all web developers would be Citrix developers making apps delivered via a frame buffer stream. The gap between HTML and a frame buffer stream is the opportunity. RxHTML takes HTML and reactively binds it to a data tree. Data changes manifest as DOM updates at minimal cost. Forms send messages to Adama, establishing the read-write loop. Web Components are the escape hatch for anything that needs deep code. The entire front-end game -- turning data into HTML and capturing interactions back into data -- gets collapsed into a thin reactive layer.
I'm building the IDE entirely with RxHTML. The goal: a website that can create website makers that can then make websites. Three levels of meta. I'm excited, and I need to stop wandering.