Q4 2021
Retiring to Build Open Source Full-Time
Here I am sitting on a super power (I think), and I already want to build version 2.0 without any manifested success. The project was on life support. I was stalled, waiting to move back to the midwest, struggling with the gap between what I want to manifest and what business needs from me.
Then I retired.
As I approach 40, I decided to embrace the impractical and silly side of life -- like building a foundation around a programming language for board games. My career was built on pragmatic decisions, but being practical was devastating for my mental health. When my mental health suffers, my wife suffers too. It's the creative artistic soul wanting to manifest. Instead of worrying about all the mistakes I've made, I need to commit to the current path. Embrace the mistakes. Prioritize which warts to clean up over time. Build community that thinks "hey, this is kind of neat."
With commitment came clarity. I laid out the five secrets -- the things I know that could lead to a niche empire.
Secret one: DIY database within a document. The first experience of writing code is pure joy. Then you hit persistence, and joy dies. Marshalling state to disk, handling partial failures, dealing with the impedance mismatch between your objects and your database. Adama's model is simple: you just write code, change state, and the magic of persistence is handled for you. Think of your computer's suspend and resume -- the VM wakes up by loading a snapshot, accepts messages, persists changes as data differentials, and sleeps when idle. You focus on the experience directly, as if you are a beginner again.
Secret two: automatic undo. Since Adama monitors state changes to produce data differentials, it records the inverse differential too. Rewind the document state to any point. The biggest complaint my social group has playing online board games is the lack of undo. As I get older, the game is less about cut-throat competition and more about socially connecting. Mistakes happen. That's ok.
Secret three: the dungeon master pattern. Adama documents have a labeled state machine that executes code on transitions or after delays. For board games, this state machine behaves like a dungeon master -- it can ask players "which card would you like to play?" and wait for the answer. The async/await mechanism simplifies server-side board game logic dramatically, but the implementation is tricky. The transactional boundary spans multiple messages. The state machine must survive server restarts. A player might go away forever. These problems are fixable: force messages into a queue within the data differential, throw an exception when data is unavailable, emit a differential only on successful completion.
Secret four: reactivity paired with privacy. Board games need goldilocks privacy -- sharing too much or not enough is equally bad. Defining state with privacy rules and letting the language automate the copying, transformation, and serialization per user is a productivity boon. Reactivity combines pulling and pushing into a single idiom. The networking cost is minimized by sending only updates. The client cost is minimized by ingesting only changes. The server cost is minimized by recomputing privacy only on data changes.
Secret five: massive scale via read replicas with client-side prediction. Log data changes first to achieve millions of observers. Writes against a read replica become the beginnings of client-side prediction. If the state is mostly public, a specialized client-side predictor could be generated from the language to estimate how local messages manifest local data changes. This area is fascinating to explore, but I must prioritize away from it.
With the secrets articulated, the real work began -- and the real suffering. I spent November wandering through architecture decisions. How should persistence work? I could build a raft logger, but I need to focus. Tight-coupling to DynamoDB or Aurora feels wrong but would let me move fast. The answer: keep the interface open. Build the core as an agnostic library, then build a sample integration using AWS.
The WebSocket situation was worse. I wrote an entire essay on everything that goes wrong when you use WebSockets for real products. Your socket will last minutes, hours, days -- stretch your imagination beyond millisecond request handling. Connect storms emerge when connections fail simultaneously. Your code must work damn near perfectly because a broken stream looks exactly like an inactive stream. You'll deal with messaging stacks that mostly suck. Pub/sub sounds great until back-pressure hits and your queues grow without bound. In BladeRunner (our SOSP21 paper), 90%+ of messages were thrown away -- the value proposition was accepting as many as possible and discarding intelligently before the last mile.
The solution I arrived at: Adama embraces request-response resilience. You could just poll the state of a document -- incredible resilience. The WebSocket is a downstream optimization where updates flow as differentials. On the upstream, clients send single messages to single documents with traditional request-response semantics. No fan-out, no pub/sub, just a one-to-one relationship that creates an easy-to-measure sense of reliability.
Null was also on trial. I decided Adama takes the position that null was a big but necessary mistake. No reference types, no null by fiat. Instead, the Maybe monad. This creates a burden on the happy path -- developers must contend with the possibility of something going wrong. In my experience, optimistic code creates more technical debt and unreliability. Division becomes especially interesting: x / y returns maybe<double> where absence represents undefined. Five divided by two is 2.5, not 2, because Adama aims at non-hard-core developers and the principle of least surprise.
The mission for 2022 crystallized: launch a minimal SaaS offering to enable people to cut their teeth, launch small web properties, and unblock my board game ambitions. The license changed to MIT. The key technologies will always be open source such that people can take the core and plug into their ecosystem.
I couldn't build a six-nine service alone and I didn't need to. For the board games I intend to ship, ten minutes of downtime within a week is fine. Honesty is probably the number one job for any infrastructure service. Reality is hard to contend with, and I'd rather set reasonable expectations than pretend I'm something I'm not.
The important thing, in life, is that at the end I can say I tried.