Let's talk about this "Adama Platform" thing. This web-page is a deep dive into Adama as it relates to building web apps, games, or anything you can imagine. So, let's go!
As a guiding principle, we have to keep our eye on the prize when making software. The goal of software is to help people get things done, and this is done by connecting people to data with logic to operate on their data.
It's a simple model, but it is painfully easy to get distracted by complexity. For you see, each endeavor requires an essential amount of complexity, and the limits and constraints of the day create an abundant source of accidental complexity that only slows us down or makes life just harder.
Before we begin our deep dive, we set the stage for how to build a web product using traditional web engineering as this allows us to compare and contrast.
Generally speaking, there are two macro components to construct: the backend and the frontend.
While Adama focuses on the back-end, the front-end is vital to consider even as a separate discipline because a back-end without a front-end is not a web product.
A great user experience is going to depend on the back-end and front-end working well together.
The good news is front-end web engineers live within a sandbox box defined by the browser. That has the effect that most front-end engineering work is focused on making a good product or user experience rather than system level challenges.
The bad news is that the browser is a bit of a mess, and this has caused the industry a great deal of headache with frameworkitis. There are many frameworks out there to help tackle this beast especially as it is a moving target.
For now, the important thing to call out is the finitude of the interaction between the front-end and back-end. The limited interaction models create an intrinsically beautiful game to build products which we can greatly simplify.
Adama starts at the end-game with fully real-time web experiences that are deeply interactive.
We will come back to the front-end after we talk about the back-end. The back-end has system level challenges because it often isn't done in a sandbox.
The concerns are very different. For example, a back-end cares about durability, reliability, availability, privacy, security, latency, and so many more dimensions.
These concerns manifest in a variety of different services for developers to buy, adapt, configure, and secure. So, let's look at how many web applications end up.
Here are the various components of a web application: load balancer, gateway, your service, various caches, a messaging stack, maybe workflow, and multiple data-stores.
It starts with the logic that defines the product and related business mechanics. This is the stuff you want to build.
Since persisting data is exceptionally hard, a web app will store and fetch data from a database or a variety of services. The database has the concern of ensuring that data actually ends up on disk that has regular backups or some kind of replication.
This is the most important concern for infrastructure: keeping user data alive.
As you scale the traffic, you'll replicate the number of machines that process your product logic. Either your adding capacity or have a heterogenous number of services that need to look like one service. Regardless, a load balancer is brought into the picture to route user traffic to appropriate machines.
As you scale up and out, the database is going to experience pressure which will result in a poor user experience. This pressure is taken off by introducing caches at various levels and a legion of policies.
At some point, you'll want to offload some work, so you'll use a queue or messaging stack. This allows product logic to run outside the main request and provide a snappier experience for long running tasks.
Furthermore, a queue is sometimes the only way to make sure your software is actually reliable and correct in the event of power loss.
The messaging stack can also be used to provide real-time updates to users. This requires a gateway which users maintain a connection to listen for updates using a real-time capable protocol.
Finally, some products will require the use a workflow engine to connect multiple people together. For example, workflow systems connect users to restaurants and restaurants to drivers or users to customer support agents.
Finishing up this, there are a lot of pieces that make up a website and this is a small fragment of possible systems out there available to buy.
The challenge emerges because the thing you want to build is going to require you to buy a bunch of systems which then you have to configure and adapt too. This has many implications, but the notable observation is that building and operating a web property is non-trivial.
This challenge sits on the essential task of actually building your product.
Buying alone is a bit of a combinatorial nightmare in and of itself. In many ways, this is the status quo. The above journey introduced a variety of components from concerns that evolved over growth.
Cynically, we could say that this mess is good for engineers' job security as there is plenty of toil. However, for us, this is a recipe for burn-out. We want to help people and not configure systems. Adama is our answer to building sustainabily with a vertical stack.
Adama has inherited perspectives learned from operating at hyper scale. Let's discuss three lessons that are relevant to set the stage for the philosophy powering Adama.
First, we have the pareto principle where 80% of the impact is from 20% effort. When applied to customers and their associated machine load, this principle is recursive. 90% of customers will barely use a service while 10% are fairly active. Of the 10%, there is another minority that are disproportionately active. And then within that minority is a yet another super minority.
Second, every service that you buy operates on a fairly refined and well defined atomic unit. Generally, these units are small for a variety of reasons. This makes the service easier to build, but it transitions a burden of the composition onto the builders of the product.
Third, there are a predictable number of decisions and steps in building any new service around an atomic unit. There is also a predictable cost in terms of human resources as well. The novel aspects all relate to the form and function of the atomic unit, and everything that exists outside that atomic unit is fairly well established literature and repetitive.
The more services you build, the accidental complexity vastly overshadows the essential complexity that defines the services. This observation explains why microservices and the associated orchestration is a popular trend.
The lessons inform me that there may be an interesting bet against a service orientated architectures. AWS promises a number of scalable services with the allure that the combination is scalable, and the Faustian bargain is that using AWS saves you both operational burdens and system engineering costs.
To AWS's credit, this is mostly true although many services have fine print which requires architects to read. The open question is the cost.
Keeping AWS as a case study, It's easy to dunk on AWS as it is expensive, and there are plenty of stories of people migrating to dedicated hosts for their needs saving cash. It's good to be AWS as many of their services create tremendous lock-in which require tremendous talent investments to escape.
However, the cognitive cost of weaving together AWS services with your logic is non-trivial and requires an engineering team well versed with AWS's offering.
Worse yet, if you start to see user data as a liability, then all these services are breach points. The service model forces you to shred your data across many machines, and in the cloud that may mean a single user has data on a thousand machines. It's easy to dismiss this as fear, uncertainty, and doubt.
However, we should be collectively tired of hearing about large scale data leaks. Based on this alone, we must strive for a radically new and simple organizing principle.
We can call this new organizing principle a “micro-monolith”
This “micro-monolith” concept emerged from attempts to build an online board game where the service approach really breaks down. It was just too hard to describe as the emergent state machine for a complex board game is potentially massive.
Recall, the goal of software is connect people to their data and transform it. We have come to believe that board games are a perfect model to measure architectural complexity against.
With this new architecture, we believe we can get rid of many services in one swoop. The aspiration is that we can focus on the essential product and business logic without the messy adaptation. As dreamy as this sounds, we must address the cost.
The core cost of eliminating these services start with a new design game. Instead of breaking down the problem into very small atomic units, you break down your software down into large atomic units that map into the domain of the endeavor.
For example, instead of breaking a person down into several rows across a few tables in a database; you have one big Person object. And here, big can mean GB rather than KB. The ideal is that building a back-end with Adama should mirror the speed of building an Excel spreadsheet, so I aspire this helps software be built faster and have a significantly lower operating cost.
Let's talk about how this all works...
The executive view from 10,000 feet is fairly simple as there are three core concerns: routing, Adama, and the log storage.
Adama operates on structured documents as the atomic unit. Much of the design follows a scalable document store.
With documents as the atomic unit, the router is simply a layer of indirection allowing infinite scale in terms of the total count of documents A unique facet of this router is based to the fact that Adama operates with stream-centric APIs rather than traditional request response. Stream-centric routing is exceptionally hard.
So, let's talk about the core streaming API. It starts with users establishing a connection to a document. On establishing a connection, the document is sent to the user. When the document changes, updates are also sent to the user on the connection. At any time, the user may send messages to the document which will change the document via the internal language.
The language powering Adama allows the documents stored within Adama to be changed in a few ways. With the language, you statically structure your document with fields, tables, maps, tables within tables, etc. You then expose message handlers via channels such that users' messages can change the document.
This message handling allows Adama to transact against the document and produce delta changes. This means that when a message comes in, the Adama code is executed against the document, changes are observed, and if the message finished, then the change is persisted. If the message was aborted, then the changes are reverted.
This has some interesting consequences.
First, a major consequence of this model is that documents can become large as everything is related to the rate of change. Contrast to other document stores where you reading and writing the entire document will discourage document growth. By focusing on data changes, Adama follows a more traditional database structure except the language translates domain changes to delta changes rather than speaking the database language.
Document changes are written to a log. Log storage can scale up tremendously since writes are sequentially written. On a commodity machines, we can easily achieve multiple GB/second without deep engineering investments.
Pulling back a bit, If the user experience depends on one document, then the entire user experience is served from memory and changes manifest in exactly one write to log storage. Thus the user experience is both fast and reliable.
At this point, the basics of Adama are as a document store with real-time updates, so now we dive into two expansive pieces. First, we need to talk about the language.
Second, we need to see how the language and system work together to provide a vast array of features for building web sites beyond being a document store.
Adama is a complete platform.
Adama is yet another language with braces in the tradition of C It has many familiar constructs for control flow like for loops, while loops, if statements and etc.
However, unlike other static languages with legacy behavior like integer division, Adama applies the principle of least surprise yet maximal correctness.
For example, let's recall back to math class where we were taught to call out division of zero; Adama forces your hand by having division escape out the expected types.
Adama requires thinking about failure very precisely and there should be no surprises as surprises are generally bad. Adama takes the view that the type system should eliminate most problems.
For example, the square root function in Adama's math library returns a complex number; this manifests a maximally correctness.
As a developer, you will create an Adama specification file which is just a text file that outlines the state of a document. At the root level you have fields which can be single values or records.
Records can also be kept in a table and tables can be held at the root level or within another record. Interestingly, tables are the way way to create "new" data. The language does not expose a memory model that developers have to think about as everything persisted is accessible from the root document.
Adama embraces table queries as this not only makes code more expressive, but it also provides the compiler the ability to optimize and have static query planning.
Once you have structured your data, you can populate a document at the time of construction.
Change is the only invariant in life, so once a document is constructed; changes may require us to upgrade the data. There is a load event that allows us to gate off of state to upgrade or mutate the document based on new code.
Once constructed, message handling is one mechanism for documentation mutation. Here, we simply open a channel called “draw_card” which allows players to ask for cards. Clients simply send the message to the document, and the changes in document are made manifest and whole.
As data fills the document, you want to expose computations and tables in a meaningful way to the clients; this is done reactively via formulas. So, here, we express the count for the number of cards and a boolean indicating if there are any cards available. These fields update when the deck update.
As you expose data to people, it's important to consider the privacy of that information. This is vital if you want to maintain secrets. Here, we have a custom use_policy on the super secret data field. The policy is evaluated when that data is being vended to people. Protecting fields is not enough as we also want to limit side channels, so we require the policy to even know that the object exists.
In this example, the policy is used for both a specific field and the entire object which is a bit of overkill; however, this is a robust method for ensuring that people can only see what they are allowed.
Privacy policies provide a security model to eliminate information disclosure; however, it is not the most efficient way to handle many scenarios.
This is where the privacy bubble comes into play where fields can be reactively computed with the viewer in mind. These viewer dependent queries allow for efficiency in vending data. These fields will leverage privacy policies to prevent accidental disclosure.
Another way to change the document is by using time via the state machine. Each document has one state machine label to run at any given time. Here, the document starts with the countdown variable set to 10 and every minute that passes will decrement the countdown variable.
Using the state machine, we can invert the typical control flow of message handing to message asking. Here, imagine the document is a dungeon master that is asking players to answer questions. Here, the document is asking the current player to make a move. This provides workflow capabilities, but also helps enforce game logic by not accepting messages out of context.
As we build up the document and make it do something useful, we will want to lock down who can read the document. This is available via the connected event which must return true for the given user to establish a connection.
We can further lock down who can create a document via static policies within the Adama specification. This document and language are the perfect place to stash configuration and access control policies for everything that isn't document related. The below create policy really locks down who can explicitly create a document. Document invention is the process of creating a document on demand with zero arguments for a constructor. Combine document with a flag to delete the document when everyone closes is a great way create ephemeral experiences.
Philosophically, most behaviors and configuration belong in the adama specification to further simplify operations.
Adama takes care to protect data from accidents, so document deletion requires internal logic. Either the document decides to delete itself, or an external agent can force a deletion which requires a policy to allow it.
We can leverage the language as well to open more ways of talking to a document. Here, we allow read only queries via HTTP GET and a mutable HTTP put. This allows Adama to speak via Ajax, but it also allows web hooks to communicate to a document.
So far, we have been dealing with document structure state; however, Adama also handles big blobs that are attached to documents. Adama has a streaming upload API that requires first a permission check via the can_attach event which informs whether or not the user can upload the attachment, and then a attached notification is fire once the attachment has been uploaded to a blob storage system like Amazon S3.
In the language, an asset is a ID to locate the blob of bytes, the content type, file name, various content hashes. When combined with with the web handling capabilities, each Adama document can become a mini privacy-aware CDN such that content is protected by deep logic while the actual asset is served from a machine very close to the user.
Before we jump back to the front-end, let's talk about the immediate future.
The most likely feature to land this year is Adama being a good citizen by calling out to other services, and I aspire to have many great services with examples.
We intend to build Adama's billing system with Adama using Stripe, and this may be a new business line as I have my own business logic power usage based subscription billing.
The log storage system right now puts data at a 5 minute risk period on machine failure. This is due to a desire to minimize latency while maximizing throughput, so I’m looking to buy a logger to durably persist each write where the number of writes that are allowed in-flight is bound by config. The key tension is latency and durability risk.
However, we also want to extend the length of the logs to be infinite as I do bound the length of the logs with periodic snapshotting. The core reason to extend this length is that I like the idea of having infinite auditability. With a log system, a powerful investigation tool would allow me to write a search predicate like “explain the value of this field in the document” and get a complete history of how the field has changed and by whom.
Beyond auditing, the log creates a powerful development tool to understand the effects of change. For example, I could fork the entire log and maintain parallel representations while testing a few feature.
Right now, a document on a cheap machine can handle thousands of users with thousands of messages per second. By replicating the document state out to another region, I can increase the total users reading a document by horizontally scaling the query and privacy logic. This requires replicating the document from a primary region to other regions.
This will also serve to decrease the latency for the read only web handling of a document if you can tolerate inconsistency. This will turn Adama into an edge-compute CDN. That’s pretty cool. This will require some policy changes like having the connected event be configured to static for some users to let them connect. However, it should be possible for every single person on the planet to have read a private version of any document. Because we have a language, we can introduce new primitives to allow massive influx of writes. The key scalability limit will be writing to a document, but can boost in two ways. First, we can temporally batch writes together to bundle them into the log. Second, we can provide a reducer in the language such that the messages from multiple people get aggregated together into something meaningful for the product.
So, as promised, we are back to the front-end.
By embracing a reactive back-end with streams, the game has been reduced to connecting to a document for reading and then sending messages to update it. This creates a radically simplified game, and what we have found is that just using the browser as it with a reactive back-end is exceptionally fun. However, We are not needing the typical features of many frameworks.
So, in a very real sense, building a modern interactive application only need one reactive layer; the reactive back-end really simplifies the front-end to almost being a template. The only question is making a great user experience, so we find ourselves facing a capability spectrum
With the browser as the runtime, this spectrum has a tension between just plain old HTML which works great for everyday people doing everyday tasks with all regulatory compliance like accessibility. The other end of the spectrum are super complex scenes powered by game engines.
As an example, RxHTML adds new HTML elements like lookup and new attributes that enable iteration and conditions. These new facets allow the HTML to be a fixed function of an Adama document much like a template. This approach also allows E2E static analysis where the typing of Adama’s document can find problems in RxHTML. For applications that only need interactive HTML, this is a full stack.
We have delayed this project, but in the context of building a Roblox for online board games. We want an image templating system that reactively updates to documents as this is a very simple gaming runtime.
Part of this future has a WYSIWYG editor where the template is buildable online such that design changes manifest instantaneously in the product. However, given the current engineering demands, this is a stretch goal for 2024.
Beyond the mini-frameworks that we are investing, Adama also can nicely co-exist in any existing framework like React or Vue.
So, we believe Adama is a powerful replacement for many back-end services. We've walked through the various features that make Adama rather complete as a service. We shared the future of Adama which is interesting. Also, we've talked briefly about how a reactive back-end simplifies front-end almost to the point of being a dumb terminal.
We further believe we can go much further...