November 26th, 2023 Mr. Jeff versus 20 Trillion Dollars By Jeffrey M. Barber

Take a look at the Cloud Native Landscape, and then realize… somehow… we have collectively made software harder (and probably much worse at greater cost). Now, this is the path of good intentions (along with some resume driven development and a dash of inexperienced leaders trying to be kool-kids-too), and every single box within the cloud landscape exists for a very good reason. I can read each little box and then see how it fits into the picture. If the people making all these things have a decent focus on sales then the brochure always makes sense until you find the tiny print of the various trade-offs…

At core, there are too many gaud damn decisions to make when trying to buy anything to stick together. Each decision then has related decisions until you need a fucking google maps inspired interface to navigate your own product. This then unfolds even more, and I recently watched a few talks from Microsoft’s Ignite. Specifically, I learned about Radius which is a new “open source application platform”. The first ten minutes are a good recap of the problem I’m working on (in a way), but then it veers off along with my anxiety about a competitor.

Some highlights from the intro are worth talking about:

But I would say that the first version of Cloud that we built basically just took everything that was already in your data center and it put virtual in front of it. There are virtual machines, and there are virtual disks, and there are virtual networks. There’s a lot of value in having things given to you through an API instead of being given to you from an order that you put out to CDW or to Dell or whomever it is.

This is under Cloud 1.0 which is where you basically have an API to do things that used to be manual. The innovation here was around logistics, and my claim is that everything since then has been… shuffling chairs around the deck. Yeah, some things may appear easier, but then a whole new lexicon emerges which is yet another layer of leaky abstractions.

There is a lot of value in learning how machines behave, and if you want a seven-figure job working within the cloud then that’s the game to play. However, the added cost (both financially and cognitively) is not worth it, and the whole cloud native is a good reason to simply retire from the entire game. Maybe, learn plumbing because dealing with literal shit is better than this. I’m 100% serious, I had my main line backup and a basement full of shit, and cleaning that up was at least satisfying at the end. Customers with a basement full of shit are very gracious for help, so #lifechoices.

I’m fixing computers that don’t exist, in a data center I’ve never been to, for people I’ve never met.

a quote from Peter van Hardenberg - Why Can’t We Make Simple Software?

While I was a high-level “I’m kind of a big deal” engineer at Meta, I had goats and shoveling their goat shit off my patio was more relaxing than some meetings. #lifechoices. Now, the Radius speaker was making good points around the infrastructure not being the actual application.

But the question that I have for you, and the question that I think we were struggling with when we went and built the Radius project was, where is the application in all of this? Because I think what we’ve done is we’ve gone and built a bunch of application oriented APIs, concepts that are familiar to developers, but we never actually built the application itself

This is insightful as infrastructure is a meta-game. The meta-game is helping developers make applications, and it’s a hard game to play because building is a highly dimensional game with innumerable trade-offs at every level. Again, as much as I criticize the landscape, each of those boxes exist for good reasons driven by some demand.

As crazy as this sounds, Adama is a competitor not to any specific box but to the entire landscape because I’m starting from first principles around the users (i.e the people that want stuff) and their state (the thing people care about). My thesis revolves around keeping it simple around a reasonable atomic unit of state. An atomic unit of state could be a user or an entire small business. The formal model of the approach I’m taking is an actor model and Alan Kay’s version of object-oriented, and I’m working to scale it up as an entire platform.

Perhaps, I’m just a crank, but let’s go through each core technical asset within the landscape: databases, streaming & messaging, application definition & image build, continuous integration & delivery, scheduling & orchestration, coordination & service discovery, remote procedure call, service proxy, api gateway, service mesh, cloud native storage, container runtime, cloud native network, automation &amp configuration, container registry, security & compliance, and key management.

Databases

Databases are great! However, they are a giant pain in the ass on at least two fronts.

First, interacting with them is the worse and the source of so many huge problems (hello Bobby DROP TABLES). How many of us have invented our own kind of object-relational-mapping tool? Yes, it’s a fun meta problem that will help you deal with security and productivity, but you’ll eventually just give-in and mangle some strings or use parameters starting with index 1 (why, because fuck you, that’s why). It’s just a sticky mess.

Second, schema changes require coordination as those string munging interactions are untyped. For the type purists and mathematicians like myself, SQL is a living nightmare of tracking changes throughout a system.

The suckiness of Databases is evidence by the various NoSQL solutions that have emerged over the years, and MongoDB is a $25B company precisely because databases have many problems. However, there is no free lunch, and the various NoSQL solutions solve a fraction of the problem.

Adama mixes the two by having a NoSQL architecture designed around a document store model where the documents have the bounty of databases within. Plus, the language has integrated query which is statically typed so bugs in every query can be dealt with during compile time. Now, this silos data away preventing large aggregations and other such queries, but developers have to learn to not do that because it’s expensive anyway.

Furthermore, as privacy becomes an increasingly important concern, data silos help minimize the fact that data is also a liability. With Adama, I welcome the great privacy regulations in the future!

As a consequence, the future playbook is to have a silo of data as the authoritative truth of application data with replication as the primary and only way to enable larger scale aggregation and queries.

I’ll have later posts on this topic, but organizations eventually realizes that the source of truth is fundamentally a log that you can tap into to build all sorts of services tailing it. There many names for this, but it’s yet another can of worms…

Streaming & Messaging

The only reason streaming and messaging exist is because databases are request response by nature due to the history of databases and the cruelty of storage hierarchies. However, databases ultimately leverage a binlog stream, and logs are the foundational unit of everything we do.

Adama is no different except the developer is in charge of the log implicitly by writing code that changes documents. When an Adama developer mutates a document, a delta is written to the log, and this log enables all sorts of future possibilities.

However, once the delta is written, connected viewers are updated reactively. The only explicit message is a mutation to a document, and all other messages are based on data changes. This creates a platform where messaging is built-in from the get-go. Seriously, the default in Adama is 100% interactive-reactivity and it is glorious.

The problem with both streaming and messaging (or pub/sub) is the nature of back-pressure and responding to failure. Queues can’t be infinite in length, and there are two responses: drop updates or compact updates. Adama chooses to compact updates when back pressure is needed, and this translates to a more reliable experience since messages compact with an upper limit looking like a poll.

Having dealt with planet-scale messaging systems and how they interface with data, Adama is… kind of a big deal for marrying the two together in an elegant spreadsheet kind of way.

Application Definition & Image Build / Container Registry / Container Runtime

Here, Adama doesn’t give two shits about images because it is a fundamental greenfield design where the entire application is split between a frontend RxHTML forest (i.e. a reactive template for a browser) and an Adama specification (the adama source that developers use to define a space). You can build entire web products with just two files which is… kind of amazing.

The core truth here is that there are way too many programming languages to define products. It’s a babel tower and a deep curse, and the entire cloud native landscape exists because no one can agree on anything. This giant mess is the consequence of the cloud meeting developers where they are at because developers are like fussy cats.

Yes, programming languages are hard problems and we are evolving, but… users don’t care. Users care about their state, how it is transformed, and getting results back quickly.

Now, of course, I wrote a programming language in Adama, but I can easily support WASM at some point such that the modern languages can plug in. However, I’m not interested in that right now.

For Adama, the application definition is basically a spreadsheet. The container is a spreadsheet, and the runtime is of my design.

Continuous Integration & Delivery / Scheduling & Orchestration

With a single script, you can push to production.

In Adama, this is a non-issue as deployment and upgrades are a first class citizen even at the language level.

Less is more when it comes to pushing out updates, so… you don’t need a bunch of complex stuff.

Better yet, because the ecosystem is deterministic, you can test entire products on your developer machine. I’m even working on pulling production data to the devbox such that production investigations can happy directly without affecting production.

Service Discovery

In the actor model, service discovery is free when all actors have names. In the context of Adama, this means if you know the space and key, then you can send it messages, make queries, or connect to it.

Having a unified model where every atomic state is addressable eliminates the need for service discovery as there are no services to discover. There are only actors that exist in their cosmic nature doing their duty of taking messages and updating other actors.

Extending Adama with services fundamentally requires putting a service behind a load balancer which has a lot of benefits. Now, a good reason to avoid a load balancer is that you want some kind of stickiness or less overhead with your service calls (direct host selection rather than a middle-man), and that’s a fair point. However, at this point, I’d rather focus on bringing external third parties in via a publicly addressable HTTPS endpoint.

Remote Procedure Call / Service Proxy / API Gateway / Service Mesh / Cloud Native Network

What’s worse than having so many languages is having so many network protocols, codecs, libraries, and what not to communicate data.

With Adama, the entire network stack has been simplified and no external clients are allowed in the code-base. This currently means that integrating new services requires writing a new client and adapting it as an Adama service, and this keeps the binary small and allows connection sharing.

A key part of RPC is how to deal with failure, and there are many ways to handle this. However, these failures should be managed and application developers should never need to write retry logic or instrument a call.

Using Adama should be painless where you simply make a remote call, deal with success or failure, and then move on with your day. By design, a failure should be considered terminal to simplify the product logic.

Cloud Native Storage

Just use Amazon S3 (or, a S3-compatible storage service). Amazon S3 is great. All praise-be to S3.

Automation & Configuration

Adama is in the convention over configuration camp with as few exceptions as possible, but a beautiful thing about a language is the configuration can just be embedded in the language.

This, tragically means there is no dynamic configuration, but I’m OK with that. Dynamic configuration is rarely fully tested and can have unexpected results.

Less is more.

Security & Compliance

Adama shines when it comes to security since people are a first class citizen even at the language level. The Adama specification has a notion of privacy built in, so it should be easy to be compliant using the platform as-is from a privacy and information disclosure perspective. With people as a first class citizen, the platform has the important job of enforcing security and access control, and my personal goal is to never allow developers the opportunity to violate security in the first place (I mean, as owner, I can… but developers can’t)

Within cloud native, it’s the wild west and it’s takes a tremendous effort to ensure every developer isn’t doing bad things. And even that is a leaky bucket! And impossible when you consider state-actors as rogue employees.

As a management tool, the language has the potential of providing executives an easy to read report of how data flows based on policies. I envision a standard process where executives and senior engineers get a report of recent policy changes and then audit the ramifications.

Adama will enable next level visibility and control over security, privacy, regulations, and other fun aspects to protect user data.

Key Management

Currently, the current plan is to just encrypt secrets within the Adama specification (i.e.) and then empower the platform to decrypt them. Rotation can happen with deployments.

With the chaos of all the litany of machines, key management requires a dedicated service indicating who and what can access important information. Then, there is a need to plumb critical information through infrastructure such that E2E information disclosure is impossible.

Seriously, if you have access to production, then you can do bad things.

Observability

The more you have, the harder it is to understand what is happening. The current trend of tracing user requests are shred across many machines to then reassemble the thing of interest means there is a tremendous waste of data being used to put together a forensic story.

It’s madness how expensive investigations are to execute and enable, but you want to know what is better?

Determinism.

There is no reason that I can’t take an actor from production, plop it into my devbox, and then poke it to understand what is going on. Furthermore, there is no reason I can’t take an actor from production and rewind the clock to understand how in the hell something happened.

This is an area where I believe Adama’s model will set an industry standard in terms of understanding why software behaves the way it does.

Beyond enabling super powers, Adama currently provides me with deep performance data where I can instrument everything by default. I plan to have this information available in vscode so developers get daily feedback on performance and cost. This is then combined with some feature work to enable planet-scale read so executives can understand cost and be able to dive into how much each feature costs along with usage.

Popping back to the main point

OK, I’m not sure how useful that list is, but the thing to consider is that I’m building an alternative platform to the entire cloud-native ecosystem. Yes, it’s weird… named after a goat… who was named from my favorite TV show.

At core, the real issue at hand, is that the cloud is a complexity trap and a lock-in. It’s great for cloud vendors, but not so great for small teams. Nor is it great for users. Once you build this way, you’re stuck and your software is no longer yours until you need a rewrite.

However, the skills needed to do the rewrite will be captured by cloud vendors. Most companies can’t compete with seven figure comp packages…

Operationally, it will be expensive, and the associated software will fade as if it never existed when companies go into the abyss. I don’t believe it is good for business (except the cloud vendors). Nor, do I believe it is good for customers.

Instead, I believe in a better world where most software is local or easy to install on a basement server. I want to be able to take a single file, install it on a platform server within my basement, and then be able to use it within my house and share it locally with my friends.

Now, it may be paradoxical that Adama Platform is a new type of cloud, but you can run it in your basement if you have a basement. Not everyone has a basement, and running a server is kind of a pain in the ass.

This was the best thing about the internet growing up where you got to mess around with your router to play games with your friends without any central authority. It was freedom, and it was glorious.

The entire cloud is a prison, and I just can’t anymore. I worked on the cloud, and there are aspects that are nice (like Amazon S3 and EC2), but everything else mostly creeps me out. It makes me cringe.

Or, maybe I just don’t understand it all, and I’m just an old man ready to fade into oblivion. I should just accept that I was a tool of big tech to solve big problems that probably didn’t need to be solved.

Except, I recently fucked up by taking on a client, and now I have to be an adult and operationalize this platform fully. The war paint against chaos is being put on. The general has dusted off this old armor and is diving into the fray once again.

Only time will tell if my idea manifests in something or I’m just yet another Don Quixote tilting at windmills. The important thing, in life, is that at the end I can say I tried. My last thought will not be regret.