End of 2023 Work Log (Or what is capable if you give a shit about coding)

Well, it has been an exceptionally busy year for myself as I’ve been building out this cockamamie platform. Since I have a few clients and a handful of users, I wanted to spend a day summarizing the progress made this last year.

It’s been a good year for getting things done.

January (10 commits, 2 posts)

Developer tooling gained the ability to bulk upload assets, and this is part of the plan for Adama to compete as a static website hosting provider. This came with a myriad of improvements and operationalization of assets. In Adama, assets are attachments to documents where we know their document unique id (UUID), their original filename, content-type, file size, sha384, and md5.

There were a handful of bugs around HTTP/1 that were addressed, and the path towards a CDN started to form with AsyncSharedLRUCache was built to reduce the pressure on the database and latency for looking of certificates.

It was a fairly low-key month.

February (15 commits, 2)

DataObserver work started as an experiment, and this is key towards massive scaling such that a billion people could observe a document. Replication is going to be exceptional important, and this was some groundwork to start making replication a consideration going forward. At this time, the missing piece for document replication is replica placement and polishing up the document replication aspect.

A round of annoying polish was done such that config was unified around hyphens rather than a mix of underscores and hyphens. It’s just wrong that “http_size” and “websocket-size” would be in the same JSON. Yuck

Since SSL is so expensive, the restore logic used a common connection to S3 and Adama gained the ability to do connection pooling. The connection pooling logic is 100% organic and grass-fed without a library, so it may have problems… However, it has been working exceptionally well for the last 10 months.

Assets can now be cached within the web tier providing exceptionally low latency asset fetching. An important aspect of assets is that they are immutable after translation. That is, we leverage Amazon S3 to provide an immutable blob store such that we never overwrite an existing asset it. This allows Adama to perfectly cache the result from Amazon S3.

March (0 commits, 0 posts)

Nothing got done in March as I was touring the south west in my RV, and the grand canyon is exceptionally grand.

April (37 commits, 1 post)

Recovering from the long road trip had a round of polish on the documentation. Lists gained the map function.

public formula foo = iterate _table where x > 0 map @lambda z: z.id;

RxHTML was improved greatly as error handling started to become considered. Functions got some paint so .await() can be used within a procedures. date, datetime, and time, and timespan became types within Adama. These new types also started a debt on building a robust date/time stdlib which will last forever.

RxHTML started to feel real with more single-page application features. Channels got the ability to fetch with a timeout as chess requires that to some degree.

May (131 commits, 3 posts)

A new intern joined the team: Seyi. Seyi started work on clikit which simplifies my life by generating the code for the CLI via an XML specification. Switch statements came to the language.

A dumb experiment was started to move faster with RxHTML which was great to bootstrap from reflection, but the l-tree system would prove way to complex. There was this whole “edhtml” phase that I’d rather just not talk about. :/

ViewState became controllable from the backend. ViewState.goto changes what the viewer is looking at while ViewState.merge will merge a delta into the client’s vierw state. Fix a huge bug when comparing integers as Integer types are not comparable!

bundling backends became a thing as projects started to grow and require multiple files. Lambda (@lambda and where statements) gained the ability to capture time as part of their closure. Function painting includes viewer so @viewer can be made available to procedures.

A lot of date math was shipped during this time. AmazonSES was integrated as a first party service, and all the SigV4 stuff became supported directly rather than relying on the AWS SDK which added 50MB to the final jar.

Domains became a first class citizen. This is a big deal as a single domain can point to a document, and businesses using Adama can treat the domain like a level of indirection. Many… Many bug fixes were shipped during this time around a great number of topics.

Document authorization became a thing where documents are able to be an authority for user identities. The stdlib around dates and time shaped up tremendously.

June (44 commits, 5 posts)

A massive bug was fixed such that infinite cycles are now prevented. If you allow a record to include itself, record R { R r; }, then that is an infinite structure which explodes everything! Annotations were added to messages and records. A great number of bugs were sorted out as the service was hitting reality with a client. Stripe started to become a first class citizen. Services were becoming more of a thing. A service can now leverage the dynamic type since static typing and remote service is difficult (especially for complex services like stripe).

The biggest release in June was a massive re-ordering of the type system. The type system was originally designed to be top-down such that if type X appeared prior statement Y which uses X, then it would work. However, if type X appeared after Y, then it would fail. I introduce a topological sort to re-order how typing happens, and it would take five months to realize that I could do typing much better (which I intend to take on at a later date).

apikit started to gain the ability to generate a java SDK to talk to Adama. The research into building out Multi-Region support started, but it proved to be insane to split endpoints. A different strategy started to emerge, and the control plane service was shut down.

The stdlib gained the ability to speak roman numerals, so that’s just fun. ;)

clikit shipped fully! Congrats Seyi!

July (169 commits, 3 posts)

I leveraged Seyi’s clikit to fully migrate from the hand-coded cli parser to the code-generated formalized version. This greatly simplified and unified all the groups and commands such that help is generated, and this was a huge boon to growing features easily and cheaply.

Introduced @self such that formulas and bubbles can pass the containing record to a procedure.

Multi-region design started to take form with a split between a global service (and single point of failure) and regional services. This required a bunch of upgrades to apikit and the start of a lengthy process to clean up the API implementation and sort out the dependencies around the database. Ultimately, the goal is to first wrap the database behind a global tier and then leverage a replicated database. It’s going to take years to fully remove the single point of failure.

The edhtml experiment ended and was removed, so that’s both good and sad.

The devbox tool form to empower Adama developers to work 100% locally with sustainably sourced code. The devbox also started to gain telemetry plumbing to test out some new production code. Caravan got a massive upgrade to be free of document ids as there was a substantial bug with that approach, this also took pressure off the databases as we now store the Key within the disk.

The stripe API was generated from an OpenAPI spec.

There were an innumerable number of bug fixes. For example, –config now works independent of where it is in the CLI.

For improving how developers start projects, a new kickstart tool was developed to start a new space with all the trimmings.

Since it was clear I was never going to change the version of the jar, switched from 0.2 to MAIN. I’m using a mainline strategy and version numbers are now code-generated to be part of the platform.

Domain support fully shipped to a useful state. @web support greatly improved and was enabled to be async so Adama can be a fully fledged web proxy now… which is cool. A new service identitysigner allowed private keys to be embedded into a document (securely via encryption) so documents can sign identities under an authority.

August (251 commits, 2 posts)

There was a lot going on in August, and most of it was small fixes here and there along with massive improvements to RxHTML.

The true nature of the stdlib for date and time took form. Conditions in RxHTML could now leverage data for nice effect. RxHTML transformed from from the compiler to rxhtml.js, and the ability for the devbox to pull from rxhtml.js was finished such that the devbox was fully supporting my workflow. A tool to convert the reflection to a diagram was made to visualize the entire product via mermaid.js.

The search operator was introduced

public formula search_result = needle =? haystack;

This operator provides a nice way to think about a fulltext search. Currently, it returns true if any of the words within the need are present within the haystack. At a future time, this will provide a nice way to do full-text indexing within tables.

The start of a dynamic query engine was started, but this has stalled out. The vision is to be able to have a limited query language from the client to provide enhanced searched using operators and strong value checks.

surge.sh failed me for the last time, and I migrated all static Adama properties to Adama.

Content-MD5 was integrated into the asset system such that full data-integrity is a key consideration.

Regionalization work was in full swing to isolate the database from the core service via interfaces. stdlib got some abstract list love with flatten. Tables of messages got materialization to provide indexing.

message M {
  int x;

  index x;
}

#some_code {
  table<M> t;
  t <- {x : 1};
  // ...
  t <- {x: 100};
  var mat = iterate t materialize;
  var foo = iterate t materialize where x == 50;
}

The value of this is to allow control over when indexing happens such that developers are not stuck with insertion sort mechanics. This provides a nice way to organize data on the fly with a balanced view between performance and flexibility. This body of work came with many bug fixes for indexing.

Since communication between client and server may want additional signals, lossy and require were introduced as field paints such that a field within a message could me marked lossy such that ingesting the message into another thing wouldn’t result in a failure due to the thing not having the field. Similarly, since ingestion is partial, required was introduced to force ingestion to check the present of a field.

Replication from the document to another data source was sketched out and soak in my brain. The key idea at play with replication from the document is to use the Adama document as an authoritative data store and then replicate out to large scale cross-document services. This hasn’t fully shipped, but a lot of plumbing was put in place such that we can maintain the thinking as things evolve. The hardest aspect is ensuring deletes happen, so a new mark for deletion and commit deletion was introduced. This provides the book-keeping required to ensure the document is fully drained over time.

Again, during this month, so many bugs were fixed across the entire stack.

The devbox debugger was rewritten to be 100% reactive and a nice dark-mode for the hacker-feel. The start of a new book generator was started, but I got stuck on how to implement search.

Conditions in RxHTML got all the comparison operators. The devbox got linter support along with some tactical improvements such that data just doesn’t disappear due to a file going poof during a sync.

Document secrets were abandoned in favor of using host key signing which reduces pressure on the database as well as an automatic form of key rotation over time.

The amazing and complicated tree.js got insanely efficient by batching up subscriptions and publishes to effectively chunk data. It is insane how fast it became within this.

Load shedding became a data service priority which allows a fast evac from a machine or during load events.

The start of a new type system would get kicked off only to languish since the apocalyptic issue has yet to fire. There are severe bugs with how free() works with where clauses for various reasons.

Metering and billing went multi-region and Adama became the reducer function for every space. This provides a semi-real-time daily view of billing along with a monthly summary.

During this time, the devbox became truly great!

September (166 commits, 2 posts)

More updates to RxHTML around time based eventing (rx:delay:100=”…”). There was a serious reactive invalidation bug that was fixed. A new canary tool was developed to do load testing. For performance reasons (and based on load testing), privacy policies are cached within a broadcast. A fun transform “time_ago” was introduced to RxHTML to convert a datetime stamp to “5 minutes ago”.

Slow readers were getting disconnected due to the ordering of idle handlers within the WebHandler, and this allows people with 50Kb/sec connections to download adama.jar. Fixed a huge bug with infinite invalidations firing based on certain reactive topologies. Some safety features were added such that records couldn’t be part of a message as there is a native/reactive boundary to preserve.

There is a new order_dyn feature which allows sorting tables or anything based on some kind of signal from the client. RxHTML also go order-toggle as a command that will preserve the prior ordering so users can sort by two fields and the sorting is stable.

Google SSO became a first class service for documents to leverage. A gossip bug was fixed such that it will never remove itself locally.

Adama documents are now metric providers.

metric user_count = _users.size();

Is now queryable from the CLI tooling, and you can sum all metrics up or single out a specific document or document subset by prefix. The query operation is fairly expensive at the moment, and there are plans to integrate the metrics via some kind of GET uri with an authentication story.

Since control should be delegated, a new space policy was introduced such that owners can delegate control to an authority over a space. Domain support now supported assets for attachments. rx:repeat came to RxHTML which allows dynamic variable content.

The account recovery process for the platform got prettier!

Invalidation of bubbles based on viewer changes only happen when the associated viewer fields changed. This require functions to record their viewer dependencies directly.bubbles and channels got a default policy mode which allows privacy to be enhanced and further auditted.

Rate limiting became a concern with a bunch of support, but it didn’t land fully. It’s in a half baked state with networking support. The design is to limit by token buckets using consistent hashing with Adama as the providing fleet. The protocol between client and document now supports the ability to enforce a disconnect of a person.

Even more performance upgrades to tree.js, and it really sings now! Yay!

Type inference around the viewstate started such that tooling can visualize the various view state per page.

Logging became a concern and tests were done again Logz.io until they deleted the account for inactivity. It was such a bad sales process, so they are no longer on my list. Added a bunch of diagnostics based on some odd behavior that I couldn’t figure out until I realized Java 11 was missing some stuff, and migrating everyone to Java 17 was the fix.For operations, any Error exception now crashes the entire process for good reason.

Safari is strange, and the transition towards bulk commands was started such that events are atomic. This work is still inflight. RxHTML got auto variables which allow templates to have a unique id to use for id/label binding. The devbox got attachment support! apikit now generates the internal Adama service for Adama to talk to Adama. Various caches are now being swept to keep the system free of memory leaks.

There was a lot of polish done over the 166 commits.

October (123 commits, 1 post)

Adama started to stabilize and my focus shifted towards helping clients. RxHTML continued with polish with rx:monitor being a new way to watch an integral data point such that rising or faling edges can run commands.

Super awesome mode work was started such that tables produce table events rather than “hey, the table updated”. This works reduces the reactive pressure in favor of more specific events. This reduces compute during reactive updates by more than 50%, but this work wouldn’t land until November. During October, the work sat in an accumulation style mode.

Connection management in connection.js got a lot of love and is more inline with enterprise requirements. Various bugs with maybe types were introduced due to Java’s type inference needing casting in certain places.

WebPush and the associated machinery was started. Adama is now capable of supporting push notifications at a small scale, and the goal over the year is to expand push such that Adama can support massive scale with push. The push work has Adama picking up the role of device registration, so this means providing a map of identities to devices and keeping them accurate. The devbox also supports push in a limited way as well. Part of this is including web worker for doing push notifications on browser.

Since having a unique id makes data synchronization significantly better both performance and data transit wise, messages got the unique field paint such that if the developer marks an id field with unique then they super-duper-promise that it is actually unique.

Again, so many little bug fixes like indexing by principal.

During this time, a beta endpoint of Adama was launched (beta.adama-platform.com) and RxHTML got the ability to render results based on the environment using a new for-env attribute.

More stdlib love around date and time… it never ends.

The where clause now supports unions which means id == 4

id == 8 now will do two lookups rather than a full table scan.

So much RxHTML polish!

Graph associations was started to understand how to fix poorly organized data. Since nesting tables within tables provides non-trivial benefit, the idea of using a graph to organize tables spread out became a thing. This work is in progress, but the groundwork has started in an optimistic fashion.

Operationally, there is an upper bound on the hertz between updates such that resources are not spent foolishly.

Adama now has string template supports such that copy-pasta can dump entire text/html files into a document. This provides decent email template support. The templates also support variable substitution which is #neat.

November (149 commits, 1 post)

The ability to stash identities into cookies provides a saner and more secure way to handle credential handling. This stashing also serves the purpose of providing authentication to the @web handlers for secure delivery. A start on various social media authentication schemes was made by a contractor.

Maps now have an inside and outside operator.

record R {}
public map<string, R> m;
public formula in = "x" inside m;

unique became a way to provide distinct data within a list query.

record R { int z; }
table<R> tbl;
public formula u = iterate tbl unique first z;

Image transform logic was written to consider a future thumbnail service of assets. This needs a lot of thought given the potential memory impacts. A start of automatic formatting for adama code was started based on frustrations… The visitor for formatting is done, and the key now is the litany of special cases around comments and white space rules.

RxHTML got line and character numbers injected for debugging support. A tremendous refactoring took place such that the services were migrated to a new system module such that Adama-in-a-Box was available for testing. This allows standing up a fully fledged Adama instance for testing.

Due to performance challenges with how authentication was working, an alternative was stood up and the old method is planned for deprecation which requires a lot of work. I’ll need a way to tell existing spaces “Hey, this feature is going away on 2/1/2024”.

RxHTML got a new <connection-status> element that provides visibility on the connection status.

Push notification support got a lot of love since WebPush is hard on many fronts, but it works well enough now if visitors accept them. WebPush has better metrics so I can debug behavior, and there is now support in RxHTML to nag users when it isn’t available.

“app mode” work started in design, and eventually it would become a frequency setting such that updates are gated at a frequency. This allows the quadratic workload of a highly interactive app to be amortized over time. This induces latency, but it increase capacity greatly.

Super Awesome Mode was launched fully and the old invalidation pathway was disabled. Bubble invalidation got improved greatly to increase capacity along with a new settle pass that gives more tactical leeway to how invalidation happens.

Crypto work is now done in an offloaded executor that is shared which reduces pressure on the document’s thread. It was a giant mistake to have the crypto done within the main thread. Support Jitsi via a new service contract.

An RxHTML type checker has been started to provide better developer insight into large RxHTML projects. RxHTML got “domain-get” which allows bringing in a non-reactive JSON document into view.

Since deployments are painful along with machine restarts, Amazon S3 is now a bytecode cache for the JVM. This allows multiple machines to use the same artifacts and the deployment process can now regenerate the bytecode for the new platform such deployments are no longer outages for large spaces. This bytecode cache was absolutely massive change.

December (100+ ? commits, 4 posts)

The bytecode cache was fully launched to production.

Mobile support capacitor was started in gusto as a hackathon, and now RxHTML apps can be run on mobile via capacitor. This work also included getting push notifications working for Android and the associated upgrades to token registration. Since firebase is required for Android, a new generic product config was added to domains.

A lot of polish around HTTP error codes was added and unified between various HttpResponders; this provides me with better metrics on all the things that go wrong with the world.

A new “forced” event was added to input elements within RxHTML such that we can detect when the backend updates an input element.

Various onboard bugs were fixed as the webapp template moved to the new authpipe scheme. Authpipe can now signal success events which mutate the document.

The ability to download an archived document has been added which provides operational tooling to query the stored history of a document along with compacting the document to run in the devbox.

Capacity work was started such that I could upgrade the machine running Adama, and this yielded many insights on the work remaining to scale up capacity dramatically.

There were a lot of bugs discovered and fixed. For example, sending the viewstate a true and parsing as an int would hard-crash the entire service. Array support with domain-get was fixed.

Not shown in this is a lot of work around the private tooling I use to manage production, and getting it better organized. I’m not 100% done with the tooling, but I manage EC2 via tooling.

Many new queries were made available for insight into documents. Cron jobs are now part of a document:


@cron some_job_name daily 8:00 {
  // code to run every day at 8am
}

The cron job is synchronous to convert atomically based on a time stamp. To introduce async behavior from a cron job, a new enqueue has been developed such that channels can be called after the document has settled down.

message JustId { int id; }

channel foo(JustId task) {
  // do something
}

@cron some_job daily 8:00 {
 JustId task;
 task.id = 123; // or something
 foo.enqueue(@no_one, f);
}

This enqueue is available to every code context since it is just a future pending task, and this allows complex transactions to be orchestrated in many ways. A future extension to this will allow time to be a factor. It is worth noting that the transactional aspect of this is exceptionally important as we commit the intent to act based on time in a single transaction.

A new delay service has been introduced to allow time to be a factor in complex workflow environments. I also migrated the devbox to a isolated module since it is getting exceptionally large now. Then I finished the year off by making the language server protocol part of the devbox, so errors are in the IDE.

Wrapping up December, the first third party service has been added “httpjson” which allows Adama to talk to arbitrary endpoints if they ingest and return JSON. This opens up the door exceptionally wide, and 2024 is feeling exciting.

Conclusion!

Wow, I got a lot done in 2023!

It’s been a wild ride, and I reward myself with another year of hard work. Fantastic!

Seriously, I am doing this as a solo operator right now, and a key is writing unit tests. Currently, Adama has 6219 unit tests and a dashboard with 1027 metrics in a dashboard. I audit the logs periodically looking for noise and investigate everything no matter how small it may be. Doing good work is the only sustainable way regardless of the technical challenges ahead.

I’ve also written a bunch in the last year, but it hasn’t done much for me (which is fine since I 99% write for myself).

I am considering a possible twitch play and do some live stream coding of sorts.. I’m operating in several domains right now at my cognitive peak, and it’s great. If you want to join me, then I’m excited to have a conversation.

December 31st, 2023 End of 2023 Work Log (Or what is capable if you give a shit about coding) By Jeffrey M. Barber