January 23rd, 2024 Next Level Software Testing By Jeffrey M. Barber

At the time of this writing, Adama has 6,344 unit tests for the platform. The dirty secret right now is that the RxHTML side of the house has zero tests because JavaScript is a garbage language and the browser is a dumpster fire. Now, ultimately, I want to have fantastic testing for RxHTML but also products that use Adama fully. There is a reason that many product engineers don’t write a lot of tests, but they are all bad reasons. However, I empathize with the challenge of protecting frontend assets, so I am going to do my thing and and invent a new way of doing testing.

Generally speaking, a test has three steps: (1) get the product into a specific state, (2) do a thing, (3) validate the thing done was the thing human want. It’s fairly straightforward, and the hardest thing is getting a complex product into a complex state with determinism. That’s the testing game, and the core difference between small unit tests and a full systems test is the sheer amount of state and the complexity of the interaction.

With Adama, all the state for an experience is within a single document. That document emits a log of updates. All mutations are consistent. The language is deterministic in many regards. There are a lot of nice things with Adama, so let’s exploit that with a new way to drive quality.

Crawl Mode: Information, Actuation, Compare

The journey starts with a snapshot of a document. In a dev box, we can fire that up and then have a copy of the entire product at a known point of time. This means we can crawl the product, find broken pages, take screenshots per page, capture the backing data per page.

We can then leverage the limited interactive model of RxHTML such that only explicit forms are allowed to mutate the document. Each mutation is an opportunity to fork the history of the document, so we can create a parallel universe for every action independent of each other. Based on this mutation, we could then have each parallel crawl the product looking for differences.

At the end of the day, this can be used for bulk regression testing such that every action has a before and after snapshot (both data and image) for. These can be compared to find how actions mutate different pages, and a prior set of snapshots can be used to form a comparison grid to track changes.

The key goal is to create various data sets which will generate different crawls, and tooling is then needed to audit changes. This approach does two things.

First, it will regression protect a product at scale providing developers and QA a way to audit the scope of changes made and verify positive changes or find negative impacts.

Second, it will generate performance data on the entire product experience. If the data is good, then performance data will be relevant to the production experience.

Record Mode: Seed, Execute, Compare

The crawler can be tuned to go deep, and this is best done by recording a session. Developers would initiate a recording session to start after seeding data has commenced. The system would then prompt when it is safe to interact with the product (green light go, red light wait).

Now, beyond regression testing a deep interaction story, this also provides the ability for staff to record a bug live and share the bug with all appropriate context.

Existing stuff? Boring, but OK

Now, Adama isn’t super unique for testing stuff.

An existing test environment could be leveraged using a WebSocket to talk to the product, so this isn’t outside the realm of possibility. A node.js test runner could simply use a WebSocket client to test a devbox, or selenium can be used out of the box.

This isn’t next level testing, but it would work right now without the crazy crawling mode.

Making the magic happen.

The key magic is to add a testing component to RxHTML and the associate libraries such that I can detect whether or not the UI is “done”. That is, there are no pending events, and everything is stable assuming the server is exclusive to the test user. That’s the key step I have to sort out to bring the crawler the life.

Just backend and unit tests

Adama has limited unit test support, and this is only used within Adama to test Adama at this time. I’ve considered removing support for it, but it does help test aspects that are internal to the document which would be hard to surface outside the document. Instead of cutting it, the goal now is to leverage it to great effect as a first class citizen within the devbox.

Imagine every time a deployment happens, the unit tests are run against the current state of the document in an isolated container. That would be… pretty cool.

I’ll start on back-end unit tests, and then investigate a crawler as I can.

I’m excited about testing as it will remove my shame around RxHTML.