As I was building the IDE with RxHTML, I ran into a problem that I’ve been thinking about for a while: escaping the silo with service calls. Suddenly, a nice-to-have or neat-to-think-about feature became a priority zero. The motivation came from attempting to keep RxHTML minimal without adding too many new components or javascript-code. For example, just creating a space and document at the same time is problematic.
The less I want RxHTML and the front-end to do, the more I need Adama to pick up the slack. Unfortunately, Adama documents are isolated from the rest of the world and even other documents.
What has landed recently is the core support for building a service, and it is fairly straightforward. First, the parser has been upgraded to define services
message SendSMSRequest {
string to;
string message;
}
message SendSMSResponse {
}
service sms {
std = "http";
method<SendSMSRequest, SendSMSResponse> send;
}
A service boils down to configuration aspects (a mapping of strings to Objects) and a bunch of methods. Fundamentally, this allows an Adama script to define the coordinates of a service, parameters used with the service, and the methods to interact with the service. A key assumption is that every service takes in JSON and emits JSON.
At some point, it may be worth exploring other formats, but for now Adama is fully within JSON-land. It’s good enough. There is a whole arc of importing gRPC services, but I intend to avoid that until I hire other engineers as that’s a great project.
Service methods can be invoked in multiple contexts. First, we can have reactive formulas.
public string message;
public formula send_message =
sms.send(
@no_one,
{to:"+15555555555",message:message});
The first complication to contend with is side-effects as the ‘sms.send’ function takes time against a remote service, so we must model this as a concrete thing to provide feedback. The behavior is similar to a maybe<T> except we need to know why the value isn’t present as that may provide actionable feedback to the user or developer. Traditionally, this is called the Either monad, except I’m going to call it result<T> since has three states: waiting (value == null && failure == null), failed (failure != null), success (value != null).
The second complication to contend with is that reactivity is a harsh mistress, and ‘sms.send’ has a rather brutal side-effect: an sms message is sent to a phone (at cost). Given that formulas are ephemeral, this could trigger a cascade of bad times. This is the motivation behind the recent work RxCache where formulas can execute requests against it and the results are deduped.
As of now, results are forever cached and garbage collected over time as changes to the formula manifest. The RxCache as a storage mechanism handles many of the core issue, but it is possible for duplicate requests to fire. For example, if the document changes hosts while the request is processing then the new host will issue a request while the old host manifests a routing issue.
Arbitrary services, like the ‘sms.send’ method must have some responsibility of deduping requests to minimize excessive side-effects. Alternatively, methods must strive for idempotence as retrying a failure request is a thing.
I’ve gone ahead and got the plumbing working such that I can define services, have services be available in different contexts (currently: reactive expressions, message handlers, procedures, and state machine transitions). There is a giant hole in making ‘@web put …’ asynchronous to support this, but I’m thinking about it; it is low priority at the moment. Another place to support services is bubbles (or view state queries) which requires sorting out how caching works at a per-user level. There is much fun work to do.
However, the big priority now is hooking up services. The first service is Adama itself, and this will solve some of the immediate tensions. Fortunately, Adama talking to Adama is straightforward enough for things like creating documents. Once I define this and enable sending messages to Adama documents from peers, Adama becomes a full actor model implementation. An Adama Document will be able to
One fascinating aspect to explore is expanding the service from request-response to streaming, and there is nothing preventing me from turning the new result type into a “most recent document” which is updated reactively. This creates some interesting efficiency challenges, but I also landed a new type called ‘text’ which uses operational transports end to end for CodeMirror 6. This new ‘text’ type illustrates a way forward to efficiently share document changes efficiently which requires primarily protocol investments.
I’ll most likely need an entire new type dedicated for storing document differentials.
Recent changes to Adama’s storage system required the introduction of a per-document mapping binding documents to hosts, and if I relax some of the current rules about how documents map to machines then portions of the induced actor network can localize which increases availability and lowers latency while increasing risk. This is pretty neat to explore as a low-latency junkie.
However, I need to be practical as I prioritize the execution of this.
I’m working at getting Adama integrated within Adama as the first service. However, the second service will most likely be twilio for various reasons, but now I have to solve how to store user secrets like API keys. This is going to be fun!