I love making plans, but…
Everybody has a plan until they get punched in the mouth. Mike Tyson
My problems started when I started load testing.
At first, it seemed my problem is that I’m using MySQL via RDS as a poor man’s logger on a basic instance. I’ve also never optimized my queries, and I’ve been running with undue optimism. Well, maybe not too much optimism since I’ve got all sorts of dashboards ready. Load testing was my final readiness step as I intend to measure how much an Adama host can handle so I can build the pricing model. Let’s just say that I have my work cut out for me.
Before I bury the lede too deep, this is a fantastic place to be! While it is super disappointing that the code, which I am only confident as (mostly) correct, isn’t performant. The two technical items that I had blocking the my previous concept of an “early access” launch were (1) a web page to recharge accounts balance using the Stripe API, and (2) enforcement of service cut-off. I still need to finish the tutorial and sketch more of the reference book out, but I’ve been knocking down the technical aspects with gusto. The thing bloody works E2E! I can taste it, and I can share it.
This ability to share and get something into other peoples’ hands is the most important aspect of a new business because you can start the virtuous process of feedback.
However, now, I need to take step back and see if my thesis for this business is worth while. A core desire of mine is that people adopt Adama internally within their own businesses because I believe this model is amazing. So, is it? Well, the only way to know is to do the math. Even though I’m not happy, let’s take a look at the cost. I’m using three hosts in my production cluster right now. An a1.medium ($0.0255/hr), a1.large ($0.051/hr), and a db.m6g.large ($0.152/hr).
I then bench-marked Adama with 1600 player connections connecting to 400 games with each player making a move every 10 seconds. My thesis is that Adama is cheaper than leveraging a bunch of difference services together because the compute and distribution are all local, so let’s compare Adama’s raw cost ($0.2285/hr) to AWS and Ably.
With AWS, we can use lambda, gateway, and dynamodb to build the board game. We will assume that each game requires at least 1 MB of memory and 50 ms of time to process.
| Service | Rate | Units Used | Total |
|---|---|---|---|
| AWS Lambda | $0.2 per 1 million requests | 576,000 plays = 1,600 players * 6 plays/minute * 60 minutes/hr | $0.1152 |
| AWS Lambda | $0.0000166667 per GB-second | 11,250 GB-sec = 1,600 players / (4 players/game) * 1 MB/game * (1 GB / 1024 MB) * 576,000 plays * 50 ms/play * (1 sec / 1000 ms) | $0.1875 |
| AWS Gateway | $0.8 per billion messages | 0.00288 billion messages = 576,000 plays * (1 message to send + 4 messages to get) / 1,000,000,000 | 0.00230400 |
| AWS Gateway | $0.25 per million-connection minutes | 0.096 million-connection minutes = 1,600 players * 60 minutes/hr / 1,000,000 | $0.024 |
| AWS DynamoDB | $1.25 per million write | 0.576 million writes = 576,000 / 1,000,000 | $0.576 |
| AWS DynamoDB | $0.25 per million read | 0.576 million reads = 576,000 / 1,000,000 | $0.144 |
| AWS | - | - | $1.04900438 |
Adama, even in failing my expectations, costs 78% less than a modern server-less approach with AWS. A cheaper option could probably found with their GameLift service, but that is a radically difference beast since that is heavy compute driven, and their example pricing page suggests $650/day with a peak of 6,000 users. The interesting aspect is that I should consider running Adama on GameLift, but that degree of vendor lock-in is worrisome.
Playing with the numbers in various ways, it’s clear to me that Adama is competitive, but maybe another more specialized service is better. For this, let’s compare Adama to Ably.
| Service | Rate | Units Used | Total |
|---|---|---|---|
| Ably | $1.25 per million messages | 0.576 = 576,000 plays / 1,000,000 | $0.72 |
| Ably | $10 per 1K peak connections | 2,000 = ceiling(players / 1000) * 1000 | $20 or $0.028/hr amortized |
| Ably | $10 per 1K peak channels | 1,000 = ceiling(players / 1000) * 1000 | $10 or $0.014/hr amortized |
| Ably | - | - | $0.76 |
It’s worth noting that Ably is incomplete as it only provides messaging, but Adama costs 70% less and does more than messaging. However, Ably has amazing latency which Adama can’t match (yet) along with many integrations (which Adama will never have), so it isn’t a great comparison. It does signal that I’m on a good path as I do intend to lower the cost and drive latency down.
What doing this math does do is confirm to me that I do have something at hand which lifts me from my depression as my expectations were not met during random uncontrolled load testing. Obviously, machines have limits, and I can’t expect magic that a single core host. Maybe, something in my math is wrong, or server-less is just a cruel joke.
Regardless of the limits and my limited budget, I’ve decided to go ahead with a launch in March. However, I want to be honest, so I’ve updated the FAQ with all the caveats. I’ve updated Pricing with a notice of early access. I’ll avoid taking any money until things are up to my expectations as this grants me the freedom to misbehave. Since the documentation is incomplete, I’ve setup a discord channel to assist the curious.. The tutorial is finished.
The technical aspect of my mind is focused on solving the data side of the house since I’m none too pleased, and profiling has revealed the link between web and Adama is not where I want. Funny enough, Ably has posted about the gRPC issues.
Clearly, my recent github activity indicates I’m excited to get cracking and making this kitty purr. Yes, I’m probably going to write my own networking layer. Yes, I’m probably going to build my own chain logger thing. As always, I aim to misbehave.
However, there is a tension within my soul between two forces. There is the entrepreneurial spirit of enabling people with new solutions versus the perfectionist craftsman. On one hand, there is the thrust to get to market, build community, and have a grand ole time. On the other, focus within the workshop grinding on all the details until there is harmony.
Which force should win?
In thinking about this, problems can be classified in various ways. If the problem can be solved by throwing your money (as the customer) at it, then it’s only a question of margin. This applies to my gRPC problems. If problems can’t be solved by throwing money at, then that’s a real problem if there is growth. This classifies my data tier challenges. If problems are related to growth, then that’s a good problem to have.
Currently, it’s not clear if I have a good problem since I have no growth. This is at the root of releasing in this strange “early access” state since my #1 business priority is to find believers in the idea that are willing to look past the warts. Fortunately, I’m at a good place with the idea that I can advance my strategy (link) and start work on producing the IDE.
For the next few months, I’ll embrace randomization and forgive myself for being a conflicted soul. I only hope that you’ll forgive me for not embracing your needs, but please join the discord or send me an email.