Writing to disk and the tyranny of small things
By Jeffrey M. Barber
Today, I’m writing about the process of writing to the disk and building a durable solution which can be survive both a process death and power outage. Well, actually, this is wasn’t written today but was written over the course of a week of writing code; in this spirit, it’s really a journey of how I approached a problem. At core, I’m building a custom embedded data store like thing for Adama.
I’ll also be looking into using RocksDB before I embark on writing a specialized solution for Adama. The goal is to solve the “single node” storage problem for Adama with a reasonable backup solution, and I’m optimistic that a reasonable solution can be found.
Actually, scratch that, this isn’t about reasonable, this is about having fun to squeeze pennies and make computer go fast. The central problem is what could be called a “tyranny of small things” where the hardware works on predictable batches. In other words, the more you give the OS, the more that will get done. Let’s validate this to ensure we are always close to what the metal can provide.
Progress on a new data store
By Jeffrey M. Barber
I’m happy to report a few users to the system! That’s pretty great, and I’m ahead of my own expectations.
Progress has been made on the high priority blockers as well as continued polish on the main infrastructure. Just the other day, I sorted out an interesting race condition where writes were out of sequence. This was detected as I was load testing the new storage engine. I’m feeling joyful as the new storage engine is shaping up to be better on a few dimensions, so let’s chat about the changes.
Introducing volatile data
By Jeffrey M. Barber
For those keeping track, I have a manifesto on user interface architecture where I outline how I intend to build a minimal UI kit on top of Adama. I’ve had a few starts of it, but I was getting distracted by the lack of polish on Adama. Now that Adama is stable where the primary critique is a combination of performance and scale, I’m paging in the second leg of my strategy. One thing that is on my mind is the role of volatile data.
So, I replaced gRPC with net and discovered more about latency
By Jeffrey M. Barber
As I’ve been mentally tortured by poor performance (and my inability to prioritize), I replaced gRPC with vanilla netty and very simple code generated codecs. Today, that effort went to “production” and the results are…
Next steps post early access launch
By Jeffrey M. Barber
So, I launched and the next steps should be to go solicit people, hone my messaging, and try to get adoption. Fortunately, I have a few users, so my #1 priority it to support them. Unfortunately, I’m too embarrassed about the current of state of things and intend to focus on key technical limits. The key limit right now is the inability to scale up, and I’m perfectly happy to scale up without paying customers. Since I’m the largest customer of this infrastructure, I have to be happy.
I measured, got punched in the face, and I’m “early access” launching anyway.
By Jeffrey M. Barber
I love making plans, but…
Everybody has a plan until they get punched in the mouth. Mike Tyson
My problems started when I started load testing.