December 17th, 2022 Hacking past the age of 40 By Jeffrey M. Barber

One year ago to this day, on my birthday, I left Meta as one of the few unicorn engineers to focus on living a good life (whatever that meant). While I miss the monotony of my balance sheet going up and to the right, the freedom to do what I want (or nothing) has been fantastic. The urge to do a performance review has been creeping, so I thought I would check in and share things. Beyond a rundown of statuses, there is one bit of wisdom that I want to share and focus on: faith. I mean faith in a multi-dimensional sense.

First, it’s important to contextualize as I’m thinking about how to share this with you. Please allow me to ramble on this subject.

The state of the reported world is, too often, bleak. There are people running about doing bad things for stupid reasons causing all sorts of havoc. It is hard to not be nihilistic. Nihilism and its cynical cousin are the toxic mental traps which will devour you.

I’m not a religious man, but I have seen a burning bush. Instead of buying a religion, I built one based on observations and hacking over the years. Inspired by the phrase “ars gratia artis”, the Latin phrase above MGM’s roaring lion, as a north star; it translates “Art for the sake of art”.

I don’t write code because I want to make all the money in the world (don’t get me wrong, money is nice), I write code because I have no choice. As moth to flame, I’m called to it. For some reason that is beyond me, this activity has chosen me. Whether or not this is a fortune or a curse has yet to be revealed.

See, you are born into a world knowing nothing. Everything that isn’t organic nor stone has been designed, built, and scaled by your fellow humans. Steve Jobs has made this point before.

Life can be much broader once you discover one simple fact: Everything around you that you call life was made up by people that were no smarter than you and you can change it, you can influence it, you can build your own things that other people can use. Once you learn that, you’ll never be the same again

Steve Jobs

In other words, there really is no magic if you study hard.

Now, what Jobs failed to mention is the chasm of knowledge between a newborn and a practitioner in the field. It seems like young people today have tremendous anxiety and depression stemming for far too much knowledge of this chasm. There is always some reason or story to tell yourself to descend into depression.

I have not been immune to these dark mental forces over the years. A common story that creeps up on me is that I feel, often times, that I’m just not smart enough. Sometimes, when the bitterness and envy rise, I was born too late and missed my moment.

The primary tool to overcome such darkness is faith.

1.

The first faith is within yourself. Everyday, you must improve, learn, hack, and do something. The faith is that what you do has some impact. Maybe, the impact isn’t felt by the world, and that’s ok. At the very least, you must have faith that it is ok. The brain, however, is very much so impacted whether you realize it or not.

For years, I had some side project or another which for some reason or another failed or I stopped working on. I’d solve some novel thing, get bored, and move on. It’s easy to start, quit, give up, and fail; finishing is hard.

Genius is 1 percent inspiration and 99 percent perspiration

Edison

Sadly, the world, really, hasn’t felt my personal impact directly in anyway beyond a few people here and there. My prior employers, on the other hand, have felt my impact in a multitude of ways which is why I can now write freely. I’m a good resource, cog, and cog leader for the machine.

A career is a great way to learn how to finish as all worldly impact is derived from finishing. With good leadership, you can learn what it takes to start, operate, and exit a successful endeavor.

2.

The second faith is that if you show up, do things, then what you do in life matters. The hard part of embracing this faith is that you must avoid the path of bitterness. It’s easy for the bitterness and envy to creep up if you compare yourself to those the media talk about.

Status games may have been helpful in small communities, but it appears to be more harmful within an internet connecting billions of people. It’s easy to see your life as some disposable resource for the whatever giant commands the empire you were unwillingly birthed into. Nihilism and cynicism are only moments away to rot and decay your life.

Now, modern enterprises have learned to give people vision and mission that if you work for them or vote their way, then you matter. You join a team and instantly your life matters for the great cause.

This may be true in a cosmic sense.

3.

The third faith is that it is ok that progress is not monotonic in one direction. There will be setbacks, fuck ups, and other things going just plain wrong.

I worked at Facebook when shit hit the fan, and I saw, in real-time, faith evaporate during the cambridge analytica reveal. Suddenly, the mission of “connecting the world” hit an edge case of “connecting the world to assholes”. It was in that moment, that the energy shifted, the air felt different, the tater tots tasted less potatoey, and the company started to become yet another big tech company.

Here, the difficult faith is to realize that progress can swing backwards. We learn things, and that’s the impact. For whatever vile bile people throw at social media, we must agree that such a thing is an inevitability. As a species, we must learn to deal with it.

However, it begs a question at the personal level as the faith gets muddied. At some point, the beginner slides into a practitioner where people can pick up personal missions (like having a family). This transforms the organizational mission and purpose into a personal pragmatic means towards an end.

These personal missions will carry people forward only to a point, and everyone ends up in some kind of mid-life crisis at some point. A common refrain of many successful people is “why the hell did I work this hard for this?” which has the shockingly common response “welcome to the club”.

4.

The fourth faith is that your journey from beginner to practitioner is only the beginning. We are, in many ways, just apes that have learned much while wearing clothes.

The truly hard part of embracing this faith is finding the courage to manifest a good work, but what is good work? I believe good work manifests when you truly think different and build for elevated virtues. We must ask why the pyramids, cathedrals, and other great works of art were created; we can call this a divine purpose. The divine is a source of transcendent understanding of something beyond.

For myself, as a mathematician first and hacker second, I’m drawn to simplicity, elegance, and beauty. These are what I want to manifest in my good work.

1 + 2 + 3 + 4

Have faith in yourself, in what you do, the direction of what you do, and manifest virtue. This sounds good but putting it into practice is exceptionally hard. The tragedy of inequity is a real thing, but this is not a reason to not try.

What I don’t know how to do is help people suffer their burdens and trials. Worse yet, we live in a chaotic time where cures may be worse than their affliction. I have great empathy for people seeking solutions to their woes, and I’m specifically thinking of depression and anxiety. I don’t have faith in modern pharmacology, but I have faith in psychedelics.

It matters not whether the revelations are deep cosmic truths or good ole brain damage, I have seen and felt things. There are two things that I can relay which I believe are important experiences.

First, there is the judgement of god, and remember… I’m not religious, and this is the best way to describe it. The “judgement of god” is to witness yourself without ego and see what you are without the bullshit and lies. The choices that you have made and how good or bad they were. In this state, there is no relativistic morals and there are absolutes. A bad trip which manifested in fantastic outcomes as I’m working out, eating great, and finding the courage to manifest virtues I believe in.

Second, the “church of the engineer”. I called my dad while fucked on shrooms, and while I suspect he was confused and concerned. I went to church of engineering seeing the manifest glory of all the layers of abstraction working together in harmony. Our built world is no accident, and it required us apes to think beyond ourselves and build. It requires trying new things, study, and courage. The challenges of today are simply problems for us to gather requirements for, hack on, and then build.

The transcendent experience is something to trigger deep study. The ways of old start to make a whole lot of sense. The burning bush was definitely a real thing… metaphorical, sure… real none the less. Along this journey, the phrase “traditions are simply solutions to problems we forgot” is one cause for introspection.

As a hacker, builder, or guy that just likes math… the tradition that I find most appealing is that of building a cathedral with an eye towards beauty and perfection: a testament to engineering.

5?

Before I get to the status of my cathedral, a faith that often eludes is the faith in others. There is nothing like working within a team moving in one direction. As of now, I’m building this cathedral alone, and I would like to change that.

If you want to contribute to open source or just have a conversation, then please reach out.

Building the cathedral

As I have been working to repair my body and mind over the last year, I have also worked on Adama. Adama is my good work… my cathedral.

I’m not even close to done yet, and it’s easy to get discouraged. I started writing this post as I was writing to someone about a workshop related to manifesting creativity for a year as I’m coming to a year of doing a similar program. It set the scene for me to illustrate the faiths that I’ve been thinking about, and I figured I should not send a small novel as an email to a stranger.

The status of this cathedral is broken down in a variety of investments, and I just took stock of where I am, and this sets the scene for my personal performance review.

Status: Language

I have invented a new kind of programming language as part of this platform. The language was how I got started, and I have since productionized the language and fixed many bugs. This year has been successful at using it internally as the IDE uses the language, and I major problems as they occur. I have plenty of ideas about how to improve the language.

Status: OK

Status: Service runtime

I have a dynamic runtime where a single machine can multi-tenant host many different “spaces” (digital experiences, apps, games, or what have you). The key here is that I figured out what the service looks like on a single host which I then can scale up to many hosts.

Status: OK

Status: Custom data storage engine

I tried a variety of storage engines like RocksDB, and I found them lacking for my data patterns. So, I wrote my own called Caravan and transitioned away from my MySQL version which increased the scale of a single host tremendously. This came at an increase risk of durability loss within five minutes as I archive warm data to S3 periodically. Only hot data is at risk on a single machine failure, and I have thoughts to rectify this.

Status: Good enough for two-three years

Status: Networking protocol

I started to use gRPC and felt dismay about the number of dependencies it added along with poor streaming performance, so I designed a simpler binary protocol using mutual transport security. As part of this, I wrote a codec to help me serialize and deserialize messages using Netty’s existing library to minimize issues.

Status: Great

Status: Fleet health

Starting with gRPC, I designed a very simple gossip protocol that is efficient. The gossip protocol has every machine knowing the health of every other machine such that failures are known in seconds such that routing around failures occurs quickly. With the migration away from gRPC, I then baked gossip into the core networking layer such that the failure modes of the data plane are then represented by the health plane.

Status: Great

Status: Internal networking client

Using the networking protocol, I created a client such that I can introduce indirection and scale up to many hosts. The network client went through a pure reactive phase until I wrote the custom data storage engine. The complexities of the reactive approach were crushing, so I switched to using the database to map documents to individual hosts which greatly simplified the logic. However, I traded away reliability as single machine failures can cause availability loss for hot and warm data.

Status: Good enough for a few years

Status: Simplified HTTP

I simplified, instrumented, and built out a web server to handle much of the business of talking to the web. This simply adapts Netty to the idioms that I want to speak within the core service, but it also defines the service layers beyond WebSocket.

Status: Good enough, need to monitor for abuse

Status: WebSocket API codegen

Since I’m using streaming as the primary product-data plane, I wrote a simple code generator to help me enforce consistency across the WebSocket API. This allows me to have a single definition for the service which can generate the routing, metrics, validation, request types, and documentation.

Status: Great

Status: SaaS

I used the API codegen to design and seed the API, and I filled in the needed holes. The resulting API fits into the web server, and the API uses a combination of a database client and the internal networking client to implement the initial API. The core service is operating and running right now in production with… minor traffic. This included all the needed bits to authenticate and start adapting the browser to the platform.

Status: Good enough

Status: Ecosystem

The AWS libraries were a complete mess, so I wrote my own library to handle authentication via signature V4. I’m using this for S3, SQS, and SES. It was a nice throwback to when I implemented signature V4 in S3.

Status: Good enough, need to test more failure modes

Status: WebBrowser client

I have a simple wrapper around the browser’s websocket client with appropriate retry mechanisms for networking failure, and I upgraded the API codegen to produce stubs within the generated client.

Status: Meh, error handling needs some love

Status: AdamaTree and RxHTML

Using the infrastructure is fairly simple on its own, but the task of having HTML update can be made easier. RxHTML is a rough web framework for building an SPA with a single file bound to many Adama documents. The IDE uses it today, and the core issue that I’m working on is better error handling.

Status: Needs love around error handing, presentations, and more

Status: Integrated CodeMirror

I upgraded the Adama platform to handle CodeMirror’s OT format such that Adama can power collaborative editing. There may be a few bugs as I need to re-read the code, but it basically works.

Status: OK for now, but the UI needs a lot

Status: Roslin

I started on defining roslin, but it is nowhere ready as I haven’t even rendered anything. The thesis is similar to RxHTML, but instead of producing HTML I am producing an image with a <canvas> element.

Status: Just started

Status: WYSIWYG editor for Roslin

Earlier in the year, I made a prototype game editor. I then decided to change everything, but I got the editor’s core working again and live within the IDE. You can create and move around boxes. I have created a rich plan to upgrade the editor early in 2023.

Status: Not functional yet, but a demo is building up

Status: The IDE

The IDE was built using RxHTML, some custom JS modules, and allows managing 80% of Adama without the terminal. It introduced some management gaps, but the plan is to address those in time. Currently, I’m using a kitchen sink strategy to just throw things together without much thought. Next year, I’ll tighten up how the pieces fit together as I want to try to a new design for ultra-wide monitors.

Status: Meh, it's something that I can show people

Status: Web Hosting

I got important work done to migrate web hosting from Surge to Adama. Adama has some built-in web stuff such that documents can be accessible to the web. For example, an Adama document could be a twitter bot for SMS or a discord bot. The hardest part of this is automating SSL certificates, but you can map a custom domain to Adama via the terminal.

Status: Need to migrate properties

My performance…

Clearly, my performance has been well beyond redefining exceeds. As such, I grant myself a bonus multiplier of 10,000.75! Sadly, I have zero metrics to amplify…

However, the above should speak to a deep faith.

My cathedral will start to make sense in a year or so.

My cathedral will be a testament to simplicity in a few years. Now, the critic may look at the language and deem it unacceptable. For example, I’m trying to make a better Excel/VBA rather than some functional or dynamic thing. The philosophy of simple here requires to extend your view beyond the language and see the forest from the trees.

Take whatever language you want, your next step in today’s crazy world is to put it into a container. That container then needs some kind of infrastructure as a service. You’ll most likely need a database, a messaging service, and other pieces of infrastructure.

It’s true, there is much available to buy, many services to consume; my goal is to not need any of that. “Less is more” is the way to go. Instead, the language and platform represents the entire picture of bringing people together around data. My immediate goals are practical around board games as that’s an area that I enjoy.

However, I have faith that the humble board game is a proxy model for a tremendous number of scenarios.

This is what I’m hacking on.