I’m probably going to transition this blog into a personal blog and rethink the content strategy for this platform some day. However, I’m helping a handful of people grow as engineers with a few of my clients, and my #1 goal in life is helping other engineers grow into their best self.
The constant advice I give is Always Be Coding (i.e. a cultural reference to “Always Be Closing”). There is no short path towards excellence, and teaching yourself programming in tens years is accurate.
The open question on my plate is whether or not I can help. Is the journey a solo journey? Or, are there ways to grow faster under the guidance of a master? Are you willing to submit? Are you willing to go alone?
I took the solo path for most of my life and had the good fortune of working with excellent management, and I wanted to illustrate the path as I remember many (not all) details of my code and I have maintained my own history. So, I’m going to consolidate it here from my time as a kid until now as a 40-something former big tech principal trying to build a new super platform.
Now, you could skip ahead to the way I distill my story into how I think about helping people grow…
Elementary School (IBM PC)
I wrote a bunch of QBasic because that is what I had! Beyond messing around with mode 13h graphics on an IBM PC powered by an Intel 8088 with fractals and binary trees. I was the kid that could fix Nibbles or Gorilla when someone screwed up the file (and they did because sixth grade was a #wild with awkward IBM machines).
Middle School (IBM PC -> Gateway 486 -> IBM PC)
At some point, I would transition from the IBM PC to a 486 where I would eventually discover the internet.
I started to learn C/C++ using DJGPP and was reading books like Algorithms in C++. Perhaps, I was a strange kid as I asked for computer science books for Christmas, but I was learning algorithms and visualizing results. By default, computers were slow until you bent them to your will.
The most notable project occurred when I was banned from using the Gateway 486 because I screwed around in my English class and got a D. Since I tend to follow rules with extreme precision, I restored the IBM PC in my closet and worked on a game. Paradoxically, not having the better machine allowed me to focus on an actual game. I wrote a tiny text-based dungeon crawler called Avalon since I loved the tale of Excalibur and would call myself “Excalibur Knight” as a handle, so I invented a little story along with game mechanics. This is when I invented algebra.
High School (Gateway 486 -> IBM 200 Mhz)
I was reading books and tinkering with C++ and making tiny game engines, and I was exploring algorithms and how they help the machine behave and bend to my desires.
I took the computer science classes offered at high school and made a complete game of Risk that could be played at the same computer. It was fun as I was learning the Win32 API and MFC.
During this time, I would transition from my hacker handle of “Excaliber Knight” to “ZenGeneral”; this will provide context on the excessive usage of Zen in naming things.
- ZenRisk - My friends and myself were addicted to Risk and Monopoly. We play board games more times than I can remember, so I wrote my own version in Windows for my Computer Programming Course. It worked really well until I started on the AI which was a disaster.
- Zen’s Univeral Proxy - Networking, is the computing equivalent to physics’ magnetism. It amazes me every time I interact with something here and something over there happens. We take advantage of all this Internet stuff today, but it was just coming out when I grew up. I wrote this simple proxy to intercept and enable me to discover what the protocol was actually doing so I could hack together a client or server to do my bidding.
- Counter Strike Cross-Hair - I’ll be honest: I like to change the rules of my games. It annoyed me to no end that Counter-Strike had sucky cross-hairs. So, I made my own. I hacked the windows graphic context (GC) to write pixels directly. I also made a zoom which would put a zoomed version of the context behind the crosshair in the upper right hand screen. It was fairly baller at the time.
- Zen’s Network - I wrote some networking code. Specifically, I wrote a very simple Chess game that used TCP to transmit state changes. I had a very poor understanding of state machines, but I clobbered something together. It worked, and I added chat and started to try to generalize the framework. I failed because networking is very hard.
- zEngine - 3D Graphics were the thing back in the day; the leading edge of computer graphics. I had one of the first “3D accelerators” and I experimented a bunch with OpenGL to make my first terrain engine. I fancy terrains because it is a fairly complex problem when done right. This wasn’t done right, and it demonstrated my need to learn some real algorithms.
- Ant Colonies - My first game using zEngine. A Strategy game around building up an ant empire. It had trees and a crude form of path navigation. If I am being honest, then I wished I pursued this more because I was building a 3D voxel implementation that I still believe is novel when compared to minecraft. The key idea uses marching cubes and a new data structure for representing space as spans of “interval blocks”.
- ZenGame - I started experiments with VBA and building a game engine for board games. I like the simplicity of board games, and I wanted to implement Triopoly. This was a fun C# project as I was able to embed the VBA engine inside of the application allowing the game to be scriptable.
- ZenGameEditor - In working with ZenGame, I made several tools. For instance, I had a tool to design and manage polygons. My first serious attempt at a proof was here when I tried to prove the correctness of my “is this point inside a polygon” algorithm.
- Zen’s 3D Warp Engine - Marching Cubes is among my favorite graphics algorithms. I invented my own data structure for this project. Essentially, I wanted fully destructible terrain. I wanted the ability to add or subtract land and the terrain engine would efficiently update. The kicker was how to evaluate marching cubes efficiently. So, I invented a very simple data structures to simplify the cubic algorithm into a quadratic one (which is the terrain problem). This taught me about permutation and table based programming. This was an evolution of zEngine since Ant Colonies was hard.
- MFC+OpenGL - I love OpenGL, and I loved MFC. This was a easy to use abstraction letting me working with OpenGL inside MFC applications.
- Zen’s Terrain Editor - My first 3D editor. I wrote a simple MFC terrain editor with render. It had various tools to modify and manipulate terrains.
- zModel - Once one has terrain, one needs ants, right? I was annoyed that (a) loading .OBJ files was text and seemed slow. So, I wrote my own binary format along with a tool to convert .OBJ to my .zmc format. Then, I realized I need to animate them and got addicted to writing plugins for Milkshape3D. There’s something special when you write a plugin and see it working in another person’s program; it was a special time in a boy’s life.
- Wealth - A complete text-based game built around the “Acquire” board game. Again, I like board games. So, I wrote a text based version of this game which I thought would be a fun game to play. It was complete with kludgy AI. The game turned out to not be very much fun. I did however learn a bunch about writing text applications
- Zen’s Theatre - Animation was the name of the game. I was learning and experimenting with two things. Making quake 2 models animate, and then figuring out half-life’s mdl files. It’s a special time when a boy creates animated life that can wave and jump around on his command.
- Zen’s Multicast Toolbox - Internet2 was the thing along with IPv6 that was going to storm the internet. The core idea with multicast was that I wanted to have a massive server, and if you can trust the players, then you can use multicast to provide fast positioning. Instead of a server handling n connections to sent n updates; imagine sending one update to “the network” and all the magic would work for you. I made it work on my home LAN, but it didn’t work on the internet.
- Zen’s Resource Engine - Making a game requires a lot of art in a specific format. I made my own version of .tar that was similar to .pak files. Essentially, I could stream data from a giant file or I could override it on disk. It was to enable modding. A fun value-add was I made it easy to replace files on the fly for productivity and then dynamically load on the fly.
- Zen’s Heap - Memory management sucks, so I “invented” reference pointers. OK, I didn’t invent them. But, I thought I did. I made my own heap which could compact itself and what-not. Essentially, you never held onto real pointers. Instead, you held pointers which had pointers to the actual data. The clever thing was making this testable, I would randomly move around memory so bugs would be detected fast.
- Zen’s Overlapped Server - In the failure of multicast, I wrote a windows server to use overlapping IO to use a pool of threads to handle many concurrent connections. I will say this: threads are hard. Sharing threads between requests is even harder. This was a fun experiment where I ported a bunch of my other code over and experimented with concurrency.
- Zen’s Console - How did quake parse and process commands? I found it fun to “hack” quake’s console, so I made my own. I made a very basic programming language which started to get more complicated and ever harder to reason about with special cases. I eventually extended it so that I could have it manipulate config. It’s magic when you type code into your own program and it does things.
- Zen’s File Compare - Given file A and file B; how do you compare them? comparing the file names can be tricky since people can’t spell. So, you have to go deeper and use a check-sum. This was a toolkit for comparing files in strange ways. Essentially, I was trying to determine if two music files were the same. So, it dug deep and used some very primitive heuristics. Then, I scrapped the whole thing to just use a md5 over the file.
- Card Counter - … I was about to graduate high school, and we were going to have a senior celebration party that had black jack. I wanted to learn to count cards, so I wrote a tool to help me. Then I used it online to try to give myself a competitive advantage. It didn’t help, but writing it was fun.
- Zen’s Quad Tree - How do you render many several ants that are on screen versus several thousand ants that are off screen. Playing with OpenGL’s frustrum and culling wasn’t enough to optimize rendering. Instead, you had to not send data to OpenGL which you knew wasn’t going to be shown. I played with a simple BSP to begin with, but it doesn’t scale like a quad tree. Quad trees are fun, and I recommend. They make a good architecture/design problem to make generic enough to be re-usable.
- Zen’s Standard Library - I had a bunch of code that I organized into a library. The crown of this library was a radix sort that was extremely efficient (if you had the extra memory). I was into sorting since sorting solves so many problems if you apply it correctly. Radix sorting is extremely fun, and I highly recommend it. I used it in clever ways. For instance, I’d sort the middle of an array to figure out how to pivot the quick sort.
Job 1: Black & Veatch (during high school)
My first job was at Black &. Veatch where I was a techno guru. This was the first project I took on to implement things that people wanted. Granted, I was also the guy that cleaned out the tech closet (and had to clean out a cubicle of junk for my first office). I also provided technical support on printers, networking, Microsoft Office, and Excel. So, a bunch of odd jobs. The most important thing I produced was an intranet that existed prior to sharepoint. I also made this form where people would sign-in and out of the office; this was a common secretarial task before groupware which enabled office calendars.
This was at the age of 16, I built an intranet application based on a problem my dad was talking about at his office. The secretaries had a paper sheet for engineers to indicate if they are in or out of the office, so I made a digital form of that. The original version was written in C and used CGI. Honestly, the first version was awful because I have zero idea of concurrency and I was using bare bone file IO from the CGI app with no locks (and a bunch of guess work around pointers), so it would randomly crap out. However, my dad took me to a workshop where I would talk with engineers about technology and gave a demo of the app.
This demo then lead to my first job where I re-built the demo into an ASP.Net 1.0 application. Technically, this wasn’t my first job as they couldn’t hire me due to my age (it’s a real engineering firm with actual engineers that build power systems), so I started a sole proprietorship and wrote my own invoicing and tax software to help me
cheat on my taxes (ok, I didn’t make much, but I expensed everything since I reinvested everything back into myself via more books and equipment).
- Document management for distribution of memos and a cache of policies
- Sign in/out sheet
- “SSSSS” - An interface to an “s-curve” depreciation tool that was written in FORTRAN which I wrapped in Access 97 to make it useful. This was a wild ride in dealing with bad data.
- All sorts of tools to help secretaries
- ZenDirPrint - Roseanne was a co-worker at Black &. Veatch, and she had a problem. She wanted to be able to easily print out a directory listing. I made a very simple tool using Visual Studio along with the appropriate .inf to install a context sensitive menu to print the directory. It would simply list the directory, put the result into a temporary html file, then open it in Internet Explorer to print. I was a hero.
- Zen’s Excel Toolkit - I was a mad wizard at Excel. This was a library of Excel macros, functions, and subroutines in VBA that enabled me to do some nice things. I can’t remember what, but I got a bunch of praise for the effort. No one used it though since it was fairly complicated :(.
I went to JCCC since it is a cheap way to get the “weed out” classes done. As a general rule, I think it’s best everyone do a community college for the basics like math, chemistry, and biology. I was well beyond their computer classes since community college programs focus on things I already mastered, and it wouldn’t transfer anyway.
- Stocks! - A friend and myself were interested in predicting the stock market. We were looking into localized patterns. Yes, you can locally predict the stock market to a degree because people are predictable. I designed a stock market search algorithm, and it worked. I could predict various patterns. However… my IP got banned from Yahoo because my connection was pulling tons of information. The search algorithm worked, but wasn’t fast enough to be practical as the market moves very fast.
- File Sharing Accelerator - Napster was the thing, so a friend and I decided to get out in that market. We built File Sharing Accelerator to do just that. It was “File Sharing Accelerator, the next generation of peer 2 peer client software, brings a new level to file sharing over the internet.” It had (a) segmented/chunked download, (b) downloading from multiple sources, (c) automatic download resume, (d) rate control for both upload and download, (e) a file being downloaded will survive disconnects, software crashes, and power outage, (f) file preview/stream, (g) file fragmentation avoidance, (h) fast action queue (WTF did this mean?), (i) a multi-threaded file server.
- Monty Hall 2002! - Do you know the Monty Hall question? Well, this is one that gets people in trouble. The answer is “Switch Doors” because P(Win) = 2/3. Well, the proof sometimes bounces off people. So, I wrote this program to test either choice. It simulated the game and gave the expected result. It messes with people since the code is step by step and perfectly “logical”. Why do people prefer statistics over logical rigor?
Undergraduate degrees (B.S. Mathematics & B.S. Computer Science)
I went to Kansas State University because I did horrible on the ACTs, and I didn’t even take the SATs. I just wanted the paper because that was the expectation. I didn’t have a plan or vision for my life.
However, I went from computer science to both computer science and computer engineering.
Then I dropped computer engineering and added mathematics because mathematics had women in it, and I love me some women!
- ZIMS and MIMS - I wanted a project management system, so I wrote one. It used a blog format with special codes to track project status. It was like tagging before tagging was a thing. ZIMS = Zenerd Information Management System. MIMS = Mathsex.com Information Management System.
- Evolution XP, EvoServer, Evolution 4000 - Revolution, a mac game, was a cute little game. I worked on reverse engineering it to PC along with networking support. I used my prior 3D engine, and I had a nice integration. However, I became more interested in A.I. and Differential Game Theory (which I still study). I also started building a trading engine and network engine (EvoServer) to simulate an economy for a galactic empire. Evolution 4000 was the first C++ engine, and Evolution XP was the next version (which didn’t get finished.. :( ). EvoServer was my first C# server.
- AsyncServer - A C# version of Zen’s Overlapped Server; the same interfaces, but using Microsoft’s .NET platform. It worked very well and was used on a funded project for AIdentity Matrix.
- Open Scene Graph (beta) / GG+ - I redesigned my game engine using Open Scene Graph (OSG), and I implemented a GUI system (Graph GUI plus : GG+). I released GG+, and I got good feedback. However, I got too many demands. I also found that OSG had way too many bugs, and didn’t do things the best way.
- Progressive Mesh - I implemented a progressive mesh algorithm for my Written Communication for Engineers course when I was a sophomore (later, I would be told that it was senior level course… oops). I pwn’d the exiting seniors. I also devised my own texture format.
- LOTR Risk - Using C#, I whipped up a very basic implementation of lord of the rings risk. The novel aspect was a complete rule engine and multi-layered graph engine (cards could be played to go under mountains which changed the graph), and I scanned art assets in from the board game.
- libQuest - This was based on GTA3. I wondered how it did the quests, so I reverse engineer how GTA3 did it using a language based approach. I didn’t write the parser, but I had eval() done. It worked very well, and the fog lifted of how games could be built using data rather than lots of code because code and data are one.
- SliceEm 3D - 3D Modeling is hard, and I just took a topology course. I designed a 3D compiler to take cross sections and stitch them back into 3D models. I designed a troll that looked horrible, and I don’t think it would be adequate for real artists. However, I had some neat ideas. Fun fact, I basically designed algorithms that did the inverse of what 3D printers do.
- Zen’s Time Line - This is my first customized MFC control. It is a drag and drop Time Line editor for scheduling jobs. It was constructed for Character Exhibitor as the primary way of editing time line events.
- Character Exhibitor - Animation is hard, and I suck at animation and 3D modeling. This project was designed to help me provide a mathematical description of animation. Needless to say, it didn’t work. However, I did make some neat code.
- RaGE - This was my first 3D engine designed for multi-threaded processing. I just took the Operating Systems course, and I was very interested in building multi-threaded games. I knew that systems would be dual-core soon enough, and I wanted to be prepared. I was right.
- Cauldron 1.0 - My ultimate 3D rendering engine after RaGE. I took everything I learned in RaGE, made it better. Essentially, I developed a very lean rendering pipeline and figured out everything that makes a modern 3D engine… modern (well for OpenGL and it’s fixed state machine). The solution is to separate out the culling from the marshalling, so you cull your scene down to N objects, then you organize those N objects such that it makes sense. Sorting objects in real-time is very easy.
Incomplete graduate work (with room-mates)
Honestly, I didn’t so much desire to have a normal job, so I stayed in school and became a research assistant in computer science and a graduate teaching assistant in mathematics. My original plan was to get a PhD in Mathematics, then a PhD in Computer Science. Then I was going to solve NP != P by constructing a special topology inspired by galois theory with polynomials derived from a special Turing complete language. I believe that NP != P on the basis that all practical work fits within a machine with a finite life, so you construct a language with just for loops and prove that all P algorithms fit within it. Then, you extend that language with an infinite loop and prove that extension is irreducible and boom. I figured this would take two decades of my life to make progress on seriously.
I love math because it is the essence of elegance and beauty.
I also did a deep dive in my hero, Leonhard Euler, who was unbelievably prolific.
Seriously, there is a project (Opera Omnia) to compile his work which started in 1908 and is still active TODAY.
If there one person that I aspire to surpass, it is Euler. Some that know about him may laugh in jest, but that’s just their bitterness that they don’t have the courage to try. The only way to do it is to be yourself and commit yourself to a life dedicated to a field.
- phpMathML - I was typing several math documents for school, and I didn’t want to learn LaTeX since I can do better. I was happy to see that there were projects working towards MathML. However, all the solutions were client-side. I wanted a server side solution, so I wrote one. It had roughly 60% of work (integrals are amazingly hard) done until I decided to learn LaTeX.
- Bayesian Networks in Java 3.x - Working with Dr. Hsu, I re-architected BNJ by rewriting the inference engine improving the performance exponentially, enable lazy evaluation (so it could compute multi-GB spaces with sub-GB memory at the expense of CPU), and enabling algebraic inference (hence why I needed to learn to build a computer algebra system). I also rebuilt the UI to be usable and enable students to learn how bayesian inference works at an algorithm level. There are some videos of this here: @kddresearch.org.
- Graph Modeling & Editor (GEM) - My first specification compiler of goodness. Most A.I. applications use a graph of sorts. I like graphs. I wrote a generic graph library for BNJ and adapted it to suit a generic needs. I wrote an XML specification to annotate and describe graph rules. The graph rules would be used for the editor to ensure correctness. The editor could compile the code into C++ or Java for serialization. By automating this process, I could easily describe an A.I. domain with a graph and have both an editor and routines in my language to use the graph. It is a bold project… that went no where.
- vec4u - I love fast code. If you want to do vector math fast, then you must do SSE. I wrote a specialized vector math compiler that lets me describe a vector formula in terms of scalars, vectors, matrices, and quaternions. It had a typing system and a C compiler.
- massForge - An attempt to recreate the technology used to create the scenes from lord of the rings.
Graduate work while homeless
My room-mates had left to live their lives, and I was scared.
I was busy and focused on life, and then I became very depressed when my mother attempted suicide.
The feeling is indescribable and a new part of my personality was born to handle this chaos… The WarLord was born to isolate the part of me filled with joy and the part of me that needs to get things done.
It was time to take charge of my life.
I sold all my shit or gave shit away to live an ascetic lifestyle.
Maybe being homeless would suck and I would just fucking kill myself.
Instead, I gained my liberty and a new sense of freedom and I improved on all dimensions.
Man is meant to be free.
You can get caught up in the rat race, or you can say “fuck that shit” and go sit around the quad while all the busy people rush around doing stupid shit.
People are sad because no one has time to smell the flowers because they live in fear of bullshit when all you need is a handful of meals a day.
I slept in the computer science building because I was committed to a life of the mind.
In this state of freedom, I have the privilege to be mentored and shaped by Dr. Burckel with an independent study around set theory. I loved Dr. Burckel for many reasons, but he also helped shape my mind from a chaotic machine into a focused beam of Mathematical destruction with the importance of the precision of words.
Mega Project: Kitchen Suite
I decided that Cauldron 1.0 didn’t do what I wanted since the introduction of geometric shaders broke everything, so I rewrote everything and went for the killer platform. This turned into Kitchen Suite.
- Butcher Block - Once one has a scene graph format, the next thing to do is to build the editor. This was my editor for making and defining scenes in Kitchen
- Fork - my GUI editor which would enable me to design a user interface and the compile it to a single texture. The coolest thing in this was the space packer which not only packed boxes, but packed to preserve transparency so there could be overlap.
- Grill - I wrote my own scriptable VM with my own programming language parser. Essentially, I wrote my own programming language along with my own interpretted byte code.
- Cauldron 2.0 - The mega version of all my graphics engine research and development; it was a massive beast of progress which supported: Geometric Shaders, BSP Tress (for transparent objects), Textures, Shaders, Articulated Skeletons with Animations (successor of zModel), LOD based terrain rendering (SOAR), Texture Atlases, Scene Graphs, an optimize context-switch minimizing rendering pipeline, a re-entrant rendering system (supporting both mirrors and cameras).
- CounterTop - My version of SDL. I do not kid. I was writing linux/windows platforms wrappers so that I could speed a very high level language. I got OpenGL and DirectX 9 working.
- Spoon - was an GUI agent modeler. Essentially, I tried to bake my own reward/penalty graph model. It didn’t work, but it was fun to use. You felt real productive.
- Knife was a set of experiments in computation geometry. Essentially, given a complex environment, it would cut up the open space and turn it into a graph. This enable 3D path finding for very complex AI behavior. I did flow analysis to find choke-points, and it was slow as dirt. However, the output would be a complete map to a static environment which enable one to easily write intelligent looking agents.
- Pantry - was a asset manager/file system/tar/vault. Essentially, you loaded the pantry into memory and you could access your textures, models, scripts, etc. This was a more modern version of Zen’s Resource Engine.
- Math Gladiator Automated Tutoring - I was… depressed when I first taught College Algebra. I wanted my students to have the tools to succeed. I wrote a computer algebra system using C# that could operate over ASP.net; it would show its work and solve problems up to Differential Equations. It had a bunch of neat stuff including the ability to render images of complex forms, and I was working on CLI which would be more interactive. Here is where I started to get depressed with teaching mathematics. It’s stupid to teach something that computers can do better. instead, we should teach the ideas of modeling and how to use computers to solve interesting problems. I also had a “math web server” to render my own form of latex and equation rendering. I invented a form of “math markdown” before “markdown” became a thing.
- Fire/Inferno - XML is ok, but I hate the idea of having text serialization. I love binary serialization. So, I combined them into a unified form. Essentially, development would work in XML and all assets would be stored in XML. So, I created a language which enabled me to describe an object model. The tool would interpret the language and build serialization code for both XML (C# + C) and binary (C only). I could write my tooling in C#, then load the data in C. Then to ship to production, I could convert the data to binary. This idea of automating persistence makes software much easier to write. This powered Kitchen such that I could move exceptionally fast at shipping features by just writing code.
Job 2: BRKZ
I was pulled out of graduate school because a good friend wanted to build a social network, and I was feeling despair because my acceleration in my studies made me see the sadness within Academia. Sayre’s law
Academic politics is the most vicious and bitter form of politics, because the stakes are so low.
I realized that I could help my friend, and it would be an adventure. Life must be an adventure, so I wrapped up teaching my students for the semester and commuted between Manhattan, KS and Overland Park, KS for four months (it’s a 70 minute trip that I would do twice a day). I lived with my friend, and he built a team. That team assembled an apartment for me to live in because I was focused and didn’t want to pay rent because I prefer freedom.
Mega Project: Hurox
This could have been the social network for the next ten years. See Killer Startups. This was a crazy project that we actually made work… technically. It had a web desktop, a virtualized web-os (of my own design), a set of default applications, profile pages, custom user pages, a bulk media uploader, ways of setting price for your content, a very complicated layout system. Oh, and the for those that want to cringe: it only used absolute positioning.
It allowed people to create web properties online using the built-in web editor using a novel layout and construction engine. They could so much and the business was to enable micro-transactions on content. It also has a platform game built in turning this web properties into a Mario like game. It was pretty sweet. However, it was too much for one product… It was kind of a crazy thing executed by five dudes.
Retrospectively, we should have focused and we could have built Gum Road.
Mega Project: Jove3 & Jovian Katana
I made a connected IDE and complete developer experience using C# and a bunch of tools that I bootstrapped from Kitchen Suite. This is how we built Hurox.
- Jovian Katana - A WYSIWYG composer for building responsive applications using a C# inspired layout system. I basically invented a new way to think about layouts such that they could expand both vertically and horizontally and fit the screen like applications rather than documents.
- TableSys - The data system was powered by tables such that tables would be ingested into the web app and then a publish/subscriber would be used to fan them out to keep the entire app consistent to the last write by the same user. Instead of JSON (which I don’t recall knowing about), I simply used a CSV-like format for all read operations.
- AppOS - I invented a micro web OS where apps could have their own context space and be independent from each other. This allowed hurox to have many applications where communication was decoupled.
- Gravity - I built a locking collaborative file system where the IDE would check files out, and then coordinate changes. I didn’t like SVN, and I don’t think git was that popular yet. So, you know, I built my own source control system because #yolo.
- EncodeEngine - I created a Lisp-like template engine to create a HTML encoder system for complex trees to convert complex tables in trees using an L-system. (I didn’t know about JSON or JSON wasn’t popular back them)
- Grill - I was working on a compiled scripting language for Kitchen as well. The novel aspect was it would compile on the fly and then produce C++ code that could be converted for a production experience, and there was a mixed mode where compiled C++ could be mixed with scripting to provide a good developer experience.
- TowerExperiment - I was using my game engine with a friend’s assets to build a tower defense game in my engine using my language.
- My Name is
Jeff Max - I wrote a book
- MassSHARD - Sharding a database sucks, so do it at the beginning. This was my ORM scheme where I defined a schema in a novel language, and it would generate the code to operate that schema and enable me to formally specify how it will shard the data. This enabled me to keep a tight lid on my schema and understand the complexity growth. Since I created my own language, I also overcame many limits of ORM allowing me to write typed SQL queries and then got converted to well typed functions and immune to SQL injection.
- Kapow - Take any spreadsheet, Kapow would convert it to code and make it into a web-app. Bam! Killer enterprisey app. Essentially, I used this to define a very high-level problem which then I’d compile to C. I’d throw the C code into a simplex solver and find the answer to my problems. It was fun, and I used open office (an open office file is just a zip, go, take a peak)
- ArcaneWorks - I like Excel, I like the idea of Access, but I love MySQL. I tried to build an excel/access replacement. Essentially, I wanted to build spreadsheets as closures on top of a database. It seemed like the two could mix together in strange and special ways. Then I imagined combining it with WEKA and doing really powerful things. But, I stopped after making my own Excel like UI. I did however get evaluation working, and this leader to Kapow.
Job 3: Red Nova Labs
After the pain of failing to find funding for Hurox and the distaste of being almost sued, we got adopted by an experienced CEO where we explored many ideas. I was the CTO which… honestly was a bullshit title. We suffered from too many chiefs and not enough indians.
There are some lessons here in that we had an opportunity to sell the technology for a bunch of capital, and that would have been a win for the future. However, we wanted more, so :shrug:
We built a way to find storage units and sold storage owners a web listing. The primary value add was a professional way to drive leads by playing the SEO game.
I architected the system and built many services.
- Caret - a web framework that made vanilla PHP easy to use with modern SEO practices. The key goal was to have a page describe itself which has become standard practice with efforts like Jekyll using yaml as a header for markdown. There is a lot of power in having content and applications describe itself rather than maintaining an index and having to keep things in sync.
- Project K - I killed Kira because developing a new language is hard, and I didn’t know what I was doing. So, I used C# to create a couple of microservices before microservices were a thing. I picked up Kayak HTTP and built services with it and some standard libraries.
- Search - I created a fast in-memory search engine for finding storage units that used a compiled quad tree. It was my first endeavor in exploring eventually consistent products as the tool simply scanned the database every ten minutes and then indexed everything. This was used with google maps to provide real-time geo-queries that was sub 25ms.
- Packer - I created a box packing algorithm in C (for speed) that would help customers visualize their inventory. This also had a dynamic PHP library that cached itself to produce box images on the fly with colors, so an image could given a visualization of the packing.
Job 4. Amazon Web Services
I left Red Nova Labs because I wasn’t happy, and my mother committed suicide which broke me. It just fucking broke me….
It was a long departure and was a root of why I did my best to transition the company to “off the shelf” technology because I was miserable on many dimensions. My ego stepped in as I felt like a big fish in a small pond, so I went to work at Amazon S3. I abandoned my friends to focus on my journey. I became a man and moved my wife from our hometown to Seattle in pursuit of adventure while running from a deep pain.
S3 (SDE II -> SDE III)
Holy shit, it was crushing to my ego. However, I learned that I was not a good nor disciplined engineer. I was cookie dough, and I had just entered the basement of fight club for engineers. Amazon S3 is a serious project of unimaginable scale.
My first year was rough, but then I started to rock the fuck out like a boss. I also found one of the best relationships in my professional life which I’m deeply grateful for.
- Server Side Encryption - I worked on the mainline read/write path for encryption which is non-trivial and led to a painful COE which was “good”
- Lifecycle Management - You know what’s awesome? Building the thing that could destroy an entire business. Deleting from S3 was hard for customers, so we made it easy… at scale.
- Archive to Glacier - Once you have a system to delete data, you can also move that data to a cold system. Yay! I’m proud of this work because I created a “file format” that supported terabytes and I had the library tested at 100% code coverage.
- CORS support - I worked across project management along with a junior engineer to ship CORS support for the S3 Website feature.
- Website Redirect - Another project for a junior engineer to ship a simple feature where website could issue a redirect.
- Website routing rules and root domain - I designed a system for another engineer to build where the goal was allow redirects en masse and also enable root domain (i.e. jeffrey.io) support.
- IronGiant - I wrote an acceptance test framework along with thousands of tests that were parameterized. This involved custom JUnit runners and a whole new running framework such that tests could be 100% reliable. This is where I evolved the philosophy if an API is too hard to test, then how on earth do we expect our customers to use it?
- DFDD - I took over DFDD which was used for fleet membership and health checking across the entire litany of services that made up S3. At the time, management was worried about it falling over so I worked at refactoring it, cleaning up the heap, optimizing it, achieved 100% unit test coverage, and improving gossip speeds by 128x using in-rack propagation and side-channel coughs. This is the service that when it fails, the internet fails. It was awesome like riding a nuke!
- Melody - So, I had this idea about writing a visualizer for sheet music. I was studying piano, and I’ve always had a tough time reading the music in real-time. The goal of this project was to understand the xml schema for understanding music, and I got that mostly. Ideally, I wanted two integrations. First, print sheet music out with color. I could then color code the keyboard and improve accuracy. Second, write an app to play the sheet music with the right timing. However, I learned how to read sheet music better due to this undertaking and this drifted off with my interest in piano.
- World Bootstrap - A generic 2D vector editor. World Bootstrap is a computer aided design program with an emphasis on extensibility, open data, and ease of use. The goal of World Bootstrap is to make it easy to: design a game level or world, plan a garden, build a quick software diagram (Visio style), design the layout of a room, make a photo collage.
Two month interlude between AWS and Facebook
Nearly four years at AWS is like 28 years at any other place, so I took a little break to recharge and scratch a few itches.
- PixelMaster - I wanted to finger paint pixel art for lewt game using my tablet. So, I made this as a starting point. I need better icons, but it mostly works on my Kindle. I had some interesting problems with garbage collection since I was using C#.
- lewt - I fell in love with the concept of the Dungeon and Dragon board games like Wrath of Ashardalon that I started to design a map and player system to feel similar. I had very basic exploration and combat figured out.
- OldSkool - I was build an old school game engine for just 2D sprites for lewt and had graphics, joystick, and mapping.
- Play - I was making a new kind of JSON stream database. The idea was to have a fairly rich language of how to manipulate a JSON object. The rich language was to describe transactions against an object, and the mutations to the object would be serialized and synchronized to multiple clients. This could be used to make a fairly expressive game server and shared environment. I ran into a few performance bottlenecks as I was re-teaching myself C++, but I liked the overall direction. I wrote a custom network protocol that was inspired on Redis..
I left AWS because I had a sense of closure, and I shipped a lot of code beyond the main features. I took deep advantage of AWS and read thousands of COE reports spanning many services. Look, you can find academic papers for ideas, but you need COE reports for how ideas go wrong and shit goes south.
I wasn’t cookie dough anymore, and I was a bad mother fucker that was beloved. My team cried when I left, and that… hurt. Ultimately, I wanted more to build my own thing, but that’s exceptionally hard. So, I went to Facebook to get new product-related experience and re-roll the dice.
- Improve web chat reliability - Measuring reliability is seriously hard and non-trivial because you have to remember that it’s not about 9s and instead is about product experience. Real-time systems with pub/sub is seriously hard to fathom at massive scale. I took over the service and help push it forward over years until I killed the fucking thing with StarGate (see below).
- Pylon - a radical simplification of how pub/sub worked based on various ideas from a personal project designed around “super streaming consensus”. This was actually rooted in many of the idioms I invented while on break within Play but adapted to use Facebook’s Thrift and modern C++. The key idea at play is to replicate a data structure that is easily differentiable like a map of keys to immutable values (i.e. machine addresses) and then union them up on read such that progress is made based on in-region addresses. This meant that people in the same region would get publishes super fast while global subscribers would get it in good time.
- Nexus - a big multi-year refactoring of all real-time systems into on cohesive platform which improved reliability and increased scale at a much lower cost. This service became the foundation for Facebook’s Live offering since TAO (the graph engine) wasn’t up to the task. See my SRE talk.
- Request Stream - I designed a protocol and got a patent which allowed me to trap engineers and call out bullshit. The protocol provided a mechanism to build reliable real-time systems that is based on multiple transports and various important signals. See the patent. My philosophy here is that “timeouts are a bullshit mechanism for shitty software”, and you have to think radically different with streams. A stream across multiple machines behaves like a conveyor built and you need E2E signals to detect breakage of any piece since “a broken stream feels like an inactive” stream.
- StarGate - a cross-org project to simplify how devices connected to real-time infrastructure using Request Stream. MQTT sucks ass. This powers how billions of devices talk to Facebook.
Each of these projects finished, and I created the leadership to drive a handful of ideas and agendas forward. My last year was a bit rough because I was searching for a new direction, and that can be exceptionally tough for senior engineers that are creative with radical ideas… So, I left to re-roll as an entrepreneur.
- FarmCron - I had some goats and 2 acres on an island, so I wrote some management software that did payroll for a handful of household employees using schedule H. Yes, I wrote my own payroll system for three people and did my own taxes.
- Wake -
The Now (2024! The middle of my journey and 30+ years of understanding)
So, here I am building Adama, and Adama is big. So, let’s take inventory!
Inventory of Adama
- Runtime - The language is very different and has a lot of customized data containers that are reactive and differentiable. At core, the language targets creating a specialized implementation of a “LivingDocument” and many roads end at the LivingDocument.
- Core Service - The core service is responsible for the routing and management of multiple documents within a single host, and this also provides billing among other things. It bridges the gap from the living document to the data store.
- RxHTML - Differentiable trees are super powerful, and RxHTML is a a way of describing an HTML template that sits on top of a differentiable tree such that a minimal data change manifests into a minimal DOM change.
- libadama.js - The runtime for managing differentiable trees, connections, and the glorious RxHTML op-code library.
- Stdlib - Every language/platform needs a robust library to provide things out of the box.
- Caravan - I’m creating a specialized logger designed around NVMe to saturate NVMe at low cost with multiple keys. I’m using a persisted heap
- devbox - The developer experience is paramount, and the developer box is designed to provide a near production like experience. This is how to develop locally with Adama.
- net - I’ve designed a very simple codec generator to tear-down and build-up messages between Java and TCP. Beyond just message passing via stream channels, I also have load balancing and heat exchange. There is a bunch of work to do with respect to capacity planning and how host selection works, but it’s in an OK state now.
- clikit - An intern that I hired help simplify my command line parsing, help, and routing such that I could stop writing manual code and instead use XML to dispatch commands.
- saas - The software as a service where Adama’s core and associated control plane are exposed as a WebSocket service.
- sassclient - As part of how I think about multi-region support, I have the apikit define and generate a client that lets Adama talk to itself using the public endpoints via WebSocket.
- apikit - An XML specification based code generator for defining the API within saas. Beyond defining methods, it provides a way of thinking about policies, metrics, validation, logging, and unifies the frontdoor for security concerns.
- common - a whole bunch of generic stuff for building the platform that doesn’t really fit into one place except “common”. For example, a general asynchronous caching library for how I do async callbacks.
- data-mysql - I’m using MySQL as the control plane for everything that feeds the service. MySQL will scale appropriately for this need, and there isn’t much reason to get fancy since the service is already complex enough. The database is used for storing users, spaces, document index, authorities, domain mappings, etc. There will be scaling issues, but caching is the key here for the heat.
- web - A simplified web framework for working with HTTP that is deeply integrated with the service rather than “yet another web framework” or “yet another HTTP framework”. The purpose of web is to provide simple abstractions such that I can move from HTTP/1.1 to HTTP/3 once I have time to investigate and explore modern protocols.
- services - Adama has a language to transorm state, so why not connect that state to other services like Stripe or Discord? The long term vision is to integrate a handful of awesome first party services (where I have an opinion on the service) and third party services (generic HTTP services).
- overlord - The “other” service for doing global actions like storage inventory for billing.
- data-train - I’ve started to think and scheme out how I make the service highly available and durable with Raft
- control - The tooling I use to manage EC2.
This entire blog is dedicated towards talking about Adama, and I have two years of posts along with entire book that I’m working on.
At this time, I have two clients which I’m working with on my platform. Until they launch fully with public properties, I’ll keep them secret.
Offshoots and side projects
- headwindcss - I love tailwind! It’s just so great, so I started a tool to start reversing arbitrary pages (using computed style) into TailwindCSS, and I hired my intern to complete it.
- webed - The start of a HTML editor that is live using Adama. This is scheduled for resurrection at some point.
- roslin - A rust-based WYSWIWG editor such that I could compose board games using rectangles. I had a layout system and composer tool where entire applications could be built with basis components. The runtime was such that a board game could be designed while being played since the state breaks down into server-side state and view-state which could be serialized.
- satisfactory-plannnnnner I wrote a tool to help digest how to plan and think about the scale of playing Satisfactory.
- edhtml - A failed meta layer on top of RxHTML where I combine the reflection from Adama and L-System rules to make forms and other such things. This turned out too complex to explain, and it is not compatible with product design.
- card-compose - A quick java tool to compose cards for a clue/alibi game. I intend to turn this into a feature of Adama and the transform capability.
- Murder Most Foul - I used AI to generate characters for a clue-style game based on the players for a game with a few novel rules. For example, beyond the “make an accusation” rule which I extend to pick either clock-wise or counter clock-wise, I added “ask the group a question with a yes/no question” and “ask any player a question which results in a number”
- AoE4Coach - I wrote a screen reader for Age of Empires 4 which would tell me when things were done producing, so I could react to the event rather than check on it. It was limited to scrapping and eventing, but I stopped playing for other reasons.
Applying my story to you
At core, I believe Life is a journey, and you should set out to tell the best fucking story you can.
I also believe in working every day, so I want to boil down a handful of lessons.
#1. Always be doing
You will get tired, and you can take breaks here and there.
However, you need to be doing.
Life is about action, and I love writing code.
All of life’s rewards and glory go to those that exercise persistence over long periods of time.
When I reflect on my past, I’ve given up on many projects.
Either I failed, got way out of my depth, or got bored. BUT, I never gave up on myself.
I used to feel shame about all my failures, but they were just exercises to get me where I am today.
No one gets it right the first time.
There will be pain.
There will be tears.
Things will be hard.
Just keep at it.
#2. Nurture your creativity
Have ideas and play with them.
Play with existing ideas.
From all the people I have met, creativity is the rarest attribute.
I suspect it gets beat out of people.
Or people buy into the left/right brain instead of embracing a whole brain model.
Perhaps it is just a fluke of nature to see more than just a blank sheet of paper.
Be fearless in your creativity and try things.
Just try things and see what happens.
So what if you fail or copy someone else?
All you need in life is a few good wins.
#3. Search for patterns, keep a log
The things you enjoy become patterns, so keep a history of what you’ve done.
Journal, take notes, archive, or whatever.
Take the time to reflect on your past.
Focus on the things you do based on intrinsic desire.
If you allow yourself to only focus on extrinsic demands, then it will hollow out the human soul over time.
It is shockingly easy for people to turn themselves into resources… toys for the powerful… cattle for the slaughter.
Your history also present the opportunity to change your story, to refine it, to have a pivot.
#4. Don’t listen to negative people
Look, this post is probably going to trigger someone about me being a white cis-gendered heterosexual man that grew up with privilege and blah-blah-blah-blah-blah.
I’ll admit that I grew up in a good area with a strong middle class, but this is a useless commentary born from bitterness and resentment.
The people that make this comment don’t realize that the group I belong to is vast, and I can take joy from the fact that I did my best and didn’t squander my opportunities like so many.
|The point of privilege is to leverage it to do good work.
I don’t let people tear me down; instead, I try to lift people up.
I have taught many engineers over my career, boosted them, and lended my powers to their aide.
I have saved careers and inspired people to do good work, so I will never let anyone reduce me or my accomplishments to some limited group identity concept.
You shouldn’t embrace that philosophy and instead seek to maximize your potential.
Ignore the haters and focus on those that aim high.
#5. Only listen to doers.
The world is filled with people, so focus your limited attention on those that do things. Listen what they say. Contextualize what they say in their life. Understand how they turn thought into action.
|“It is not the critic who counts: not the man who points out how the strong man stumbles or where the doer of deeds could have done better. The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood, who strives valiantly, who errs and comes up short again and again, because there is no effort without error or shortcoming, but who knows the great enthusiasms, the great devotions, who spends himself in a worthy cause; who, at the best, knows, in the end, the triumph of high achievement, and who, at the worst, if he fails, at least he fails while daring greatly, so that his place shall never be with those cold and timid souls who knew neither victory nor defeat.”