Introducing volatile data

For those keeping track, I have a manifesto on user interface architecture where I outline how I intend to build a minimal UI kit on top of Adama. I’ve had a few starts of it, but I was getting distracted by the lack of polish on Adama. Now that Adama is stable where the primary critique is a combination of performance and scale, I’m paging in the second leg of my strategy. One thing that is on my mind is the role of volatile data.

So, what is volatile data? I’m not talking about the keyword. I’m thinking more about state which doesn’t come from the server nor is a property of the user’s view (i.e. scroll bar position, page #, etc…). This is the mental diagram.

The context of this new volatile source of data for the UI to deterministically transform into a picture is animation. Ultimately, since I’m using JSON trees for both the data from adama and the view state, this new source should also be JSON. This frames the discussion as I only have two data sources to generate volatile data from beyond time.

As per the figure, I don’t have exact answers. I’m thinking this through as I write.

Recap: What is animation?

Animation fundamentally isn’t anything special. It’s basically mapping some properties of an item over time. The key user challenge in animation software is how to express these functions in a user friendly way. Sadly, I can’t reuse my 20 year old MFC component for timeline editing which others have fixed. I’m starting from scratch on this one.

The first challenge then is to create a language to define functions. The key requirement for this language is that it has a graphical way of editing. I believe that if you start with a graphical way to edit, then it will be messy. So, we will also add the requirement that this language be easy to edit by hand by hackers and control freaks.

Since I’m thinking loudly, we can start with a simple object.

{ "x":0, "y":0, "angle":0 }

is a nice little object definition of the state behind a object in a scene that could translate and rotate. We can use some hackery to augment this. Here is a thought, let’s embed the language within the JSON.

{ "x":0, "y":0, "angle":0,
  "@": [{
      "op":"lerp",
      "var":"x",
      "start":1,
      "end":5,
      "speed": 0.1
  }]
}

So, here, the intent it to vary “x” from 1 to 5 at a speed of 0.1 units per second. But, there is a problem. First, when do we start this animation? Do we reset it ever, and when? Do we loop forever? How do we synchronize looping with other inflight changes? So, not only do we need to encode the actual transformation of the values, but we need to encode conditions for when to transform.

Conditions

So, within our transform language, we need the ability to gate and control the transformation. Let’s sketch a potential skeleton.

{ "x":0, "y":0, "angle":0,
  "@": [{
      "condition": {
        "data":"/path/to/variable",
        "change":"behavior",
        "rising":"behavior",
        "falling":"behavior",
        "==": {
            "value":1,
            "behavior":"behavior",
            "when":"gain",
        }
      },
      "op":"lerp",
      "var":"x",
      "start":1,
      "end":5,
      "speed": 0.1
  }]
}

A condition first needs to select some data to monitor. Here, we are assuming that we will monitor a single variable which we can find via a path. There is an open question of how complex the pathing language should be. Should this be an extension of the local state? Should it be able to recurse into children? Should it use fully fledged JSONPath. We can put a pin in that question, and let’s assume that we can operate on single variables that are local to an object being animated. For example, the local object could have a boolean called “active”.

      "condition": {
        "data":"active",
        "rising":"play",
        "falling":"pause"
      },

Here, we key off the boolean, and then using signal edge terminology: if active transitions from false to true, then we execute behavior “play” which starts the animation. And, when the boolean drops down to false, then we pause the animation. Since there is no notion of looping, we augment the owning animation with a “loop”:”repeat”.

{ "x":0, "y":0, "angle":0,
  "@": [{
      "condition": {
        "data":"active",
        "rising":"play",
        "falling":"pause",
      },
      "op":"lerp",
      "var":"x",
      "start":1,
      "end":5,
      "speed": 0.1,
      "loop": "repeat",
  }]
}

So far, this is shaping up as something fun to play with. Before we iterate and build up the corpus of transforms, we need to address if the above has a graphical editing representation.

GUI or Bust

We can separate the GUI problem into two: the behavioral properties (i.e. condition, op, etc…) versus the 2D graphical properties (start, end, var). Here, we see that this format is shaping up to be a bit awkward, but it can be fixed. First, if we have a WYSIWYG editor, then we could drop a “Timeline” component on the canvas. We could click this component or select it from a list to lock it in place. Somehow, we transition the editor into a mode to manipulate the timeline. The good news is that the is that a bulk of the properties are not difficult, and the challenge is building up the transform aspect. Here, the “op” would be “timeline” as a transform.

{ "x":0, "y":0, "angle":0,
  "@": [{
      "condition": { },
      "op":"timeline",
      "timeline": [
          {
           "x":1,
           "y":1,
           "angle":0,
           "time":0
          },
          {
           "x":5,
           "y":1,
           "angle":0,
           "time":50
          },
      ],
      "loop": "repeat",
  }]
}

Simple linear interpolation would transition the values over time, and the editor could express where those points are by having the user control a timeframe and reposition the object. There are a variety of ways to smooth out the animation as pure linear can be abrupt, so this requires some work and tooling to clean up. So, I feel like this direction is good from a traditional sense.

Not so traditional

However, this engine is shaping up to handle a few other scenarios. For instance, we can reuse the condition language to also control playing audio. That’s nice. However, we can break out of the traditional timeline and think about position pieces.

{ "x":0, "y":0, "angle":0,
  "@": [{
      "condition": {
        "data":"location",
        "==": {
            "value":1,
            "behavior":"set",
            "when":"gain",
        }
      },
      "op":"goal",
      "goal": {
          "x": 40, 
          "y": 80
      },
      "speed": 1.0,
  }]
}

This language allows specifying how to move pieces on a 2D board when identified by location ids rather than coordinates. This solves a pressing problem for some games that don’t exist on grids. So far, this language is feeling like a nice thing, so let’s spec it out to taste it fully.

Iterating out a spec

Picking a value source: We have two sources of values: data and view.

{ "data":"...path..." }

and

{ "view":"...path..." }

Finding values to monitor: Within a single UI card, we will bias towards local state only or parent state. We need a simple language to find the data to monitor.

example	behavior
“x”	get the value of the field “x” in the currently bound object
”../x”	get the value of the field “x” in the bound object’s parent
“/x	get the value of the root field “x”
“child/x”	get the value of the root field “x” within a fixed child object.

It should be noted that there is no notion of arity with respect to children being arrays. The reason being is that considering an array would violate a single data item to monitor.

Signals: Once we have data to monitor, we then need signals to fire off.

name	value	what it means
failing	behavior string/array	fires when the witness value decreases (true to false, 4 to 3)
raising	behavior string/array	fires when the witness value increases (false to true, 3, to 4)
change	behavior string/array	fires when the witness value changes in any direction
”<”, “<=”, “==”, “>=”, “>”	object	see below

The comparison operators allows building very precise tables/ranges.

field	type	what it means
value	number/boolean	the value we are looking for
when	gain/lost	when do we fire the event; either when we first witnessed the value or when we had the value and lost it
behavior	behavior string/array	what happens when we fire

Behaviors: Once we have signals, we need behaviors to execute.

name	what it means
nothing	do nothing (the default)
play	resume the transform
pause	pause the transform
reset	reset the transform
reverse	reverse the direction of the animation

These behaviors can be placed in composition by using an array

{ 
  "condition": {
    "data":"active",
    "rising":["play", "reset"],
  }
}

It is an open question if there should be a way to combine conditions with various logic operators, but we push back on that since the underlying data source could perform that logic better. Here, we phrase our thinking to be focused on taking the data given and filling in the gaps. This begs a question of the various transforms

Transforms: Here, we have a fun game of how many ways are there to construct a function. We will start simple

name	what it does
lerp	a simple linear animation with a start position, end position
goal	move the object from its current location to the new position
timeline	a robust list of keyframes allowing complex pathing
drag	use a timeline to drag an object from where ever it is to a new goal

What’s next

So, I’m going to keep this on the back-burner for a week or so. One thing to contend with is how to introduce multiplicity. The above language works great for manipulating a single object, but how do I have five objects with the same script? Do I create an animation layout controller? What new signals does that introduce (“create”, “delete”)? What signals did I miss?

I’ve written down my thoughts as they happened and shaped them. This may be complete nonsense, but writing is thinking, so here we are stuck in field. My core task at the moment is to take inventory of my existing editors and tools to see what assets and liabilities I have.

This notion of building a graphical editor has perils of scoop creep.

The future is fun.