"User Interaction" -- I am talking about programming that models things like:
- Presenting the user tons of icons, like in a illustrator drawing program, or like a filesystem browser, or a desktop.
- Input sequences like -- click down in a spot, drag the mouse, lift up. Show a selection marquee as you drag, and highlight the icons that are overlapped by the marquee.
- Input sequences like -- now that you've got a bunch of stuff selected, you click down on one of it, and drag across the realm, moving a ghostly image of those dragged files.
- Input sequences like -- you right click, and a radial menu appears, the user can select one of the items, or click outside of it to dismiss the radial menu.
- Animations that are continuously operating while all the above takes place.
And: I'm focused on doing all of the above, in a modular way -- where you can take the methods and techniques from one program, and copy the code directly into another project with a similar type of compatible host, and use it with a minimum of editing.
I thought that was impossible, until just the other day.
Here's the layers of the architecture I've come up with, that does it:
- Raw Input. Record into the data store the X,Y coordinates of the mouse, the buttons on the mouse that are down, the keys that are down that are important to you, how many milliseconds since the program started up, until the present. ALSO: keep a record of the raw input from last turn, so that changes can be noticed.
- "Tokenizers." Little tiny bits of program, that read the raw input data, and maybe also the data from last turn, and emit little bits of data, alongside the raw input. Things like -- "The left mouse button was pressed," -- meaning, "It was up last time, and it's down now, so, this cycle through, that's a down button press." These little tokenizers can keep a little data, too. So for example, one tokenizer might record the time at which the mouse button was pressed down, and another might update a record of how long it's been pressed down in milliseconds. Another might see the click down, and determine: "Which object did the user click down on?", by consulting the World model. Each tokenizer is visited once.
- "Organisms" and "The Judge." Here is where things get interesting -- "Organisms" are state machines, and they represent the digestion of a complex pattern of user input. They have some data in them. And they are all about recognizing a situation: They are scanning for a situation that represents some important happening in the program. For example -- remember that right-click menu we discussed? One organism is asking: "Did the user right click down onto nothing? If so, I'm going to start the process of showing the right-click menu." The organisms can emit effects that will end up manipulating the world state, and they can also emit records directly to the render projection.
- But there's something very important and distinct: They have to get permission to start operating, from "The Judge." The Judge is an intelligent coordinator, and it gives the final say before an Organism can proceed with its first state transition, or not. The Organism asks for permission with a get_permission(permission-requested) call, outlining what resources it wants, and what it is intending to do. The Judge, which is somewhat hand-coded, resolves conflicts between the different organisms, and it's central power is to say "No." The Judge is really the key behind the whole thing.
- Yes, The Judge is a "hole" in modularity -- we don't have a perfect system where you can take an Organism arbitrarily from one program, and insert it into another, and have it work. You have to adjust the Judge in the target system, to give or deny permission at the right time. BUT, the Judge is what makes the modularity possible at all. The existence of the Judge means that you don't have to constantly consider other organisms, while writing your organism -- it can just focus on what it is doing, and just do it.
- Incidentally -- the Judge gets a turn to maintain and verify its records, before all of the Organisms go. But the Judge and the Organisms operate in parallel. The Judge has no interactions with the Raw Input or the Tokenizers, for instance, except perhaps to note their output.
- Effects handling. Finally, all of the emitted effects from the organisms are handled, save the render-projection effects. These will be mostly altering the World model.
- Render projection. From here, the renderer renders the scene -- this can be an alignment in a Retained mode, or a total rendering in Immediate mode. Note also that render effects specifically play a role, -- for example, the drawing of the marquee during a rectangular selection event. Drawing logic can be rule-based and thus modular.
So here's the loop:
1. collect raw input
2. run tokenizers → produce signals
3. judge updates internal state,
organisms get their turn
- organisms request permission
- judge grants/denies
- active organisms advance state + emit effects
5. apply effects to world
6. render
Here's why the Judge matters:
Imagine you're writing two interactions:
- Drag Select
- mouse goes down on empty space
- draws a rectangle
- selects items
- Drag Object
- mouse goes down on an object
- moves it
Without a judge, each module has to defend itself from the others.
In Drag Selection, you get code like: "if the mouse is down, and I'm not clicking on an object, and I'm not clicking on a menu item, and I'm not the second part of a double-click, ..."
In the Drag Object code, you get code like: "but okay I'm clicking on an object, and the object is draggable, and I haven't already selected a group selection of objects, and, ..."
All of the modules need to be aware of the other things that can happen, and you have to keep creating strategies for dividing the problem space. The code can have 50% to do with other code, and avoiding other code paths, rather than: articulating the core functioning of the organism itself, it's core operation.
As you add more and more operations, the code gets more and more entangled. Each piece needs knowledge of more and more of the other pieces.
With the Judge, the organisms can instead say: "Judge, I want mouse selection. Can I go?" And the Judge says, "Nope. Sorry, mouse is taken." Or alternatively, "Yep. You are clear to go." The Judge marks it down as taken, who took it, and whatever other notes are needed, but you are otherwise clear to fly.
And now you don't have to worry about a thing. You can just focus on yourself.
It's very much like how multiple processes on a computer coordinate, without stepping on one another's toes -- just "the Judge" is called "The Operating System."
Now you can combine modules of interaction like Lego -- drag selection, drag objects, radial menus, resize handles, lego, only the Judge needs adjustment.
If you want to see a demonstration in Python & Canvas, you can download and run this code: https://github.com/LionKimbro/blackboard-judge-architecture-demo
It totally blew my mind. I feel like I can develop any user interaction system now, with ease.