The Machinery - Part 3

2020-07-16

Finally! Code only this time (probably). The Machinery has a number of great samples and gives you a great little component template to get started with. It's jam packed with stuff! It's entirely written in C!

Folks that know me probably expect me to just hop right into Rust. But not yet! Not until I understand a great deal more about The Machinery. But aside from that I really want to get out of my head a bit and sometimes the best way to do that is to just take a break from your usual patterns and habits.

The Machinery can be extended in a couple of ways. You can create custom components for the ECS (or is it an ECE), you can create new editor tags/interfaces, and you can create nodes for the graph component. You can do quite a bit with these extension points... you can do pretty much anything. Custom data, editors, and gameplay systems all kick off from there. The Machinery is sort of wild.

But today is not the day to figure out all the ways to do very subversive and wild systems engineering. Today is all about getting our vocab built-up and coming to an understanding the fundamentals.

First Steps

Q: What makes a plugin to The Machinery?

A: Any dll/so/dylib, whose name begins with tm_, that exports the function:

void tm_load_plugin(struct tm_api_registry_api *reg, bool load)

The first thing provided to a plugin is a pointer to the registry api. According to the docs the registry api provides a "Global registry that keeps track of loaded APIs and interface implementations."

So out of the gate we have a distinction between an "API" and an "Interface". Helpfully the docs immediately clarify the distinction between the two.

An API is something with a single implementation. An Interface is something with potentially multiple implementations. Makes sense.

The registry api includes functions (it's rather hard to stop myself from writing 'method' everywhere) to get and set an api implementation, and it has functions to add_implementation, remove_implementation, and get a list of implementations.

Getting a list of implementations has that fun C style interface of a pointer to pointer ** and you'll be getting the number of implementations via an out arg *count. These sorts of interfaces to systems are so easy and familiar... but I'm still wary when I see it. You have to be so careful and consistent with marking things const that aren't out arguments. It's maybe less efficient or possibly just annoying, but I'd have preferred a struct with the count and implementations to just be returned.

Nobody writes C that way of course, so it wouldn't make sense for the folks at Our Machinery to suddenly start doing things like that I suppose. Also.. yeah, if I consider it more it'd be annoying to have structs for the result of every function like this. C++ has templates for this. C could just use void* but then you'd be casting for every use and boy-howdy that's some line noise.

Ok, so that's an intro to the api_registry and what we're expected to use it for but what's up with that bool load?

Based on the following two macros:

// Convenience macro that either sets or remove an api based on value of `load` flag.
#define tm_set_or_remove_api(reg, load, name, ptr) \
    if (load)                                      \
        reg->set(name, ptr, sizeof(*ptr));         \
    else                                           \
        reg->remove(ptr)

// Convenience macro that either adds or removes an implementation based on value of `load` flag.
#define tm_add_or_remove_implementation(reg, load, name, ptr) \
    (load ? reg->add_implementation : reg->remove_implementation)(name, ptr)

I'm lead to believe that this indicates whether the plugin is being loaded or unloaded. I didn't yet see anything in the UI that would specifically unload a plugin, but The Machinery can be run with a --hot-reload flag. Unless I'm mistaken this facility works by renaming the plugin dll before loading it, so that when you rebuild a plugin it can be copied again and loaded. The previous loaded code would be then responsible for unregistering it's component interfaces, apis, and other code. Here is a relevant blog post on the matter.

---

So now it's clear that in order to be a plugin you need to define (and export) the expected function in your library and use the provide api_registry API to get access to any API that you require for your plugin, as well as register anything your plugin provides.

Something I didn't expect, because I'm rather biased and trained to think in certain ways, was that if your plugin needs to use specific APIs you need to stick them someplace... but where? Well, it's C, so static global state apparently. :D In C, a static global is only accessible to the compilation unit it's declared in, so you'd want to initialize any static global pointers in the tm_load_plugin function.

To illustrate how this works in The Machinery we can just look at how the example component we're reviewing gets a pointer to the entity API:

static struct tm_entity_api *tm_entity_api;


TM_DLL_EXPORT void tm_load_plugin(struct tm_api_registry_api *reg, bool load)
{
    tm_entity_api = reg->get(TM_ENTITY_API_NAME);
}

One final interesting thing to note is that the api_registry will always return to you a pointer to an api, but it's associated functions may be NULL until whatever plugin provides them has been loaded. In addition to this, if a plugin that provides the api implementation is reloaded, your static global pointer will have been updated for you.

At this point it is already an interesting thought process to imagine what a Rust version of the this would look like. None of this is possible to do without using unsafe, although the lazy_static crate might be able to help.

A Simple Component

Components are just structs in The Machinery. Nice and simple. The example component is declared as:

struct tm_sprite_component_t
{
    float y0;
    float frequency;
    float amplitude;
};

It's called "sprite component" because I ultimately plan to make it a 3D sprite component.

Nice and simple but certainly will require some wiring before we can "attach" it to an entity and have any property values serialized to disk with the scene.

Introducing our component data to The Truth

To get started we must look to The Truth.

We need a way to describe our component and its properties as well as give it a "type". In order to give your object/component a type we have to decide on a type name (and calculate it's hash apparently). The type name should be prefixed with tm_ I think, and from what I've seen it generally just matches the struct name.

The example code uses the preprocessor to define:

#define TM_TT_TYPE__SPRITE_COMPONENT "tm_sprite_component"
#define TM_TT_TYPE_HASH__SPRITE_COMPONENT TM_STATIC_HASH("tm_sprite_component", 0x788c0feb4cc58af4ULL)

The type name is expected to be a static string, and the useful macro TM_STATIC_HASH acts as a marker for a preprocess step, a useful tool named hash.exe that can generate hashes for you, or update them in the file if they don't match. (this tool should probably be renamed as tmhash.exe right?)

Any properties that should be known to The Truth need to have IDs and C enums would appear to be the tool in charge of that:

enum {
    TM_TT_PROP__SPRITE_COMPONENT__FREQUENCY, // float
    TM_TT_PROP__SPRITE_COMPONENT__AMPLITUDE, // float
};

Properties also have types, and The Truth has a fixed set of options here that seem to cover every useful possibility "none", bool, uint32, uint64, float, double, string, "buffer", "reference", "subobject", "reference set", "subobject set".

Here is how we describe our components properties:

tm_the_truth_property_definition_t sprite_component_properties[] = {
    [TM_TT_PROP__SPRITE_COMPONENT__FREQUENCY] = { "frequency", TM_THE_TRUTH_PROPERTY_TYPE_FLOAT },
    [TM_TT_PROP__SPRITE_COMPONENT__AMPLITUDE] = { "amplitude", TM_THE_TRUTH_PROPERTY_TYPE_FLOAT },
};

---

So with the requisite metadata in hand we're ready to take the remaining couple of steps to introduce our component to The Machinery by creating the type in the The Truth:

const uint64_t sprite_component_type = tm_the_truth_api->create_object_type(
    tt, TM_TT_TYPE__SPRITE_COMPONENT,
    sprite_component_properties,
    TM_ARRAY_COUNT(sprite_component_properties));

In certain cases you might want to have your component defaulted to some value other than a zero-initialized struct. In these cases you can create an instance of your component and set it as the default that will be cloned whenever this component is added to an entity:

const uint64_t default_object = tm_the_truth_api->quick_create_object(
    tt, TM_TT_TYPE_HASH__SPRITE_COMPONENT,
    TM_TT_PROP__SPRITE_COMPONENT__FREQUENCY, 1.0f,
    TM_TT_PROP__SPRITE_COMPONENT__AMPLITUDE, 1.0f,
    -1);
tm_the_truth_api->set_default_object(tt, sprite_component_type, default_object);

In general you're discouraged from doing this. Zero-initialized data is the mantra in The Machinery.

Hard not to stop for a second and pull on a thread here. The quick_create_object function is a combination of create_object and quick_set_properties. It uses varargs to essentially call a more specialized property setter. But how on earth does this translate to updating field members on a struct?

It doesn't because The Truth deals with "objects" and "properties", and provides facilities to get read pointers and write pointers to objects which let you get and set property values on objects in a thread-safe way. The type and value you're setting up don't deal with our component struct directly, but let's put a pin in this trail of thought for now and quickly mention "aspects" in The Truth.

---

Aspects in The Truth are an interface that provide specific functionalities to object types. This happens to be the extension point which gives us the ability to have a standard editor ui for interaction with our component if desired. Things like custom gizmos in the viewport can be setup this way if I understand correctly.

I don't see it used anywhere else in the example component, but it does setup an editor ui aspect for the component with a completely default implementation for the tm_ci_editor_ui_i interface:

/* static global handle to the interface */
static tm_ci_editor_ui_i *editor_aspect = &(tm_ci_editor_ui_i) { 0 };

/* defined in our type creation function */
tm_the_truth_api->set_aspect(tt, sprite_component_type, TM_CI_EDITOR_UI, editor_aspect);

Looking at the docs for this interface this should mark the component as enabled in the editor with no special handling or representation in the viewports or editor gizmos.

---

All of the previous setup was handled in a short function static void truth__create_types(struct tm_the_truth_o *tt). A pointer to this function is given to the truth as an implementation of the tm_the_truth_create_types_i interface (which is just a single function pointer):

tm_add_or_remove_implementation(reg, load, TM_THE_TRUTH_CREATE_TYPES_INTERFACE_NAME, truth__create_types);

Gotta wonder now, how is it handled that a type fundamentally changes when you reload a plugin? Does The Truth keep track of this somehow?

Setting up our component for use with entities

So, our component is a struct that we want to store on an entity, but it's key data is also kept in The Truth. Let's look first at how we can make it possible to associate our struct with an entity.

The entity api defines a specific interface for creating components tm_entity_create_component_i and we setup an implementation of this interface as such:

tm_add_or_remove_implementation(reg, load, TM_ENTITY_CREATE_COMPONENT_INTERFACE_NAME, component__create);

In our function component__create we add an implementation of the tm_component_i interface and register it with the entity context:

tm_component_i component = {
    .name = TM_TT_TYPE__SPRITE_COMPONENT,
    .bytes = sizeof(struct tm_sprite_component_t),
    .load_asset = component__load_asset,
};

tm_entity_api->register_component(ctx, &component);

Our implementation of tm_component_i provides probably the absolute minimum. The name and bytes fields appear to be obvious so what is the load_asset function responsible for? According to it's docs this function is responsible for loading data from The Truth. We assign a pointer to our function component__load_asset which is implemented as:

static void component__load_asset(
    tm_component_manager_o *man,
    tm_entity_t e,
    void *c_vp,
    const tm_the_truth_o *tt,
    uint64_t asset)
{
    struct tm_sprite_component_t *c = c_vp;
    const tm_the_truth_object_o *asset_r = tm_tt_read(tt, asset);
    c->y0 = 0;
    c->frequency = tm_the_truth_api->get_float(tt, asset_r, TM_TT_PROP__SPRITE_COMPONENT__FREQUENCY);
    c->amplitude = tm_the_truth_api->get_float(tt, asset_r, TM_TT_PROP__SPRITE_COMPONENT__AMPLITUDE);
}

Fantastically straight forward. Get a read pointer to the object (here called "asset"), then update the component struct with the values that were stored in The Truth.

Understanding entities and The Truth a bit better

It isn't clear from any doc comments when precisely this function will be called or how to reflect changes to the component back into The Truth but my assumption at the moment is that when we registered the editor ui aspect with the object type, we're setting up a way to get a UI to update values, which would call this load_asset function.

It's not hard to test I guess...

First up, let's open the editor and add our component to the world entity:

And here is the component editor we get:

Before I remove the editor aspect and test what happens there, maybe I can add a log line to our component__load_asset function and see if this gets called every time we change a property value in the editor.

We'll need a handle to the log api first:

static struct tm_logger_api* tm_logger_api;

#include <foundation/log.h>

/* .... later inside tm_load_plugin ... */
tm_logger_api = reg->get(TM_LOGGER_API_NAME);

Now we can add a little log output as a treat:

tm_logger_api->print(TM_LOG_TYPE_INFO, "Howdy from component__load_asset!");

Build the plugin (remember to run with the --hot-reload flag), return to the editor change a value on the component editor aaaaaaand.... no log output:

Is this because the logs are filtered or because my assumptions were incorrect? Let's just add logs to all the things and see... crash. OH! I uh, did a dumb thing and put a log line at the start of tm_load_plugin before I had a refrence to the api. :facepalm:

After I fixed that and restarted the editor, oila! There are logs... and I'm seeing that load_asset is indeed called when editing the component.

So I don't know why the log output didn't appear after a hot reload, but I can put that investigation on to the backburner for a while. So I'll quickly clear the console, clear all log lines from the plugin, and finally rebuild to make sure everything continues to work as expected... and the log lines are still there. I clearly do not have a complete picture of the plugin hot-loading so I'm betting that by removing the editor ui aspect we won't see a change, unless perhaps we close and restart the editor... which was correct.

But perhaps not surprising, after rebuilding and reloading the editor we no longer have the ability to add the component to the entity!

This is interesting to consider. Without setting up the editor ui aspect for an object type in The Truth, we can't create a component on the entity in the editor... but where exactly did we hook that relationship up?

It appears that, when we register an implementation for the tm_entity_create_component_i interface, the component name is being used to connect The Truth to our entity oriented code. And what is also interesting is that in docs for the interface we register, the implementation is actually referred to as a "component manager".

Wrapping up!

Welp, this was pretty long, and hopefully illuminating or interesting. I definitely learned a lot but it's just the beginning. For the next post I'll start looking at the S in the ECS of The Machinery.