nick.recoil.org - Articles

Efficient Ghost theme development using Docker & livereload

Wed, 18 Mar 2026 12:28:23 UTC

April 2026: A new official tool has been released which gives an alternative method to the one described below. You can now use the ghst tool via ghst theme dev ./theme-dir --watch --activate.

When developing Ghost themes inside Docker, getting a fast feedback loop of edit/reload/view can be tricky. Here’s how I set up live reloading and instant theme updates using Docker and Gulp, and a bonus use of Maildev to make SMTP configuration super simple.

This setup was a lifesaver when I started using AI assistants for tasks outside my usual wheelhouse, like wrestling with complex CSS media queries. By adding a specific Theme Development Workflow section to my agent.md, I enabled Antigravity to debug layouts in a Chrome instance, it understood it could see the changes reflected as soon as it made an update to the file.

Theme Development Workflow:
When you modify theme files (Handlebars, CSS, or JS),the system is configured to reflect those changes immediately. You do not need to manually restart the Docker container or refresh the browser; the gulp watch task handles the injection and reload automatically.

Project Structure Overview #

Here’s an overview of the project root’s structure, in a simplified form. The basic gist is that I have a root directory containing docker things, and a root directory for the theme.

├── docker
│   ├── docker-compose.yml
│   ├── docker-ghost-mysqldb
│   ├── docker-ghost-content
│   │   └── themes
│   │       └── mytheme-ghost-theme
│   └── docker-mysql-data
└── mytheme-ghost-theme
│   ├── package.json
│   ├── gulpfile.mjs
│   ├── home.hbs
...

This is what the docker-compose.yml file looks like. I’m including the full file here because I found various examples of this online, but none of them were exactly right for my needs.

services:
  ghostdev-web:
    image: ghost:latest
    restart: unless-stopped
    ports:
      - 2368:2368
    environment:
      - database__client=mysql
      - database__connection__host=ghostdev-db
      - database__connection__database=ghostdb
      - database__connection__user=ghost
      - database__connection__password=SHUUUSHSECRET
      - mail__transport=SMTP
      - mail__options__host=maildev-test
      - mail__options__port=1025
      - mail__options_secure=false
      - url=http://mydevbox.local:2368
      - NODE_ENV=development
    volumes:
      - /Users/nick/ghost-test/docker/docker-ghost-content:/var/lib/ghost/content
    depends_on:
      ghostdev-db:
        condition: service_healthy

  ghostdev-db:
    image: mysql:8.0
    restart: unless-stopped
    container_name: ghostdev-db
    environment:
      MYSQL_ROOT_HOST: '%'
      MYSQL_ROOT_PASSWORD: ULTRAMEGASECRET
      MYSQL_DATABASE: ghostdb
      MYSQL_USER: ghost
      MYSQL_PASSWORD: SHUUUSHSECRET
    volumes:
      - /Users/nick/ghost-test/docker/docker-ghost-mysqldb:/var/lib/mysql
    healthcheck:
      test: ["CMD-SHELL", "mysqladmin ping -h 127.0.0.1 -p$$MYSQL_ROOT_PASSWORD || exit 1"]
      interval: 10s
      timeout: 5s
      retries: 10
      start_period: 60s

  maildev-test:
    image: maildev/maildev:latest
    restart: unless-stopped
    ports:
      - 1080:1080
      - 1025:1025

The key thing to note here is that the Ghost container has /var/lib/ghost/content mounted in the host filesystem as {$PROJECT_ROOT}/docker/docker-ghost-content/. This is important as it gives us easy direct access to the active theme. If we overwrite the files in the active theme directory, it has an immediate effect on the files served by Ghost inside the Docker container. We don’t need to go through the process of uploading another zipfile of the theme via the Ghost admin interface.

The use of Maildev is a little bonus. It’s not directly related to the livereload technique, but there’s a section right at the end talking about why it’s useful.

Gulp #

I based my theme on Casper, which is one of the core example themes available on GitHub. It already comes with a Gulp configuration, so here is an outline of how my livereload changes work.

I placed the following somwhere towards the bottom of default.hbs in my Ghost theme.

{{LIVERELOAD_SCRIPT}}

This is just an injection hook which gulp will search and replace, allowing us to alter the content as it copies the file from its source directory into the Docker container.

const LIVE_RELOAD_URL = 'http://mydevbox.local:35729/livereload.js';
const GHOST_THEME_PATH = path.join(process.cwd(), '../docker/docker-ghost-content/themes/mytheme-ghost-theme');

We need to introduce two const declarations. These could be moved to an environment variable to make this concept more portable.

The livereload server also needs to be running while gulp is monitoring for changes.

  function serve(done) {
    livereload.listen();
    done();
  }

I then added a replace() command inside the handlebars section of the gulpfile.

  replace('{{LIVERELOAD_SCRIPT}}', ''),

And when I build the distribution zipfile for the theme, it removes the injection hook.

  replace('{{LIVERELOAD_SCRIPT}}', ''),

The actual process of copying to Docker is given its own method, encapsulating the above replace() functionality:

function copyToDockerGhost(done) {
  // 1) Copy all non-HBS assets as-is (no string replacement on binary/text assets).
  pump([
    src([
      '**',
      '!node_modules', '!node_modules/**',
      '!dist', '!dist/**',
      '!yarn-error.log',
      '!yarn.lock',
      '!gulpfile.js',
      '!gulpfile.mjs',
      '!.git', '!.git/**',
      '!*.hbs',
      '!partials/**/*.hbs'
    ], { encoding: false }),
    dest(GHOST_THEME_PATH)
  ], function (err) {
    if (err) {
      return handleError(done)(err);
    }

    // 2) Copy HBS templates with live-reload script replacement.
    pump([
      src(['*.hbs', 'partials/**/*.hbs'], { base: '.' }),
      replace('{{LIVERELOAD_SCRIPT}}', ''),
      dest(GHOST_THEME_PATH)
    ], handleError(done));
  });
}

Now when I edit any of the handlebars template files, you’ll see the following output in a terminal session running gulp in watch mode.

$ npm run dev > mytheme-ghost-theme@0.0.1 dev> gulp[12:42:50] Using gulpfile …/Ghost/testing-ghost/mytheme-ghost-theme/gulpfile.mjs  
[12:42:50] Starting 'default'… 
[12:42:50] Starting 'css'… 
[12:42:50] Finished 'css' after 192 ms  
[12:42:50] Starting 'js'… 
[12:42:51] Finished 'js' after 360 ms 
[12:42:51] Starting 'copyToDockerGhost'… 
[12:42:51] Finished 'copyToDockerGhost' after 153 ms  
[12:42:51] Starting 'serve'… 
[12:42:51] Finished 'serve' after 3.21 ms 
[12:42:51] Starting 'cssWatcher'… 
[12:42:51] Starting 'jsWatcher'… 
[12:42:51] Starting 'hbsWatcher'… 
…
[12:45:32] Starting 'hbs'… 
[12:45:32] …/Ghost/testing-ghost/docker/docker-ghost-content/themes/mytheme-ghost-theme/archive.hbs reloaded. 
[12:45:32] …/Ghost/testing-ghost/docker/docker-ghost-content/themes/mytheme-ghost-theme/default.hbs reloaded. 
[12:45:32] …/Ghost/testing-ghost/docker/docker-ghost-content/themes/mytheme-ghost-theme/error-404.hbs reloaded. 
[12:45:32] …/Ghost/testing-ghost/docker/docker-ghost-content/themes/mytheme-ghost-theme/error.hbs reloaded. 
[12:45:32] …/Ghost/testing-ghost/docker/docker-ghost-content/themes/mytheme-ghost-theme/home.hbs reloaded. 
[12:45:32] …/Ghost/testing-ghost/docker/docker-ghost-content/themes/mytheme-ghost-theme/index.hbs reloaded. 
[12:45:32] …/Ghost/testing-ghost/docker/docker-ghost-content/themes/mytheme-ghost-theme/page.hbs reloaded. 
[12:45:32] …/Ghost/testing-ghost/docker/docker-ghost-content/themes/mytheme-ghost-theme/post.hbs reloaded. 
[12:45:32] …/Ghost/testing-ghost/docker/docker-ghost-content/themes/mytheme-ghost-theme/series.hbs reloaded. 
[12:45:32] …/Ghost/testing-ghost/docker/docker-ghost-content/themes/mytheme-ghost-theme/tag.hbs reloaded. 
[12:45:32] …/Ghost/testing-ghost/docker/docker-ghost-content/themes/mytheme-ghost-theme/partials/post-card.hbs reloaded. 
[12:45:32] Finished 'hbs' after 115 ms  
[12:45:32] Starting 'copyToDockerGhost'… 
[12:45:32] Finished 'copyToDockerGhost' after 140 ms 

When you modify any handlebars, css or js files, you’ll see a flurry of log messages, and then your browser will reload any page being served via your Ghost container. Magic! This has been extraordinarily helpful in lowering iteration time for work on the theme.

Full Gulpfile example #

You can find the full gulp configuration file here: gulpfile.mjs, where there’s slightly more configuration to ensure things like css editing also cause a livereload event.

Just running gulp leaves you in development mode, where it watches for file changes, and livereload is active. Running gulp zip will build the theme zipfile into the dist/ directory.

Limitations #

There are limitations. If you introduce a new file, you must bounce the server in order to get it to be seen correctly. Theme files seem to be read once at launch and then not again, unless you’re uploading a new version via the admin interface. You can simply run:

docker compose restart ghostdev-web

Permissions can also be a problem. Generally speaking the ghost content will be owned by the root user, so you might need to play around with file permissions in order to be able to overwrite the data in docker/docker-ghost-content/themes/.

Maildev #

The last thing to mention is using Maildev. It’s an SMTP server which also presents a webmail interface for anything sent through it. This means that you can easily view emails that Ghost sends, like a sign in link. All you need to do is go to port 1080 on your local machine, and you’ll see this interface.

This saves you the bother of needing a fully working SMTP user for your development environment, which would send live emails over the internet. Now emails don’t need to leave the container pool of your project.

Maildev in action

]]>

Simulating falling autumn leaves in Blender

Mon, 26 Jan 2026 16:36:01 UTC

In this article I’ll be explaining how to create a small looping animation of falling leaves in Blender using geometry nodes instead of the traditional particle system approach. We’ll break down the simulation into sections, and explain each part in detail. This is written as a tutorial, so you’ll get the most out of it if you know the basics of Blender already.

The finished animation

This guide is designed to help bridge the gap between simple geometry instancing and more complex physics simulations. We are going to ditch the traditional Particle System functionality and build our own falling leaf engine completely inside Geometry Nodes.

While that might sound intimidating, the setup here is focused on aesthetics rather than accuracy. We’ll build a “good enough” physics model that looks great and loops perfectly. If you have a basic grasp of Blender nodes and want to see an example of the Simulation Zone in action, this project is the perfect playground.

The packed Blender file, includes all textures is available to download and use.

falling_leaf_simulation.blend

The inspiration #

In autumn last year I was chatting with my partner about how spectacular the colour of the trees and falling leaves were in our local park. It got me thinking about making a small animation in Blender using geometry nodes. Currently, state-of-the-art AI video generation in this space is weak. Making the independent, chaotic motion of leaves believable is difficult.

I also wanted to push my understanding of Blender’s increasingly sophisticated geometry nodes functionality, and more broadly improve my ability to develop within node-based environments. In the past I’ve been hostile to node-based programming, thinking it was a poor substitute for writing code directly in an IDE. However as time has gone on, and I’ve experienced working within teams where changes with time and people cause programs to atrophy, I’ve come to appreciate the benefits. Node-based programming can provide a great API to maintain backward compatibility while allowing feature development, bug fixes and general improvements.

Implementing a falling leaves simulation is something you’ve been able to do in Blender for a long time. You’d stick down a particle system, get it to instantiate the leaves with some randomisation, and then apply some wind or turbulence in order to get them to fall in a suitably realistic way. This has worked well since circa 2008, but we’re now at a point where we can recreate all the functionality we need ourselves using geometry nodes, and extend it in ways which would have been impossible with the particle system.

Specifically for our purposes, geometry nodes received an update in Blender 3.6 which allows you to run simulation loops during your animation. This is the key feature which unlocks our ability to use geometry nodes for this project.

It is however a double-edged sword. You have all the freedom you could possibly want, but you need to start from first principles with a blank canvas, and build everything yourself. This can be intimidating, and the examples you can find online don’t often deal with simulations; they tend to be more concerned with procedural geometry. This ended up being the challenge I set myself. Could I learn enough to implement a complete animation?

Source textures #

I looked for photographs of autumn leaf scenes, and found this one on Freepik. It suits our needs by being geometrically simple, has nice even lighting, and a very chaotic bed of leaves on the floor. The dynamic falling leaves could easily blend in once they land.

For the leaves, I collected a handful from my local park and photographed them outside on a plain white piece of paper. It looks like the leaves in the background photograph are larger Maple leaves, but sadly I don’t live near any Maple trees. You work with what you have!

A selection of the yellow leaves I collected

Bringing these textures into Blender and mapping them onto some planes, they looked good, but presented a problem. They were too flat.

The leaf textures brought into Blender

Crumple the leaves #

If you animate perfectly flat planes, it doesn’t look very realistic. As they rotate to be side on to the camera, they disappear. To prevent this, we need to add some geometric distortion to simulate the drying and crumpling of real autumn leaves.

The geometry nodes to distort the mesh

We can create a simple geometry node network which subdivides the plane, and then uses a Noise Texture node to push the normals of the mesh around a bit. Set the scale, detail and roughness of the noise settings to get a nice random look to the leaves.

The distorted mesh

The textured mesh

The leaves now have a better side profile

The scene #

The scene setup is simple. There’s not enough information in the image to make solving the camera perspective easy, but given just how simple the scene is, we can eyeball it.

The 3D scene

There’s a camera, a thin emitter plane at the top, and a collider plane at the bottom. The emitter plane is tilted slightly to keep it out of the camera frustum, and the collider plane is flat, and approximates the ground position. There’s no specific light source, we use an environment texture to give us approximate lighting conditions. This suits the overcast day in the photograph.

I’ve also placed the collection of leaf objects out of view of the camera so that I can work on the geometry and materials easily.

The camera setup is the only difficult element. The source photograph luckily has EXIF tags, including the lens used. Unfortunately the lens is a variable zoom, and has a range of 24mm to 70mm. This means we need to eyeball the camera settings to get a similar perspective. Again, luckily for us the simplicity of the scene makes this more forgiving.

The emitter geometry nodes #

Now we come to the heart of our falling leaf simulation: the geometry nodes for the emitter. Let’s break down this fairly intimidating node network into sections.

The entire geometry node simulation network. You can click to view the full-size image.

The key to this setup is the Simulation Zone. Unlike standard geometry nodes which evaluate from scratch every frame, a Simulation Zone allows us to carry over data from the previous frame. By combining this with Named Attributes to store values like velocity and rotation, we can update the state of our leaves iteratively to create a physics simulation.

A high level overview of the steps taken per-frame are:

Randomly instantiate additional leaves
Update the velocity from the fixed acceleration
Mix in a simulated wind component to the velocity, driven by a noise texture
Update the instance position from the velocity
Update the instance rotation from the rotational velocity
Calculate the collision with a floor plane, and exponentially damp the acceleration and velocity components
Set a lifespan fade value to use in the shader
Remove leaf instances at the end of the lifespan

Spawning leaves #

Rate-limit the leaf spawns

The first thing we need to do is control how many leaves we spawn. One of the first surprising things I encountered was that when using the Distribute Points on Faces node, we need to control the spawn count by the density of the points, not the number of points. This means that scaling the spawning zone geometry will change the number of leaves spawned per second.

The most simple way I found to control this in a way which is compatible with our requirement to have a looping animation is the following:

Take the frame number and Modulo it with your looping frame count
Use that integer as a seed for a Random Value
Set a threshold, and if the random number is Less Than a chosen value, use the result output as the Selection input to Distribute Points on Faces

Setting it up this way gives us an upper limit to the emission rate. If we exposed Density directly, it creates opportunities for UI accidents, and a slip of the mouse can change the value from 0.01 to 10, and you’re suddenly instantiating 10,000 leaves each frame, tanking the application performance. Setting a reasonable ceiling value, and then giving a percentage control to that maximum is much safer for artistic experimentation.

Randomisation #

In order to give us an output which is capable of looping, all random numbers are seeded with the frame number modulo the repeat count. This gives us repeatable behaviour over a set number of frames.

Attribute setup #

Initialise the Named Attributes we will be using

Now we need to set the initial values for the Named Attributes we’ll be using during the simulation. The input values are primarily sourced from the Group Input node so we can easily experiment with different values outside of the GN editor.

lifespan: The duration of our leaves in frames
accel: The static acceleration vector applied to each leaf. In our case it only experiences a negative Z acceleration due to gravity
vel: The initial velocity given to each leaf, zero in our case
maxAngSpeed: The upper limit on how fast the leaves can spin and tumble
normAngVel: The angular velocity of each leaf, expressed as a Vec3. This is given a random vec3 from -1 to 1. This is multiplied with maxAngSpeed to calculate its rotation speed

For completeness there is one more attribute we use, but it’s calculated and set dynamically later in the network.

leafFadeOut: A 0-1 float which is an expression of inverse leaf alpha. This is used in the leaf shader.

Instantiation #

Now we create our leaf instances.

We instance our leaves as you might in a standard Geometry Node setup, but with a slight twist. Because we’re using a Simulation Zone, we have an existing source of geometry instances inherited from the previous frame. We want to merge these existing leaves in with our new instances, and then feed them into our network so that old and new are treated equally.

Cheap turbulence simulation #

Cheap and easy velocity manipulation

To provide a simple way to simulate the way leaves fall, we will sample a 3D fBM Noise Texture and add this to our leaf velocity, acting as a pseudo acceleration force. Before this is added to the leaf velocity, we scale it by an exposed value, allowing us to easily dial this strength up and down according to our needs.

Rather than this being directly added to the vel attribute each frame, we need to take into account when this turbulence needs to be applied, and when it doesn’t.

Soft ground collision #

Our leaves will eventually reach the ground, and must stop moving. Aesthetically it’s much more believable if the leaves come to a rest with a slight deceleration, rather than coming to a halt instantaneously. This means we need to create a system of soft collision. The linear acceleration and velocity as well as the rotational velocity should be arrested by their proximity to the ground collider.

It’s important to note that we’re not using a heavy Rigid Body physics engine. Instead, we are creating a position-dependent damping field. Think of it as the air getting thicker the closer the leaf gets to the ground, eventually freezing it in place.

This field will give us a coefficient we can use to multiply the acceleration, velocity and rotation with, giving us an asymptotic decay function. We effectively lerp the system to a halt by using a fractional value.

Our nodes to calculate the effect of the damping field

Above is the group of nodes we will use to calculate the effect of the damping field. Note that we use Relative as the transformation applied to the output. We’re in the coordinate space of the emitter, so we need to bring the ground collider into that space to get the correct proximity values.

The Geometry Proximity node can be set to Faces because we’re a single flat plane. Lastly we clamp the output of the proximity node to a maximum of 1, giving us our damping field coefficient, ready to multiply our acceleration, velocity and rotation by.

For some more flexibility we could divide the proximity by a scalar to give longer range to the collision. For our leaf collision purposes, a proximity of 1 is sufficient because the scale of the leaves is small.

Employing our soft collider calculation

There are two ways we use the soft collider output. One is to directly use the fInFreefall value, which is 1 in freefall, and drops to a value below 1 during collision. We will demonstrate this later. The other is to convert our freefall value into an inverted collision boolean, where it has a value of 0 in freefall, and 1 during collision.

We use this collision factor to immediately kill any contribution the position noise field makes to our leaf velocity by use of a scale node. The output of the scale node in the above image is the velocity contribution of the noise field.

In physics terms, we are applying a drag coefficient that increases infinitely as distance approaches zero.

Updating velocity and position #

The power of simulation nodes, being able to update position and velocity

Now we come to update the leaf’s velocity and position. First we grab the accel attribute and multiply it by Delta Time to get the velocity contribution. We then add this to the velocity contribution from the turbulence simulation, and add that change in velocity to the existing vel attribute. The Scale node then scales the overall velocity by the fInFreefall value.

To describe what we’re doing in mathematical notation, we can define $d$ as the distance to the collider, then we can define $\lambda = \text{clamp}(d,0,1)$. This is what we assign to fInFreefall. Then within each iteration we will calculate $\vec{v}_\text{damped} = \vec{v}_\text{original} \cdot \lambda$. This damping will be applied to the linear and angular velocity, as well as the acceleration.

Updating the acceleration and velocity attributes

In the above image, we can see the vel, accel and normAngVel attributes being updated. The velocity has already been scaled by the fInFreefall value, but the other two still need to be scaled before being updated.

Updating the rotation #

Updating the rotation of the leaf instances

In order to calculate the updated eulers, we need to scale the normAngVel by the maxAngSpeed and multiply it by Delta Time. This gives us the change in rotation. We then convert this to a rotation and use it in the Rotate Instances node.

Fading and deleting the leaves #

The last two things we need to do is fade out the leaves and delete them at the end of their lifespan. Note that while the attribute is named lifespan, it actually stores the end frame value—the specific frame number at which the leaf instance should be deleted.

Calculating the fade out value

The following pseudo code describes the fade out calculation, where k-prefixed variables are our constants defined for the whole network, and attr is the prefix for named attributes.

fadeStartFrame = attrLifespan - kFadeFrameCount
framesIntoFade = fadeStartFrame - currentFrameNumber
t = framesIntoFade / kFadeFrameCount
return MapRange(t, 0, 1, 1, 0)

This is stored in the leafFadeOut attribute, starting at 0, and rising to 1 when the leaf has faded out completely. It’s effectively an inverse Alpha calculation. We need to have a default value of 0 when the leaf is visible, which will be important once we come to implement the fade out in our leaf shader.

Deleting the leaves at the end of the lifespan

The deletion comparison is easy, we just compare the current frame number to our lifespan attribute. If the current frame number is greater than the lifespan attribute, we delete the leaf using the Delete Geometry node targetting the instances.

The leaf shader #

Using the fade out value to fade out the leaves

The leaf shader is set up in a simplistic way. The lighting in our scene is very uniform and almost blown out, certainly very saturated. In order to get the rendered output to match against the plate, it’s necessary to make the leaves slightly emissive. We also have basic hue, saturation and value controls, as well as a brightness and contrast control in case there needs to be any additional tweaking.

The only other point of note here is the use of the leafFadeOut attribute to control the opacity of the leaves. In order to be able to use this shader when there is no attribute set, we rely on the default value of 0 meaning that the leaf is fully opaque. Because the attribute is the inverse of what we need to use for the alpha attribute of the standard Principled BSDF shader, we need to invert it before using it.

The final shader with unique texture per leaf

Here we can see the same shader is used, we just vary the input texture for each different leaf.

Compositing #

The final composite of the leaves and the plate

Lastly we have the compositing setup. We just use some global Hue, Saturation and Value nodes to adjust the colour of the leaves to match the plate, and an Exposure node to adjust the brightness. The leaves are then composited over the plate using the alpha channel.

Looping video #

In order to create the looping video, we need to ensure that the output frame count is at least twice the loop count. We can then use ffmpeg to chop out the section we need.

ffmpeg -i /path/to/video.mp4 -vf "trim=start_frame=400:end_frame=600,setpts=PTS-STARTPTS" -an output.mp4

There may well be a way to achieve this step within Blender itself, but I knew how to achieve this using ffmpeg.

Conclusion and future work #

I’m happy with the final animation. The camera setup isn’t perfect, but the aesthetics look good nonetheless. There are a number of areas I’d like to explore further.

Grouping more nodes together to make the top level graph more manageable, and easier to reuse in a modular fashion
Use the baking capability to make playback quicker when you don’t need to change the parameters of the simulation
Avoid the ffmpeg step, and produce a looping video directly from Blender

A split view to show the leaves more easily

]]>

Making maps without getting lost

Sun, 09 Mar 2025 20:52:01 UTC

This is a story about creating an interactive tile-based map of a fictional island from a video game. It goes through the inception of an idea, self-imposed constraints, and staying focused on delivery. There’s enough discovery, intellectual rabbit holes and ruthless pragmatism to turn any casual hobby into an existential crisis.

Panning across our fictional video game island

Inception #

I recently picked up Arma Reforger and I’ve been enjoying the chaos of its 120 player online mode. The Arma series of games have always been positioned more as a Milsim (military simulation) rather than a classic first-person shooter like Battlefield or Call of Duty. One of the hardest aspects of the game’s steep learning curve is map knowledge. New players swiftly discover that there’s no ability to see where they are on the vast 13km by 13km map. You’re instantly lost, and it’s very disorientating.

Over time you build up a visual familiarity with your surroundings, and after a few weeks of playing you start to recognise the frequently used roads and landmark buildings. Alongside this map knowledge is familiarity with the location of supply caches which play an important role in the game mechanics. These supplies form the backbone of the in-game economy of both teams. If your team’s bases don’t have sufficient supplies, you can’t purchase vehicles or specialised equipment, you can’t extend the bases with new structures, and in dire situations, you won’t even be able to respawn at them.

You sometimes stumble on these supply caches yourself, and sometimes other players show you while you’re playing, but this knowledge is harder to acquire. The caches can range from large and obvious brown shipping containers to wooden crates stuffed in the attics of some very unassuming houses.

The various visual representations of the supply caches inside the game

After some weeks of playing, it occurred to me that having a map on a second screen as you play would be incredibly useful, so thought it’d make a fun side project.

I didn’t want to walk around and scout them out manually; they would need to be 100% procedurally created. I’d have to derive all the locations of caches, not to mention the game map itself, from within the game’s data.

To do so, I’d have to work within the Reforger game engine, implement some command-line image processing, and ultimately bring everything together using a browser-based JavaScript map framework.

I wanted this to be a short project, with minimal distractions. If interesting coding or data problems came up during the work, it was important to be pragmatic and stay focussed on shipping version one. In this post we’ll go through the problems I had to solve along the way, and the approach I took.

Deconstruction
The Enfusion Workbench
Cropping map tiles
Implementing LeafletJS
Zoom levels
Map stats
Extracting location data
The finished map
Some topographic fun
Conclusion

Deconstruction #

The joy of a project like this is the polydisciplinary nature. No one single aspect of the work is likely to be be particularly difficult. However, each of the stages require working in a different domain, with different languages and different libraries, and all stages must be implemented for the project to deliver on its promise.

Another enticing aspect of a project like this is to create a genuinely useful public tool, and it’s always fun to give back to a community you enjoy being a part of.

You begin thinking about a project like this by working backwards. The interface for the tool I want is going to be something like Google Maps. Web browsers can display these really well, and it can work on any size screen. If you want to make a map website you need map tiles to display, and coordinate data to dictate where we place the map pins. To get hold of both these elements I need to start with the game engine.

Arma Reforger’s open developer tools are common aspect of all Bohemia Interactive’s games. I’d done some light modding work for Arma 3 over 12 years ago, but the technology has been completely overhauled since then, so I’ll effectively be starting from scratch.

Now let’s put the steps back in the right order:

graph LR; A[Game engine]-- Screenshots -->B[Image processing] A-- Entity query -->C[Location Data]; B-- Map tiles -->D[Web frontend]; C-- JSON -->D;

First I’ll need to dive into the most unknown part of the project, the game engine.

The Enfusion Workbench #

The Enfusion Engine is a cross-platform engine using Qt for the interface, implementing a scripting language called Enforce Script. As far as I know this is entirely their own creation, and not derived from any specific base language. Thankfully it’s reasonably C-like, so it’s not too difficult to get up to speed with.

Enforce Script and the Enfusion game engine are niche topics, so unfortunately it’s not possible to get Copilot or Claude to reliably help you develop this. We’re going to be taking the traditional route of reading the documentation and example code. How old school!

The development tools are accessible via Steam. It’s a single application with multiple windowed sub-applications that are tailored to specific tasks like modelling, scripting, audio, particle systems etc. I’m focusing on the World Editor and the Script Editor as these are the sub-application which load and render the game worlds, and the main scripting IDE respectively.

The first thing to do is load up one of the game worlds and look to see whether the supply caches are present in the map data, or whether they are instantiated at runtime. If we can see them in the offline map it’s going to be a lot easier to work with.

Supply locations as seen in the World Editor

Yep, there they are! Now let’s turn our attention to the map. Maybe there’s an existing way to export topographical data we can use for our map tiles? Looking around the editor there are various mentions of likely tools like Map Exporter, Export Map data, Export Geographic data. The only one that seems to produce an image is the Export Map data tool. It has three modes of operation, and the two of interest are RASTERIZATION and GEOMETRY_2D.

The exported image from RASTERIZATION mode

Running it in RASTERIZATION mode exports a large 4096 x 4096 px texture which forms a base map used inside the game, and you can see that above. In the game it’s also composited with vector information detailing forested areas, the road and path network, and some building data. This vector data is exported to a .topo file, but unfortunately there’s no documentation on the format for this file.

Ok, so there are no existing maps which fit the requirements. Time for Plan B; let’s write our own map tile exporter!

Looking at the documentation and sample applications it’s possible to script the position and orientation of the editor camera fairly easily, as well as capturing output images to disk. Unfortunately there is no control over the perspective matrix, which means there’s no way to make it orthographic, and we’ll have issues relating to perspective.

// Get World Editor module
WorldEditor worldEditor = Workbench.GetModule(WorldEditor);
// Get World Editor API
WorldEditorAPI api = worldEditor.GetApi();
// Position the camera 1 km up, looking down
vector camPos = Vector(3000, 1000, 3000);
vector lookVec = Vector(0, -90, 0);
api.SetCamera(camPos, lookVec);
// Now create the screenshot
System.MakeScreenshot("test");
// we now have a 'test.png' file

If we can’t eliminate issues with perspective, then we need to minimise them, so let’s emulate real-world satellite data. If we position the camera far up, and with a very narrow FOV, we can compress the resulting perspective distortion as much as possible. The balancing act here is having a narrow FOV without fighting against the engine’s desire to enforce Level of Detail restrictions on how many and how well every object is rendered. Zoom in too much and you can ruin the visual fidelity.

A vertical FOV of 15 degrees offers a good mix of fidelity vs granularity. It gives us approximately 20 cm per pixel in the resulting image, which is enough to resolve terrain features you’ll encounter while walking or driving around the map.

Control over the camera is very restricted, and it’s not possible to control the field of view or far plane distance via scripting. Part of our process will require the user to set up some camera parameters manually, but luckily it can persist across sessions, making it less of a source of potential error.

Manually setting the camera FOV and far plane is not too difficult

The process to capture the images is straightforward, but consumes a lot of disk space. This is partly due to the difficulty of controlling the size of the camera window within the editor, and therefore the size of the output screenshot. The only consistent and reliable way to control this is to put the camera into full-screen mode with the F11 key, which then makes the screenshots the same size as my monitor resolution, 2560 x 1440 px. It captures a lot more data than we require, but again it’s about making this process repeatable in a regime where you can’t have the script enforce the capture parameters.

We’ll be moving the camera across the island in two nested loops, one for the X axis, and one for Z. We want to traverse across the 13 km of each axis in 100-metre steps, which means 16,900 steps in total. That’s a lot, so we need to make the capture process capable of detecting existing screenshots, and skipping over that position. This will save us a lot of needless repetition as we stop and start the process during development.

We can also make this process detect the cropped tile from the screenshot using a _tile.png suffix, meaning that we can delete the original screenshots once we’ve cropped them. This will help keep the intermediate disk usage down. Every screenshot is around 6 MB, which means our total intermediate disk usage will be nearly 100 GB. Did I mention I was short on disk space during this project? I became a bit obsessed with storage efficiency!

The last trick of camera movement is to make the camera stay at a fixed height relative to the terrain surface. This ensures that perspective artefacts that make edge discontinuities are minimised when travelling over areas of rapid height change.

Capturing this data takes an hour or two, as pauses need to be added to the capture loop to allow the renderer to stabilise the visual image, and write the screenshot to disk. During testing, if I moved the camera too rapidly I could alter the buffer contents before it was successfully written to disk, or I could take the screenshot during the eye adaptation changes, creating inconsistencies.

Cropping map tiles #

Overlapping screenshots forming a continuous vertical strip

So now we have many many screenshots, each slightly offset from the other in a grid. The task ahead is to find out exactly the right centre crop of the screenshots such that each cropped image tiles perfectly with its neighbours. There are plenty of approaches within the Python ecosystem for stitching together images, OpenStitching or plain OpenCV for instance. These are mostly concerned with creating a single output image rather than keeping the sources as tiles, so they’re not as helpful as I first thought.

It’s also worth stating that I chose Python for this because it was the lingua franca when I worked in post-production, and the habit has stuck. The code is usually clear, the image processing libraries are numerous and mature, and support is widespread. While I was developing this on my Windows box, I used WSL to provide the runtime, although it could have just as easily been a native installation.

There are some unique properties of our images to exploit. Since we’ve ensured the spacing between each image is precisely controlled, there will be one square crop we can make which will work globally across the entire data set. We need only find the value once and we’re done.

The first step is to take an overly generous centre square of each screenshot and save it out as an intermediate tile. Stitching these together raw gives us obvious visual repetition at the borders, but we know we’ve still got the correct square somewhere in this image. Once we have these intermediate tiles, we can also delete the source screenshots. Efficiency!

The next step is to incorporate a dynamic overlap value we can control while running a script to produce a composite mosaic of our tiles. We don’t need anything sophisticated here, eyeballing the right overlap value was quick, and the right size for our tiles is 542 px.

You can see this approach in crop_screenshots.py which uses Python PIL to perform the image operations, and contains separate constants TILE_CROP_SIZE and TILE_OVERLAP.

The eagle eyed among you might see the brightness change in the screenshot images. This is because Arma Reforger incorporates eye adaptation which automatically adjusts the exposure, so the darker ocean has a higher comparative exposure than when you’re over land. It’s why we have what look like JPEG artefacts around the coastal regions.

A future piece of work will be to implement normalized cross-correlation using NumPy to find the correct cropping value automatically, but this isn’t necessary for the initial phase of this project. It’s important to recognise when you’re being pulled down a rabbit hole, and to keep version one simple.

The initial center tile crop

This tile should border on the previous tile

Tile 0/32/37

Tile 0/32/38

When we have dialled in the required TILE_OVERLAP we bake this into a final export of our LOD 0 tile set, and move onto the JavaScript to bring it to life.

Implementing LeafletJS #

Having looked at a number of different browser-based mapping frameworks, let’s use LeafletJS. It’s very established, has plenty of implementation examples, and seems to have a good ecosystem of community plugins should I want to extend the UX functionality later.

So let’s briefly talk about coordinate systems and terminology. We’re about to smash two incompatible groups into each other, game developers and GIS people.

Freya Holmér’s excellent coordinate system diagram

The Enfusion game engine sits in the CORRECT quadrant, the upper right 😊

+X is right, +Y is up and +Z is forward. The game world is flat, and all is good in the world. But wait, what’s that? We live on a sphere, and to use Latitude and Longitude to get around? Ut oh!

I don’t know which one would win in a fight

That’s right, LeafletJS was born out of needing to display real-world maps, so is fundamentally different from our rectilinear game world. Not only is there a spherical coordinate reference system, the map display thinks of the origin as being in the top left, and +Y is down.

To add insult to injury, we also have reversed thoughts on detail levels. So far I’ve named the tiles according to the game-dev LOD and Mipmap concepts. The ground truth is detail level 0, and all derivatives will become 1, 2, 3 etc., as we’re simplifying information and losing detail. Leaflet, however, thinks in terms of zoom levels. You start out at zoom level 0, furthest away from your subject and as you zoom further into the map, you get progressively more detailed tiles.

Coordinate problems are always a huge PITA. They can introduce sign, off-by-one and scale errors all over the place unless you’re disciplined. Thankfully LeafletJS has a number of approaches for tackling this, and makes it pleasant to work with with, once you know how to implement these conversions.

Firstly let’s tackle the coordinate flip. Our tiles have an origin in the bottom left, so we flip the Game Z coordinate as it’s converted into a Tile Y. We implement the following basic extension to the TileLayer class, and the +1 offset is there to account for flipping the origin from the bottom of the grid square to the top.

// Our custom tile layer which inverts the Y axis
L.TileLayer.InvertedY = L.TileLayer.extend({
  getTileUrl: function(tilecoords) {
    tilecoords.y = -(tilecoords.y + 1);
    return L.TileLayer.prototype.getTileUrl.call(this, tilecoords);
  }
});

Secondly, we need to invert the zoom numbering by adding zoomReverse: true to the tile layer, and specifying our maximum and minimum zoom levels.

// Configure our custom tile layer to use zoomReverse to match our Level Of Detail numbering
tileLayer = new L.TileLayer.InvertedY('LODS/{z}/{x}/{y}/tile.jpg', {
  maxZoom: MAX_ZOOM,
  minZoom: 0,
  zoomReverse: true,
  bounds: bounds,
}).addTo(map);

Lastly, we need to scale the coordinates correctly. This involves creating a custom Coordinate Reference System which scales the tiles such that the in-game coordinate system corresponds with the Leaflet coordinates. We firstly want to use L.Projection.LonLat as we want to specify coordinates in X, Z order. We also need to scale coordinates so that our tile size of 542 px is accounted for, where the standard tile size is 256 px.

  // Transformation() takes values to form a 2x2 linear transformation matrix
  L.CRS.CustomSimple = L.Util.extend({}, L.CRS, {
    projection: L.Projection.LonLat,
    transformation: new L.Transformation(1/12.501, 0, -1/12.501, 0),
    ...
  });

I’ve no clue why the scaling factor turns out to be 1/12.501. The whole CRS area of LeafletJS looked like a huge can of worms, so I ended up placing three map markers at the game coordinates of obvious visible landmarks and then tweaked the scaling factor until they all lined up. Good enough, let’s move on.

The final job is to create a couple of helper functions to add one last aspect of coordinate system conversion. There is a hidden offset in what we’ve created, and that’s the center origin of the tiles, vs the corner origin of LeafletJS. We need to account for this using the following:

// our tiles are 100 m, so it's 50 m to the centre
EDGE_TO_CENTER_OFFSET = 50 

function gameCoordsToLatLng(gameCoordinate) {
  return L.latLng([gameCoordinate[2] + EDGE_TO_CENTER_OFFSET, coordPair[0] + EDGE_TO_CENTER_OFFSET]);
}

function latLngToGameCoords(latlng) {
  return [latlng.lng - EDGE_TO_CENTER_OFFSET, 0, latlng.lat - EDGE_TO_CENTER_OFFSET];
}

The final implementation of all the above can be seen in EnfusionMapMaker/Web/reforger-map.js.

Zoom levels #

Now we have a set of LOD 0 tiles, we can aggregate them into LOD 1, LOD 2 etc. The animation below shows how this system works. In the bottom left corner of each tile we’re printing the LOD level, and the X and Y coordinate. The coordinates are based on scaling by powers of two. The tiles bounded by 64,64 to 65,65 become a single tile at 32,32. Then the same again, tiles inside 32,32 to 33,33 become 16,16 and so on.

It’s extremely useful to be able to turn on a debug layer in the map, and see exactly what coordinates were being used to fetch tiles, and this can be achieved with the following GridLayer definition.

L.GridLayer.GridDebug = L.GridLayer.extend({
  createTile: function (coords) {
    const tile = document.createElement('div');
    tile.style.outline = '1px solid #111';
    tile.style.fontWeight = 'bold';
    tile.style.fontSize = '14pt';
    tile.style.color = 'red';
    tile.innerHTML = [MAX_ZOOM - coords.z, coords.x, -(coords.y+1)].join('/');
    return tile;
  }
});

An animation showing the debug overlay with tile coordinates Z/X/Y

The two maps in the base game of Arma Reforger are called Everon and Arland. On Everon we need 5 zoom levels, but on Arland we only need 4 as it’s much smaller. The code to do this is implemented in make_lod() and merge_tiles() which does a bounds query, a composite and a write across each of the tiles in the previous LOD.

Last I use Imagemagick’s Mogrify tool in the compress_tiles.sh shell script to optimise the filesize of the JPEGs. This finely controls the quality, chroma subsampling, the colourspace and interlacing. This could have been integrated into the Python code, but it’s better to treat this step as a requirement of making the website, so it’s entirely optional and using existing tools well optimised for the task.

Map stats #

At this stage we’re done with the map tiles. We have all our zoom levels, and we support the game coordinate system to plot positions accurately. Stepping back, it’s nice to get a sense of the scale of the thing we’ve just created, as it’s not particularly obvious when you’re down working in the details.

Map name	Everon
Game area	13 km x 13 km
Game distance per LOD 0 tile	100 m
Tile image size	542 x 542 px
Resolution	~20 cm per pixel
Total tile storage	398 MB
LOD filesizes (0-5)	276 MB / 81 MB / 24 MB / 6.7 MB / 1.9 MB / 524 KB

I was surprised at how large the tile set was, but if you do the maths, LOD 0 is effectively a single 70,000 x 70,000 px image, and would need 14 GB to hold uncompressed in memory!

Extracting location data #

Now we have our map, with accurate scale, we now need to generate a set of coordinates for every hidden supply cache on the map. We first look for commonalities within the entity and prefab system.

Every supply cache has an inventory object you can interact with. This is usually in the form of a wooden post.

The post which allow interaction with the inventory system

If we look at the component properties of this object in the Enfusion Workbench, we can see it contains two components which look unique to these types of object: InventoryItemComponent and SCR_ResourceComponent.

The unique components which denote a supply cache location

Now we know what components we’re looking for, we need to find out how to make the queries. Unfortunately there’s not a huge amount of documentation for making Enfusion editor tools. The best place to start is the Sample World Editor Tool that Bohemia have published. This gives us the following hint.

// Get World Editor module
WorldEditor worldEditor = Workbench.GetModule(WorldEditor);
// Get World Editor API
WorldEditorAPI api = worldEditor.GetApi();
World world = api.GetWorld();

With a World handle, we can use the method QueryEntitiesByAABB() to query and filter a list of objects. The two callback methods we have available control whether an object should be added, and then when it’s added, the callback controls whether the process should continue or not.

  // Init our array to store the entities
  m_entityResults = new array<IEntity>;

  // Declare our bounds
  vector queryBoundsMin = new Vector(0, 0, 0);
  vector queryBoundsMax = new Vector(13000, 200, 13000);

  // Perform our bounded query for entities
  bool queryResult = m_currentWorld.QueryEntitiesByAABB(
    queryBoundsMin,
    queryBoundsMax,
    filterEntitiesCallback,
    addEntitiesCallback,
    EQueryEntitiesFlags.ALL
  );

  // A basic test to look for both components being present
	bool filterEntitiesCallback(IEntity e) {
    if (e.FindComponent(SCR_ResourceComponent) &&
        e.FindComponent(InventoryItemComponent)) {
      return true;
    }

    return false;
  }

  // Find all the entities, so always return true
  bool addEntitiesCallback(IEntity e) {
		m_entityResults.Insert(e);
		return true;
	}

One feature of the Enfusion Workbench I love (and honestly there aren’t many) is a hyperlinked console log. When you print a raw IEntity with Print(entity) it will focus that object in the game camera, and select it in the hierarchy. This way you can easily confirm whether all the filtered objects are indeed the type you’re looking for.

Clicking on the purple text will highlight that entity

Now we have our list of IEntity instances, we can write their position to a simple JSON file using the FileIO module.

FileHandle fh = FileIO.OpenFile("$profile:data.json", FileMode.WRITE);
if (fh) {
  fh.WriteLine("[");
  foreach(IEntity foundEntity : m_entityResults) {
    // Write a position array
    vector position = foundEntity.GetOrigin();
    string formattedLocationLine = string.Format("  [%1, %2, %3],", position[0], position[1], position[2]);
    fh.WriteLine(formattedLocationLine);    
  }
  fh.WriteLine("]");
  fh.Close();
} else {
  Print("Failed to open file for writing");
}

And out pops the array of coordinates, ready for us to plot using LeafletJS.

[
  ...
  [3215.72, 4.00446737, 2948.74],
  [3231.16, 0.0918469, 2955.15],
  [3242.65, 0.20875, 2894.48],
  [3229.39, 3.1775, 2915.56],
  [3179.32, 9.6688, 2834.27],
  ...
]

So now we can simply import this data into our LeafletJS setup and plot the X and Z coordinates.

The finished map #

Our marked locations

Success! We have our web page which accurately displays the location data, and everything is coming directly from the game engine 100% procedurally.

You can find the code on GitHub in nickludlam/EnfusionMapMaker, and a readme that takes you through most of the steps.

For the final website I took things a little further, using Svelte to provide page templating and deployment options. There were a couple of challenges relating to making the client-side LeafletJS library play nicely with Typescript and building a static version of the site, but that’s another thing which is out of scope for this particular article. The public repository contains the vanilla HTML and JavaScript to get everything working, and people can customise it as they like.

The current live implementation is https://reforger.recoil.org.

Some topographic fun #

Out of curiosity, I wanted to see what it would be like to combine the very flat composite map (effectively pure albedo) with the large-scale shading from the in-game base map from earlier. Simply compositing the layers using an overlay blend mode and some manipulation of the brightness ranges of the shaded map achieves the effect. I think it gives you a much better sense of where the mountainous areas are with the contrast in luminance and a stronger sense of the shallow water. It’s not physically realistic but it looks great.

Compositing the two maps to get an exaggerated shaded version. Click to see the full-size image

Comparison with the official map #

This shows a comparison between the official render of the island, likely based on the 2022 Houdini work, and my generated map tiles.

As you can see, the original topography and field layout is identical, with only the forest zones being changed. This could be down to performance or gameplay considerations.

Conclusion #

This was a fun little side project and took around 10 days in total, plus a little extra for this write-up and the git repository documentation. The website has proved to be very popular with players and now has hundreds of daily users, which makes me happy.

I think this is also a great project to highlight T-shaped skills. This kind of challenge can help encourage developers to push outside their comfort zone, and they can discover that their skills are more easily transferred into other domains than they may have first thought. You may not be a professional game developer, but that doesn’t stop you from picking up enough to achieve what you want from a game engine.

It’s also a good example of sticking to an MVP deliverable. It’s no fun adding another half-completed project to the pile, and there’s no shame in taking some shortcuts to get your work out the door. You can always return later and improve things incrementally.

Many thanks to Tom Armitage and Ian McEwan for feedback and help with this write-up.

UPDATE: It’s nearly a month after the site went live, and it’s served ~2M requests to well over 10,000 players! The map tiles have also been incorporated into two other projects, and you can see them incorporated into the post-match reports at the Reforger Battle Royale Mod.

]]>

Custom Dreambooth Training For Stable Diffusion

Fri, 11 Nov 2022 13:22:51 UTC

The ML image synthesis topic has always been interesting, but it’s exploded since August this year, when Stable Diffusion was made open source, for anyone to try. Since then, I’ve been running a copy of Stable Diffusion locally on my NVIDIA 3070, using the WebUI from Automatic111.

I actually started out trying to get things working on my previous graphics card, a AMD 6700 XT, but the support and performance gap between CUDA and ROCm within the pytorch framework is vast, especially because I don’t have a dual boot system to natively use Linux, where support is much better. I ended up picked up a second hand 3070 and it’s been plain sailing, but only 8GB of RAM is a little restrictive.

Training on Google Colab #

A little while later, a paper was published by Google on a technique called DreamBooth, which allows for additional training and tuning of text-to-image models. People started implementing this on top of Stable Diffusion, but it started out slow and difficult to run on modest hardware.

In recent weeks people have been improving the original approach, finding optimisations to lower the time and hardware requirements. It’s reached a point where I wanted to try it out, so I ran the process with https://github.com/TheLastBen/fast-stable-diffusion/blob/main/fast-DreamBooth.ipynb.

I assembled about 20 pictures from my photo library, cropped and blurred where appropriate, and resized to 512px square. I then supplemented them with a few specific photos I took against a white background, at various head angles. After reading some of the discussions, I also decided to run them all through convert -flop to create horizontally mirrored copies. I don’t know if this step is required, but it’s what I chose, given training data is more important than additional training steps.

The input training data I used, with an overemphasis on the head

You must also obey the naming convention for these images, with the format (%d).png, so I used the following bash script:

counter=1
for i in *.png
do
  cp $i "nickludlam (${counter}).png"
  let counter=counter+1
done

Which made each image conform to the format. I uploaded these to Google Drive, in a folder called DreamBoothTrainingImages/.

I also decided to pay for the instance. You can run it for free, but risk termination at any point, so the prospect of losing 3 hours of work was definitely worth the 3 credits I ended up using.

NOTE: I would actually add more variety to this if I went through the process again. In testing out prompts, it very strongly wants to focus in on my head, and it’s difficult to get images that show shoulders or upper torso. In order to make wider compositions more likely to be created, you need to train it with photos of similar poses.

Google Colab #

When running the notebook on Colab, there are a few buttons that need clicking, and prompts that needed filling in. Although the notebook is fairly well documented, not all of it was clear, so here’s a summary of what I entered into the page:

Variable name	Value
Huggingface_Token	[copy from Huggingface]
Session_Name	nickludlam
IMAGES_FOLDER_OPTIONAL	/content/gdrive/MyDrive/DreamBoothTrainingImages
Contains_faces	Male
Crop_images	Unchecked
Training_Steps	[should be set according to their suggestions, but I found that this caused overtraining, possibly because of my mirrored duplicate images]

I left everything else as default. The repo is being updated frequently, so double check everything you’re typing in, as things might have changed by the time you run this.

Training #

The training is split into two halves. First is the text encoder training, and then comes the unet training. For me, the first stage took about one hour, and the second stage about two hours. You’ll see an output like this:

Training the text encoder with regularization...
'########:'########:::::'###::::'####:'##::: ##:'####:'##::: ##::'######:::
... ##..:: ##.... ##:::'## ##:::. ##:: ###:: ##:. ##:: ###:: ##:'##... ##::
::: ##:::: ##:::: ##::'##:. ##::: ##:: ####: ##:: ##:: ####: ##: ##:::..:::
::: ##:::: ########::'##:::. ##:: ##:: ## ## ##:: ##:: ## ## ##: ##::'####:
::: ##:::: ##.. ##::: #########:: ##:: ##. ####:: ##:: ##. ####: ##::: ##::
::: ##:::: ##::. ##:: ##.... ##:: ##:: ##:. ###:: ##:: ##:. ###: ##::: ##::
::: ##:::: ##:::. ##: ##:::: ##:'####: ##::. ##:'####: ##::. ##:. ######:::
:::..:::::..:::::..::..:::::..::....::..::::..::....::..::::..:::......::::

Progress:|██████████████████       | 73% 2136/2940 [53:35<20:07,  1.50s/it, loss=0.476, lr=6.2e-7] nickludlam

The final step in the process is to convert the training data into a checkpoint file and copy it to your Google Drive. In this case, it’s:

My Drive/Fast-Dreambooth/Sessions/nickludlam/nickludlam.ckpt

Download this file and drop it into the models/Stable-diffusion/ directory inside your automatic1111 repository installation, and run the UI, and your checkpoint will be available in the top checkpoint dropdown.

Don’t forget to spin down the instance once you’re done!

Testing #

I experimented with a number of prompts based on some posts I found online:

One of the biggest issues I had was finding the right balance between Sampling Steps, CFG Scale and word emphasis within the prompt. If one of the suggested prompts is the following:

photo of nickludlam as an astronaut, glasses, helmet in
alien world abstract oil painting, greg rutkowski, detailed face

I found that my net was overtrained, and would just repeatedly make images of the training data. One way around this was to emphasize the target words in the prompt. So for the above, you wrap key terms in round brackets to become:

photo of nickludlam as an ((astronaut)), glasses, (helmet) in
alien world abstract oil painting, greg rutkowski, detailed face

You can add more brackets for additional emphasis. I also tried many of the different sampling methods. It took a LOT of experimentation to get some nice images out, so be patient.

As you can see, I achieved some nicely varied results, given enough time and patience with prompt crafting.

Overtraining #

When attempting to use the newly trained checkpoint file, I often find that it prefers to give me results which just feature a head, even when I’ve asked for an image with shoulders or upper body. As mentioned earlier, I would definitely include more variety in the stance from the subject.

I’ve also seen my face present in a lot of other images of white males, even when my specific keyword is not in the actual prompt, which I believe is a strong indicator of overtraining.

A portrait of Jeff Goldblum from StableDiffusion v1.5

A portrait of Jeff Goldblum from my trained checkpoint

A portrait of Brad Pitt from StableDiffusion v1.5

A portrait of Brad Pitt from my trained checkpoint

A portrait of Tom Cruise from StableDiffusion v1.5

A portrait of Tom Cruise from my trained checkpoint

I tested this by using the same conditions for both checkpoints, for instance the middle one is derived from:

brad pitt wearing a tuxedo, portrait, highly detailed, digital painting, artstation,
concept art, sharp focus, illustration, art by artgerm and greg rutkowski and alphonse mucha

Steps: 30, Sampler: Euler a, CFG scale: 7, Seed: 3120218309, Size: 512x512

Conclusion #

Overall I’m really happy with the results I can get, but there are parts of the process I’d do differently next time.

Firstly, as mentioned above, there was not enough variety in the training data. Having photos where I’m further from the camera is important. Some full-body and upper torso shots would help with the variety and creativity of prompts.

I’m also unsure whether using flopped images is as necessary where you’re effectively symmetrical. It is likely still useful where you’re looking to the left or right, where a horizontally flipped image will look sufficiently different. This wouldn’t work if you have facial features like moles or anything else which would be obviously flipped too.

I would also take advantage of the ability to test checkpoints and resume additional training as desired. That way you can stop when the results are at the right balance between accuracy of rendering your likeness, but yet it retains creativity in composition.

Other writeups #

There’s a very detailed writeup of the same process on bytexd.com, and an additional writeup from huggingface that goes into the differences between this technique and the original textual inversion, and lots of images showing the effects of different learning rates.

]]>

Playdeo Part 3 - Technology & Tools

Mon, 13 Jun 2022 16:43:00 +0000

Work at Playdeo involved solving unique challenges, and I’ve unpacked some of them in a little more detail here. Below is a playthrough of Episode One in the game, to give you a sense of what the game looked like.

Smart, double-buffered video #

In Playdeo’s games, we compress and linearise hundreds of separate video clips into one large MP4 file, and when we need different camera angles or video sequences, we ask the video player to play from different timestamps in the file.

An example of the internal architecture of our MP4 files

There are two classes of clips. Sequences are clips of narrative video, usually featuring dialogue, and they are typically played back in order, and only once to the player. They tell the story. The other class of clips are Camera Coverage, and they are typically environmental shots, and are designed to loop. These are the clips we use when you are navigating Avo around the world.

During our prototyping phase, we were mostly concerned with the content and interactivity of the visual frame, and not so much about narrative and story. All that changed with our Alpha 2 prototype of Avo, and for the first time we had an actor, a script and a need to edit different takes and camera angles together into a coherent timeline. This instantly presented us with a problem; Moving video playback to different portions of a single video file with a seek command was much much slower when there’s an audio track. Without audio, the seek response is near instantaneous, and with audio, it drops down to about 0.2s. That might not sound a lot, but it’s extremely noticable when playing the game, and each camera cut or edit freezes the frame before delivering new frames.

The solution was to run two video decoders in parallel, both reading from the same MP4 file. Modern iOS hardware made this feasible in most newer devices and for the older iPhone and iPad models we were careful to leave an option to disable the second video player entirely. This allowed us to support devices like the original iPad Air from 2013, at the expense of the improved edits. It left a small pause in video playback each time there was a cut, but it didn’t detract too much from gameplay.

An example of how each player takes it in turns to play the video clips

The system internally was known as AV2, and made use of an asynchronous queue system, and two consumer classes representing each underlying instance of the video player. You could either request video be played immediately, or after the current clip has finished, in more of a playlist fashion. It would also be smart enough to understand that when asked to play a clip that immediately followed the currently playing clip inside the MP4 file, it would simply let the playhead run on into the new video in the currently active player, saving a swap to the inactive player, and creating a smoother experience. To put that another way, if Clip A, B and C are adjacent within the video file, and you asked to play them all back, instead of splitting it up across the two different video players, it plays all three back to back in the same video player.

Clips played back in a naive manner, distributed between video players

Smart clip playback, where consecutive clips are played by the same player

The other further development we made was to give the video playback plugin complete autonomy over when it should either loop back to a starting frame, stop delivering video. On mobile devices you commonly suffer interruptions that stop your game from being the primary application in focus. These interruptions would cause problems with video playback synchronisation, and when you return from an interruption the video player might have advanced beyond the limits of the clip you wanted to play. Because the video texture is inside Unity is marked as External, if the playhead of the video player has strayed beyond the clip it was supposed to play, the video frame overwrites the previous texture in memory. The players see flashes of video frames that are out of sequence, and it destroys the gameplay experience.

To get around this issue, every time we request certain frame ranges to be played back, the video player creates guard zones it can autonomously respond to during playback with no connection to Unity. AVFoundation on iOS has this incredibly useful function called addBoundaryTimeObserver(forTimes:queue:using:) for exactly this use-case.

The video plugin grew in complexity over the lifespan of Avo, and thankfully AVFoundation is an incredibly rich media API to sit on top of, which prevented this code from spiralling into something much more complex and time consuming. We also got an Android build running with a custom video player we implemented in house, but this never saw the light of day in a full release, sadly.

Bluetooth latency and PathEDL #

A few months after getting the double buffered video player system working, Apple announced the removal of the headphone jack on their iOS devices, and so started a huge increase in the number of bluetooth audio devices in everyday use. As these bluetooth devices became more commonplace, we started to use them with our laptops.

For the longest time, I was puzzled over the variable seek response time for video in our system. Sometimes video would be quick and responsive, other times it would feel sluggish and slow to react. It was a number of weeks before I put two and two together and realised it was my own use of Bluetooth headphones while programming that would be the root cause of this change.

As mentioned in the Smart, double-buffered video section, there’s an inherent latency of around 0.2s for video with an audio track. Bluetooth audio adds its own latency on top of this, and can vary between 0.2s and 0.3s depending on the manufacturer. So during gameplay, we need to account for a delay in any request to play new video from 0.2s up to 0.5s. This presented a problem for us in the way Avo’s movement triggers different camera angles.

Alternating between Players, including the variable pre-roll depending on audio latency

Luckily on iOS it’s very easy to retrieve the exact value of this latency from the shared audio session:

static float GetAudioOutputLatency() {
    return [[AVAudioSession sharedInstance] outputLatency];
}

So the total latency in the system is equal to 0.2s + AudioOutputLatency. Avo’s line drawing method of movement is actually the perfect uniform system for accomodating future predictions. If it takes us 0.4s to swap cameras as Avo moves, all we need to do is make Avo’s trigger point be constantly ahead of him by 0.4s, thus triggering perfectly by the time his legs catch up to the future prediction.

Bluetooth latency demonstration.

In the video above, there’s a demonstration that shows the difference Bluetooth audio makes to the latency. The white cube in front of Avo represents the distance from his current position that directly correlates to the video seek delay. The longer the audio latency, the further away the cube is, and the faster Avo moves, the further away the cube gets.

For Avo’s standard speed, the cube is roughly one width away from him when not using Bluetooth. This then increases to roughly two widths of the cube when Bluetooth audio is used. This is a small difference at his relatively slow standard speed, but within the game we give him a little speed boost every time he collects a jelly bean. This means his speed can be much greater depending on the situation. This means that the future prediction distance is correspdondingly much larger.

As my work on accounting for camera change latency started to bear fruit, it became apparent that we now had an adjacent problem to contend with. If you happened to draw particular lines which interacted badly with our camera zones, like drawing multiple loops that graze a closeup camera, the resulting video as he walked the line was dizzying, as it would rapidly cut to a closeup, then back out to a wide shot, over and over again. Along with that particular extreme scenario, there were numerous instances of these bad lines, and the player had no way of knowing if their path would look bad while being traversed. We needed a generalised solution.

The answer was once again in the determinism of the line drawing movement system. Since we only allow players to draw a line which is navigable, the line is a perfect representation of the future. This line will be walked, and the only thing that can change is the speed at which it’s traversed.

Up until then, the camera system was entirely based on collider triggers. When a collider entered into a zone, a camera change would be triggered immediately. The latency solution meant that the collider causing the change was in advance of Avo’s current position by 0.2-0.5 seconds, but it was still real-time. We needed to move to a solution where the entire line was simulated, and every future camera change collected by a system I called PathEDL.

In video editing, the Edit Decision List represents the desired arrangement of video frames that an editor wants to arrange for playback. In our case, every line drawn by the player would be its own little video edit, with a beginning a middle and an end. If we sample the points down the line, and correlate them with the camera zones they pass through, then we can form a list of all potential cameras to use for every point along the line.

We can then run a set of filters for these points, and derive a final set of camera cuts best suited to frame Avo as he walks, or in some cases zooms down the line.

The PathEDL filter system in the Unity editor

In the above image, you can see an example of the PathEDL system at work. In the right pane, you can see the geometry of the level, including the camera visibility zones, and his currently drawn path as the standard dotted black line.

In the bottom left, you can see the timeline which give you a visual representation of all the cuts that will occur. The overall duration for this line is 6.9 seconds, with 5 different camera angles that will frame Avo.

The key to this system is that it applies rules not only spatially, but across time as well. In the top left, you can see several filters that are configured to apply to Avo’s movement path.

Filter Name	Description
Start Zone Minimum Time Filter	Ensures that we dwell for a minimum of 0.4s before making our first cut. This helps the player understand the first cut with good spatial awareness
End Zone Minimum Time Filter	Ensures that the final camera cut that will occur for this path is at least 0.5s long. It provides a sense of stability and finality to the sequence of cuts that will frame Avo's movement
Remove Short Edits Filter	Removes any camera edits within the PathEDL that would last for less than ~1 second. This helps make Avo's movement feel less frantic
Select Alternate At Screen Edge Filter	ensures that short lines drawn at the edge of the screen select any cameras that are available other than the existing one. This means people can escape being trapped in a view they do not want
Exclude Camera Zone Type Filter	Ensures that short lines drawn at the edge of the screen select any cameras that are available other than the existing one. This means people can escape being trapped in a view they do not want

As you change or toggle these filters, you get a live preview of how it will affect the edit, and construct a filter set that makes appropriately player-friendly choices. This had an absolutely transformative effect on the game, and walking around felt far less unpleasant and awkward.

One of our developers Geraldo Nascimento really took the initial idea and extended it with a lot of great looking visualisation tools and quality of life improvements for later work.

The Sequencer #

The other distinct aspect of working with video inside Unity was a matter of timing. The video playback code was entirely separate from Unity, and so we needed to integrate video-time with Unity-time in a seamless way. Jon had previously used a node-based system used by UsTwo games on Monument Valley, and suggested we use something similar. We ended up with a tool we called the Sequencer.

A typical Sequencer graph

The graph starts executing from the top left node, and continues down each connection as each node finishes. Nodes can either be of type void and span no frames, or of type enumerator and span one or more frames. The purple nodes with thumbnails show the Play Video nodes, where additional node chains can be triggered by reaching specific frames in the video.

This allowed us to seamlessly mix player events and video events during the level design process, and leave the design open, so we responded to the flow. We could change something that was previously triggered by the player at any point, and incorporate it to be triggered at a certain frame in the video, or vice versa.

A basic wait node looks something like this:

public class WaitForTimeNode : SequenceNode, ISequenceNodeEnumerator
{
    public new static string ReadableName = "Delay/Wait For Time";

    public float WaitTimeInSeconds;

    public IEnumerator GetEnumerator()
    {
        float startTime = Time.time;
        if (!isPlayerFastForwarding)
        {
            while (Time.time < startTime + WaitTimeInSeconds &&
              !isPlayerFastForwarding)
            {
                yield return null;
            }
        }
    }
}

As we developed this system from extremely humble beginnings, it was clear to us that this architecture was very powerful. It took more and more of a hold on how we thought to construct authored experiences in the level, and we improved it with every title we worked on. Later iterations featured a hugely improved authoring UI, and we also expanded the node flow to allow for looping and light branching logic.

The isPlayerFastForwarding boolean is an interesting little feature related to two things. One was having the ability for players to skip video sequences during gameplay, and the other is level state. When we started to work on Avo, we had little to no idea about the player’s game progress. Would we need to store the state of doors and keys, for instance? What about spawned enemies? A player inventory perhaps? We had no idea!

In order to keep it extremely open and flexible, we based our solution around authored checkpoints. In any given level, a number of linear checkpoints are created, each of which might be to a sequence that was run to mutate the level state. We store which checkpoint a player has reached, and upon a load of that level, we step through each checkpoint and play the associated sequences until we hit the last one, and let the player carry on. This way we can leave level designers complete autonomy, as long as progression-based state mutation is always handled by the sequencer.

The video pipeline #

As the amount of data we had to process was undergoing a dramatic increase, we also had to keep on top of the data pipeline. During my years working at Framestore, I knew that for complex setups using multiple pieces of software, the number one source of issues was human error. It’s a balancing act. A pipeline that’s too restrictive will make unforseen errors cost a great deal of time, as they bottleneck on access to the developers. Not restrictive enough, you let problems roll downhill into Unity, and they become much harder to diagnose.

We refined the ingest process over many months, and initially I was the one person who was responsible for running and maintaining this process, but that slowly shifted, and became a job mostly run by our editors or other developers on the team. This went hand in hand with an increasingly complex process, as we had to accommodate new features like subtitles, compound clips, and post-production software changes. The ingest script grew to about 3500 lines of Python, and contained parsers for FCPXML, Resolve CSV files, PFTrack data, FBXs and video files.

The video data pipeline for Avo

Half the battle with this type of work is to find ways of making concrete cross-references between different files running through different pieces of software. Some software is scriptable, and we made use of this in PFTrack, where an export of the data would also create additional metadata files important for these connections. Sometimes you need to ask for specific naming conventions to be obeyed, and you enforce that during ingest.

Over the course of about 3 months, we gradually went from a situation where I had to run every single ingest step to one where others were self sufficient. A surprising amount of this time was spent refining error messages that would allow the user to diagnose issues themselves. This wasn’t always easy, and required tracking far more data than you’d make use of in the final output, like specific line numbers of input files which might go on to cause issues later.

I class this pipeline as generally successful, as we were able to bring freelancers into the company and train them in using this within a few days. Ultimately this pipeline would be replaced in later games with one based entirely on Playdeo Capture, but during the period before that, the pipeline tools were solid and saved us a huge amount of time.

Playdeo Capture #

After Avo shipped, we could see the value in being able to shrink our video pipeline down. Using three different pieces of software every time we want to set up a new scene, with the photogrammetry and tracking taking at least a day was very awkward. It prevented us from quickly prototyping ideas, and was prone to human error. Our toolset in Unity was slowly growing to make scene assembly and authoring far quicker than before, and we needed to keep parity.

The idea of using ARKit on iOS to be able to record an AR session was an idea we’d had a while back, but it wasn’t obvious if it was feasible. iOS has generally great media performance, but ARKit was still a relatively closed API. We worked with Sam Piggott on a Swift app that would record and store camera positions. In the early days of ARKit, you were prevented from doing much beyond displaying a 3D scene to the user. In order to make a recording system work, Sam figured out a way to peek into the hidden AVCaptureSession owned by ARKit and set up an AVAssetWriter to stream it out while also running the session. Luckily for us Swift has retained the core aspects of introspection and reflection that were present in Obj-C.

private func attemptToRetrievePrivateCaptureDevice(session: ARSession) {
    let sensors: NSArray = session.value(forKey: "availableSensors") as! NSArray
    let imageSensorClass: AnyClass = NSClassFromString("ARImageSensor")!

    for sensor in sensors {
        debugPrint("Checking sensor \(sensor)")

        let parentImageSensorClass: AnyClass = NSClassFromString("ARParentImageSensor")!

        guard
            let sensor = sensor as? NSObject,
            sensor.isKind(of: parentImageSensorClass) == true,
            let captureSession: AVCaptureSession = sensor.value(forKey: "captureSession") as? AVCaptureSession,

            // The first input should be the "Back" camera
            let deviceInput: AVCaptureDeviceInput = captureSession.inputs.first as? AVCaptureDeviceInput else {
                debugPrint("Failed to get AVCaptureDeviceInput")
                return
        }

        underlyingCaptureDevice = deviceInput.device
        underlyingCaptureSession = captureSession
    }
}

This code isn’t necessary with more recent versions of iOS. There are also a number of open source projects that do similar things in slightly different ways.

This workflow was successfully integrated into the entire end-to-end process, and we could produce a working iOS game featuring data recorded on Playdeo Capture. The image quality was obviously not as high as the images from Timo’s high end DSLR cameras, and the ARKit mesh fidelity was lower, but it did allow for prototyping and ad hoc workflows. The original ingest python script was replaced entirely. ffmpeg was now run via Unity. We had successfully demonstrated a workflow which had cut the pieces of software down from 5 to 2.

This project became important during the COVID pandemic in 2020. With all of the studio working from home, producing new video content became impossible. Playdeo Capture became literally the only way we could prototype new ideas, and that’s when Jack started experimenting with Playdeo Makes, episodic video featuring modular tabletop mini games and exercises you play with Avo. Nearly all of this was bootstrapped by Jack alone in his loft, with makeshift cardboard props serving to create a set and series of camera positions.

As lockdown eased, it became possible to have actors on a set, as long as a fairly strict set of guidelines were obeyed. Recent advances in iOS hardware had meant that we could now shoot in 10-bit log at 4k, and it gave a huge uplift to the visual quality, so much so that we needn’t go back to DSLRs. The only restriction was in having fixed cameras, as 10bit mode was incompatible with ARKit recording.

The last innovation that I want to mention is how we arranged our metadata. It was recognised that there was still a role for editing software like Final Cut pro. However, manual logging of this data was slow, awkward and error prone. I rewrote and simplified the way we log clip data, and we standardised on a naming convention of scene, shot, take and camera number.

Because of the original architecture of Playdeo Capture, all data was centralised on the iPhone. For a future version we’d want this performed on the server, but for now the phone held the data model for clips, and the video data. Any logged information would need to reach the phone somehow.

At the same time Timo was stuck in Norway because of travel restrictions. I set up a small rails website to allow shot logging. Timo was present on set via a Google hangouts call, so was able to see and hear what was happening. He could sit in Norway and act as Script Editor and Clapper Loader. He would type the scene, shot and camera numbers into the site, and it would calculate the take. Then in Playdeo Capture every time the record button was hit, we could query the website and fetch the appropriate metadata. This would then be baked into the session on the iPhone.

The connection to the Rails site was done with the ActionCableSwift Swift package, and the relatively complex connection was managed with a large state machine implemented using SwiftState. Both of these are excellent packages to work with, and helped encapsulate a lot of complexity that emerged from the period where this remote data solution was hacked up in.

Shooting on set with virtual presence and metadata connections

Then when it comes to exporting the data form the iPhone we create a fake FCPXML file that simulates a Final Cut Pro project. This meant that metadata would be preserved as it was passed into and out of Final Cut Pro, and we could retain the correct metadata and position data automatically.

Fastlane and the build pipeline #

Early in our prototyping we implemented a Fastlane pipeline to manage the Xcode modification, build and upload process. When Apple updates Xcode, Unity tends to lag behind upstream changes, and their Trampoline system for producing xcodeproj often ends up causing Xcode to display many deprecation and misconfiguration warnings. Fastlane is a sophisticated project that automates as much of the Apple build pipeline as possible, and allows for relatively straight forward programmatic manipulation of Xcode projects, and allowed me a hook to address these configuration issues.

Amonst other things, we used it for the follwing tasks:

Automatically fetch large assets from remote storage
Manage signing identity
Add plugin code to the Xcode project
Automatically set version and build numbers
Provide custom On-Demand Resourcing setup manipulation in Xcode
Process and badge the launch icons to distinguish between build types
Toggle feature flags
Enable/disable the complex analytics code, to reduce dev build times
Auto tag releases in git
Upload builds to Testflight
Post to Slack channels for automatic notifications

It really is a Swiss Army Knife for handling any tasks that it might be too complex or difficult to perform in Unity, with the downside of increasing project dependencies. Where possible, we always made Unity capable of producting functional output for local development, but this ended up slipping behind as the project got larger.

I’ve made a gist of the Fastlane action, Unity C# class and example usage. It’s a cut-down version of what we used, but the core essentials are present. It’s relatively easy to create your own custom command-line arguments to create a structure for build manipulation.

The rest of the build setup was based on a standard configuration of a CI server. We initially used Go CD, and later swapped to Jenkins as others in the team were more familiar with it, with installation and maintenance therefore easier to distribute among more people.

We found Go CD suffered from quirks, like an obscure bug relating to how the servers and clients are built and run as GUI tools from the Dock, leading to a pollution of the environment. In this case, to fix a bizarre error you had to ensure you unset CFProcessPath early on before running Unity. This is an oddity that dates back to the changeover from Carbon to Cocoa on the Mac, and was a particularly ancient and difficult to pin down issue.

Paying attention to your build process early allows the project grow in complexity without it becoming unwieldy over time. If you have organised structure where you can hang additional code, it helps guide people into solutions and amendments which hold up over the lifespan of the project, and a more predictable place for methods and files. Do ensure that the right people review code changes, however, as build system fragility is not always obvious until the build server breaks!

Metasploit #

Although not strictly part of the technology, sometimes you get to contribute to the authenticity of technology’s representation in media.

The metasploit output in the game

This is a geeky easter egg inside Avo, and I’m not sure if anyone ever noticed. For the hacking scene, I took the real console output of Metasploit and rewrote it to be more in-keeping with the Avo universe. The actual activity run inside it was the real OpenSSL Heartbleed exploit from April 2014.

The raw Metasploit output

The other component was a small Ruby script which allowed any keyboard interaction to trigger ASCII text to be written out to the terminal. This would give Katie an ability to type anything on the keyboard and produce sensible output, giving the impression of an expert as Billie would be.

]]>

Playdeo Part 2 - Building Avo

Thu, 05 May 2022 14:39:48 +0000

This post follows on from part one of the Playdeo story.

Alpha 1, Jan 2018 #

Our previous game prototype Sharpen was then followed by a more fleshed out idea codenamed Tiny Frankenstein. A controllable pencil sharpener didn’t offer enough innate personality, but cemented the idea for inanimate objects imbued with life. We’d need to bring more charisma to make a truly great game protagonist that the player would form a bond with.

We auditioned a lot of different fruit

I do not recall exactly where the idea came from, whether it was Ryan North’s story, or Jack and Jon’s choice, but out of all the different fruit we photoscanned, we chose the humble avocado, and thus Avo was born. The new script kept the core ideas, but now incorporated a script and narrative, where the fetch quest was given to you through dialogue rather than on-screen text as we had in Sharpen.

The basic Alpha 1 S-shaped desk

Above you can see the basic layout for Alpha 1, with a more specific lengthy area for you to walk around. Jack was still playing the role of the scientist who brought our protagonist to life, and the set was made of basic white desks. As we filmed and produced the prototype, it became obvious something was missing.

Mechanically we proved the game idea could work, but this was a rare case where the technology outpaced the film making. It became obvious that we needed to start looking for a professional actor, a more visually interesting location and more lavish props for you to be surrounded by.

Alpha 2, Feb 2018 #

Alpha 2 featured a custom built set and actor Katie Reece as our hero inventor Billie. We also saw an early version of Avo with his procedural walk system, custom props, and a much stronger dialogue and narrative. It was an immediate win, with a far more engaging interaction, plus Avo’s own quirky personality was beginning to shine through.

Jack in Alpha 1, Katie in Alpha 2

In an effort to keep development simple and flexible, our protagonist Avo was silent, taking influence from Wallace and Gromit. Since we now had dialogue and a story, we needed to edit and grade footage in Final Cut Pro before feeding it into Unity. This was a huge step for us, as it was the first time we had meaningfully used an audio track from the video to convey information to the player.

Avo started out life with just his legs in Alpha 2

Alpha 2 got more and more polished and began to generate its own gravity. We added flourishes like the collectables, which helped guide players in where they needed to take Avo. It felt exciting, fun, full of heart, and was finally something we could scale up into a full game.

Ryan North and Gemma Arrowsmith helped us create a fun and engaging story. It ended up being wildly ambitious and needed scaling back, with one or two chapters going unfilmed, but the bones of it were there. What followed was location scouting, set building, more props, full script development, table reads and all of the usual aspects of a full TV production, except done on a tiny budget, and very much in the guerilla, forgiveness-not-permission, school of filmmaking. This was all managed by the talented Lotta Boman who excelled under these resource constrained circumstances.

Alpha 2 Sequencer and Checkpoints tools

While Alpha 2 could be considered a success, many of the systems used to pull it together were not robust or scalable. This included aspects of narrative construction like the Sequencer and our save/load system based around checkpoints, as shown in the screenshot above. If we were to require a full game with many levels, our tools needed to become far more capable, and streamline many of the repetitive tasks involved in their construction.

The Shoot, May 2018 #

We’d now finished the invention phase, and tipped into production, polish and delivery. Filming started in May 2018, and continued for approximately 10 weeks. The studio was divided in two for the duration of the shoot, half on set, and half back in the studio preparing for a tranche of video to suddenly turn up, feverishly working on mesh alignment tools, level creation tools, the video processing pipeline, a data-binding UI system, integration with the Wwise audio system, and a whole host of other elements of technical debt.

I was split across a number of different areas, as well as providing overall technical leadership for the team. By far the most urgent task was supporting our data pipeline. We were suddenly generating a huge amount of video footage, more than anything else we’d previously experienced, and it all needed editing and processing. We swapped over to using Black Magic’s DaVinci Resolve rather than Final Cut Pro. Tracking was still being done in PFTrack, and the photogrammetry in Reality Capture.

Timo and I defined a new data pipeline that correlated PFTrack exports, DaVinci’s exported FCPXML timelines, CSV metadata, and the video files to form a sufficiently robust mix that would cross-reference frame counts, filenames and a host of other data to spot and highlight any human error that crept in. As we were doing ten times as much data processing as before and involving people new to the process, this was incredibly important. Bad data that makes its way into a test build needs to be easily distinguishable from errors in the code itself, as both were changing frequently, and it’s very costly to involve the whole team in diagnosing build issues.

The Sprint, September 2018 #

From September 2018 to January 2019 we implemented the eight episodes in the main game. We added subtitle support, full music and sound effects, bluetooth audio support, the save checkpoint system, localisation, analytics, general UI, IAP integration, On-Demand Resource support, AR mode, low and high resolution videos, and a whole host of other things. We had no specific producer, so our weekly planning meetings were crucial for establishing bottlenecks, and towards the end I was generally responsible for keeping the flow of work steady, as it became more and more technical. It was a remarkably intense time, and for the most part highly productive.

Our git commit graph. The dotted line is our launch day

As well as the data pipeline, I also worked on a whole host of game features. The video playback system needed constant improvement as we attempted more ambitious edits, transitions and audio layering. I also worked on any areas where we integrated with native iOS functions. This included In-App Purchase integration, On-Demand Resource fetching and optimisation for less capable iPhones and iPads.

Just before launch I managed to implement the PathEDL system, which was a way of mitigating the lag experienced when playing the game with Bluetooth audio. This was becoming increasingly common with the launch of AirPods at the end of 2016, and became crucial to maintaining a good feel for Avo’s line walking, as we had to increase his walk speed to counter the larger set we ended up building.

The launch, Jan 2019 #

We finally launched Avo at the end of January 2019. It has gone on to have nearly four million downloads and is regularly promoted in the App Store at the time of writing this in 2022. For a new studio’s first title in a new medium, in I consider those numbers a huge success. While it may seem coherent and polished from the outside, it really was a hard won product born from three years of inventive exploration in a brand new medium.

It also had some nice media attention, from this review by Leo Laporte on TWiT to Dan Hill’s a very in-depth dive into the medium as a whole. One of my personal favourites was a video review and skit from FGTeeV, made using the AR video recording mode I worked on in the game.

As with all startups, you end up wearing many hats, but this broke the record for me:

Lead engineer - initial architecture, setting team goals, 3rd party integration
Data pipeline work - video data, tracking data, subtitles, analytics
Build dev - Fastlane and the CI server
Video specialist - native video playback plugin
Game feel and optimisation dev - developing PathEDL and overall improvements to game input
Asset control and code versioning work - scripting use of the NAS server and main git wrangler
Overall tech team leader - hiring, whiteboarding, troubleshooting, reviews

The ‘Making of’ video, March 2019 #

Timo and Jack organised a behind the scenes video of Avo’s production. I’ve also uploaded a copy to our instance of PeerTube running on recoil.org. It’s a lovely encapsulation of the studio and its work at that time.

There were a large number of people involved in Avo who haven’t got a mention here, as this is told from my personal perspective, and I’ve cherry picked the most interesting aspects of what we did for the purposes of brevity. The IMDB entry for Avo and Moby Games page have a complete list of the cast and crew.

The last article contains some more of the technical aspects of the work. Read part three of the Playdeo story.

]]>

Playdeo Part 1 - Material Exploration

Tue, 03 May 2022 13:23:20 BST

Back in 2016, Jack Schulze, Timo Armall and I set up Playdeo; a studio to develop an original idea of games and interactive experiences using full screen video.

The studio just after the launch of our first game Avo

This was the first time any of us had been directly involved with attempting to build a game, although we’d successfully worked with each other at the design studio BERG on a series of projects involving complex design and cutting edge tech. Jack brought his unique vision and design sense, Timo added his in-depth camera wrangling and cinematic skills, and I had direct experience in native mobile apps, video, and a history working in visual effects houses.

This article covers the period of time from the very start of our idea through to the end of the material exploration phase, where we tipped into producing our first game, Avo.

The spark, 2015 #

Jack started thinking about the possibilities of touchable video after seeing his young kids reach out and touch the screen of their iPads while watching a movie in 2015. Everything else you do with an iPad involves screen interaction, and in their minds video shouldn’t be any different, so of course it would be touchable! Instead, all they saw were the playback controls appearing. This made Jack wonder how true touch interaction with the video picture might actually work. He started making a prototype, working with Greg Borenstein to get things off the ground.

Some time later, Jack invited me for a coffee and showed me his first working prototype running on a laptop. It integrated three key elements; the full screen video, a 3D scene and camera motion tracking. This prototype just about held together. There was no audio, the file sizes were huge, playback wasn’t smooth, rendering was inefficient; the list went on. However, in spite of all that, there was something undeniably magical about it. You could render 3D objects into the video with rock solid believability, plus you could interact with anything on the screen. It worked better than the performance of AR at the time, and was under the player’s control, unlike something pre-rendered.

An early prototype showing the video with underlying 3D mesh

All the core elements were present, but to unlock a viable future for the tech, we knew we had to get a prototype working on mobile devices. This would immediately derisk the project by a huge amount, and since I’d spent time developing native iOS apps before, Jack wanted to know if this was something I’d be interested in working on. There was clearly enough here to show the idea’s enormous potential, so I eagerly hopped on board.

The first thing I worked on was video playback. There’s no way you could ship a product with JPEG flip-books, so we’d need a way of decoding video frames in real time. After three weeks of work, starting in October, I had a very crude iOS prototype working where we could decode true MP4 video frames, and pass that into Unity as a standard texture. Crucially, this also included frame timing information, so we could look up accompanying camera position data when rendering the video frames.

This gave us the confidence to begin scaling up our work, and start working with someone who knew the Unity engine well.

Tooling up, 2015-2016 #

Aubrey Hesselgren joined us over the winter to work on the very first prototypes, bringing his Unity expertise to the team, and helping with many of the core animation and rendering tasks. Our prototypes were crude, and relied on very ad-hoc processes of ingest and data manipulation, but each one gave us insight into what looked good, and what gameplay opportunities were available.

We had to work blind in the Unity Editor, as video decoding only worked on mobile devices. We’d frequently encounter issues such as playback synchronisation problems, involving a lot of trial and error debugging. We did our debugging with high speed video of the phone screens to figure out timing issues. It was crude, but nothing existed within Unity’s standard tools that was geared up to debug this sort of process, so we had to roll everything ourselves. Progress during this period was slow and frustrating.

Slow motion debugging video frames

For the build process we used Fastlane. I disliked Unity’s automatically generated Xcode projects, as it would frequently be out of step with iOS releases, and I wanted a way to manipulate the generated project files independently. I’d seen Fastlane put to excellent work by Tom Taylor in a previous job, and knew it represented the perfect Swiss Army knife for manipulating, building and distributing our prototypes.

Hey presto, our little car appears

The headlights can relight the video

After a few months, we produced the Orange Car demo, some clips from which you can see in the images above. This was our most polished demo to date. It started with Jack speaking to camera, explaining what was going to happen, then opening a box. Out popped a little orange car which the player could control. The car headlights could even dynamically illuminate the objects on the table.

Investment, 2016 #

Armed with the Orange Car demo, we formed a company. Jack, Timo and I felt the demo showed the amazing potential of what we could make. The idea was easy to understand and, more importantly, it ran directly on a phone. Off the strength of the demo, we sought some initial investment to start scaling our work up. Chris Lee joined us as a fourth founder, bringing his important games industry knowledge.

We settled into a co-working space in Whitechapel, East London. Having spent much of the previous time camped out in Timo’s mother’s front room, with three or four of us packed in like sardines, it was a welcome step. It was a noisy and hectic space, and inexplicably our tele-sales entrepreneur neighbours would always love having their loudest conversations just outside our door. On the plus side we had one of the most important pieces of equipment, a huge whiteboard! I believe it’s the intellectual and spiritual hearth for people doing collaborative, inventive work.

Whiteboarding is an amazing way to communicate ideas

In September 2016 one of our developers Yera Diaz completed one of the most important pieces of early work, a video plugin for the Unity Editor. We would finally be able to prototype by simply hitting the Editor’s Play button rather than requiring a whole iOS build to be made before we could see anything working. This would transform the experience of exploring this new medium, and accelerate the pace of our work enormously.

Plenty of time was spent debugging video playback issues like this

In October 2016, we worked with Jonny Hopper and Mike Green from Glowmade to smarten up some of the core code, and to start thinking about gameplay and interaction. We experimented with a platform game, and at this stage were still very much treating the phone like a TV, with landscape orientation and virtual joysticks for control. We were starting to mould the codebase into a space where we could experiment, and to derive a predictable and quick pipeline.

Torchlight experiment with dynamic match-cuts

Clean Up My Mess experimented with physics

Kerbside demo exploring dynamic lights

Soon after, we started working on long string of unique prototypes. Day/Night, Time Travelling, Kerbside, Physics Toy, Clean Up My Mess. These helped bring forward the idea that you were actually playing in video, not just watching it. While each prototype was its own separate concept, all of them explored touch interactions in various ways. Should the phone be horizontal or vertical? Was a virtual joystick really the best way for players to interact? How do you receive feedback?

Exploring touch interaction systems

For some of our experiments, we needed a way of transitioning from a 2D touch to a 3D drag, as can be seen in the images above. Although we made it work, it wasn’t as intuitive as we’d have liked, and we did not find any gameplay interactions that made it feel satisfying, so we kept moving forward with new prototypes.

New offices and a larger team, 2017 #

At the start of 2017 we moved to The Trampery Republic, a workspace in East India Docks in East London, and started to scale up our headcount. This was an exciting moment, but it also put pressure on everybody. We lacked sophisticated editor tools, so all design work had to be through sheer imagination first and foremost. It was particularly tough as we were working with a new medium that lacked a back catalogue of reference material. The traditional tools such as game design documents didn’t work, as there were so many technical limitations, and no obvious genre to aim for. This was probably the most difficult time, with far more experimental failures than successes.

The Night Garden prototype with Tolla

By April 2017 we started working with Super Spline Studios and Shanaz Byrne on a project we called Night Garden, featuring a character called Tolla (seen in the image above). It was our first time experimenting with humanoid animations, inverse kinematics, enemies and a whole slew of other features.

It was an on-rails runner, with some (but limited) control over where the character was positioned, and a single long take of a camera’s path through a garden environment. Ultimately we didn’t take it forward as we felt there was not sufficiently diverse gameplay or replayability, but we were slowly improving our capabilities and ambitions. This was definitely the closest we’d come to an actual gameplay loop.

It’s at this point we’d fully committed to the vertical orientation as our preferred way of holding the phone, and using a single finger for most interactions. It was the right balance between interaction, comfort and screen visibility. This was before the rise of Tiktok, YouTube stories or any other large scale proof that vertical video would be accepted by our audience.

Racing prototype with camera cuts to reveal more of the world

In August 2017 we briefly went back to the idea of controlling small cars on a race track, similar to the original Orange Car demo, but now shot vertically with single finger steering. This featured a small but crucial new facility for us; cutting to a different camera as you approached the edge of the screen. Although this brief exploration of racing wasn’t taken forward, the idea of cutting between cameras would stay, allowing the player to explore the 3D space under their own control.

This prompted all kinds of interesting questions about continuity and video time vs game time. That is, the player experiences each use of a video clip in a strictly linear fashion, so we had to be careful with anything that visually changed the world. For example, if we showed someone putting down a cup of coffee, then each subsequent clip we used had to show the coffee cup on the table, or we needed to show a clip of someone picking it back up. We learned that video clips that were designed for reuse must try to refrain from changing the set in any way. This was a big progression in understanding which impacted on our later work.

Game prototypes, 2017 #

The next important step in our prototyping was made by Jonathan Topf. Because of his work on a previous game he created, Trickshot, he had a good feel for players using a touch screen as the primary control system. His insight was to allow players to draw an intended path of movement for the character, rather than manipulating indirect controls like a touchpad or virtual joystick. I remember being really impressed at the time, and knowing this method of input was right for our game.

Jon's line drawing prototype

It was obviously a breakthrough at the time

It’s a fantastic mechanic for player control on mobile, but it wouldn’t have made as much sense without the ability to change camera angles when approaching different regions of the table. In creative companies doing good work, breakthroughs should happen all the time, in lots of different areas. The key is to communicate these breakthroughs, and allow new possibilities to be unlocked at all points up and down the technology stack.

With the new draw-to-move mechanic, we made our next prototype with it in mind. As lines were easiest to draw on large, flat planes, setting the game on a tabletop would allow for easy filming, good opportunities for interactions between human and virtual characters, and easy player navigability. We were already drawn to the idea of small worlds like The Borrowers, but we knew fully rigged and animated characters like Tolla were too complex for us to manage internally, so we settled on a more Yokai idea of inanimate objects brought to life. This led to Sharpen (shown below), in November 2017.

The Sharpen prototype

Players needed to find some items strewn around the play area, giving us our very first taste of tasks in the form of these fetch quests. It also allowed us to get a better feel for drawing lines to navigate, cutting between different cameras, each of which would need to carefully frame the playspace. We also explored the idea of cinematic cameras, which were visually interesting, but did not necessarily work well with touch interaction. We were finding the right balancing act between interactivity and narrative, and this led to us distinctly naming these shots as either Coverage or Sequences. Coverage was used for line drawing, while Sequences were used to tell the story.

We also shot it with a very shallow depth of field, and this taught us an important lesson. The more extreme the DOF effect, the less stable the motion track as a result, meaning some video clips in the demo were very low quality as a result. If we were going to make a lot of these camera angles work, we needed to be far more modest with the focal plane depth and position.

When we had a retrospective on Sharpen, it was clear we had all the core components in place for our first game. These components were: small tabletop worlds, mixed human and virtual character interaction, line drawing, multiple camera angles and basic fetch quests. It was now time to commit to our first game release, but we needed a more charismatic main character …

Continued in part two of the Playdeo story.

]]>

Playdeo

Wed, 30 Mar 2022 22:18:02 UTC

Playdeo was a mobile games studio set up by Jack Schulze, Timo Arnall and myself in 2016 to explore the possibilities of full screen video married with a 3D game engine.

This union was very novel, with almost no precendent for the kinds of interaction opportunities it offered. In situations like these, the work ahead would require a lot of material exploration, and thinking through making. Iteration is key, and in this article I’ll be show some examples of the various prototypes we built, a rough timeline, and the key milestones we reached which finally unlocked our first fully shipped game, Avo.

In writing this piece, there are many aspects of this work which have only become apparent to me in hindsight. Material exploration can be a focused and somewhat myopic state to work in. You’ve not yet got any perspective, as there’s no whole form from which to stand back and assess. You set out to create user-centric and experiential milestones, and assemble a rough technical infrastructure to support it. You want to get to an end-to-end build by taking as many shortcuts as possible.

As humans, we love thinking about these cumulative periods of iterative work in terms of Eureka! moments, or talk of having overnight successes. I was guilty of thinking exactly this, with one key piece of work standing out in my memory of the work as THE moment it all came together, that being the interaction method of line drawing to move your character that Jon Topf came up with.

The truth is that there were numerous key moments spread over years, each usually providing a solid layer on which other work could sit. Sometimes these would be obvious, and sometimes they are more subtle, but each layer created new possibilities. In great teams, everyone contributes to this gradual layering process.

From your own perspective, there’s also a natual tendancy to discount your own work as less important than others, but that’s your own bias talking. What you’re working on is obviously not going to come as some sort of pleasant surprise. It’s also important to be humble and let work speak for itself. From an outsider’s perspective there will be many contributions from all corners that unlock a complex finished product, each important in its own right. Letting in a culture of rock star contributions is neither accurate nor healthy for long term teamwork.

So here we’ll see how we worked our way from a smoke-and-mirrors demo on a laptop through to a shipped iOS App Store game with 4 million downloads.

The Spark #

Back in 2015 Jack had been thinking about the possibilities of touchable video when he saw his young daughters reaching out and touching the screen on their iPad while watching a movie. Everything else you do with an iPad involves screen interaction, and in their minds video shouldn’t be any different. This make Jack wonder how true touch interaction with the video picture might actually work, and he set to work making a prototype. This was before my direct involvement, and I think he worked with Greg Borenstein to get things off the ground, having previously worked successfully with Greg at BERG.

Some time later, Jack invited me for a coffee and showed me his first working prototype, running on a laptop, that integrated 3 key elements together; the video, a 3D scene and camera motion tracking. The prototype barely held together. There was no audio, the file sizes were huge, playback wasn’t smooth, rendering was inefficient, the list went on. However, in spite of all that, there was something undeniably magical about it. You could render 3D objects into the video with rock solid believability, and you could interact contextually with anything on the screen. Better than the performance of AR at the time, and under the player’s control, unlike something pre-rendered.

The core elements were present, but to truly unlock a viable future for the tech, we knew we had to get a prototype working on mobile devices. This would immediately derisk the project by a huge amount, and since I’d spent time developing native iOS apps before, Jack wanted to know if this was something I’d be interested in working on. There was clearly enough here to show the enormous potential, so I eagerly hopped on board.

The first thing I worked on was video playback. There’s no way you could ship a product with JPEG flip-books, so we’d need a way of decoding video frames in realtime. After about 3 weeks of work in October 2015, I had a very crude iOS prototype working where we could decode true MP4 video frames, and pass that into Unity as a standard texture. Crucially, this also included discrete frame numbers, so we could look up the accompanying camera metadata when rendering the video frames. This gave us the confidence to start scaling up our work, and start working with someone who really knew the Unity engine. While I’d picked up just enough to get this small breakthrough working, Unity is a vast system, and we needed someone who was comfortable sketching and developing with it.

Throughout late 2015 and early 2016 Aubrey Hesselgren joined us on the very first prototypes. These were crude, and relied on a very ad-hoc processes of ingest and data manipulation, but we began to get a feel for the challenges ahead. Builds of our test code would only work on actual phones rather than inside the Unity editor. Building and deploying was slow, and overall iteration time was painful. We’d frequently have issues like playback synchronisation problems which involved a lot of trial and error debugging. To diagnose timing issues, we burned in timecode into the video to be able to check that the actual frame being displayed was the same as the number that was given to Unity by the plugin. Because this system had to run at a faultless 60 frames per second, we often used the ultra slow-motion video recording facility in iOS to check our synchronisation. Looking back now, I’m struck by how crude our debugging systems were, but back then we were at the cutting edge, and anything we needed we had to make ourselves. No part of the existing Unity ecosystem was geared up to help us with our unique problems.

As time went on, I began to bring formality and automation to our data pipeline and build process. The ingest process consisted of an ever growing Python script that formed the spine of multi stage data wrangling. Because we wanted to have near instantaneous access to any video frame, we used ffmpeg to concatenate all the videos together into a single seekable file. We used Autodesk’s FBX Python library to allow us to programatically get keyframe data from the camera track, rather than relying on Unity’s systems which always wanted to smooth this motion out. The script also started to enforce naming conventions to tie all the disparate elements together automatically, and it attempted to identify and reject human error in the upstream processes, and prevent bad data from entering builds. This would help reduce overall debugging time, even if it looked overly fussy from the video post-production team’s point of view.

For the build process we used Fastlane. I disliked Unity’s automatically generated Xcode projects, as it would frequently be out of step with iOS releases, and I wanted a way to manipulate the generated project files independently. I’d seen Fastlane put to excellent work by Tom Taylor in a previous job, and knew it represented the perfect Swiss Army Knife for manipulating, building and distributing our prototypes.

As we got more comfortable in working with this new medium, we focused on produced the Orange Car Demo, shown above. We felt this demonstrated the amazing potential of what we could make in an easily digestable way, and more importantly it ran directly on your phone. Off the strength of this demo, we sought some initial investment to start scaling Playdeo up to a full company. Games industry mogul Chris Lee joined us as a 4th founder, also unlocking the inner mysteries of the game industry, as Jack, Timo and I had not previously worked in this medium. We settled into a co-working space in Whitechappel, East London. Having spent much of the previous time camped out in Timo’s mother’s front room, 3 or 4 of us packed in like sardines, it was a welcome step. It was noisy and hectic, and inexplicably the tele-sales entrepreneurs would always love having their loudest conversations just outside our door. On the plus side we had one of the most important pieces of equipment, a huge whiteboard. I really think it’s the intellectual and spiritual hearth for people doing collaborative, inventive work.

100%

By September 2016 we were working with Yera Diaz to make a Unity editor plugin for video playback. We would finally be able to prototype by simply hitting the play button on our laptops rather than requiring a whole iOS build to be made before we could see anything working. This would transform the experience of exploring this new medium, and accelerate our progress towards our first game.

100%

In October 2016, Jonny Hopper and Mike Green from Glowmade briefly joined us to smarten up some of the core code, and to start thinking about gameplay and interaction. We experimented with a platform game, and at that stage we were still very much treating the phone like a TV. Landscape orientation, with virtual joysticks for control. We were starting to mould the codebase into something where we could truely experiment, and to derive a predictable and quick pipeline.

Quickly after, we started working on a number of prototypes. Day/Night, Time Travelling, Kerbside, Physics Toy, Clean Up My Mess. All of these helped bring forward the idea that you were playing in video, not just watching it. While each one was its own separate prototype concept, all of them explored touch interactions in various ways. Should the phone be horizontal or vertical? Was a virtual joystick really the best way for players to interact? How do you recieve feedback?

For some of our experiments, we needed a way of transitioning from a 2D touch to a 3D drag, as can be seen above. Although Avo never shipped with any draggable UI elements, we did need to blend our 2D and 3D touch thinking for the line drawing mechanic.

New offices, new people #

At the start of 2017 we moved to The Trampery Republic, a workspace in East India Dock in East London, and started to scale up our headcount. This was exciting but also put pressure on everybody, as our tools were still very much at an early stage. If you couldn’t program in Unity yourself, it was tough to achieve anything technically. We lacked sophisticated editor tools, so all design work would have to be through sheer imagination first and foremost, and this is particularly tough when working with a new medium that lacked a back catalog of reference material.

In April 2017 we started working with Super Spline Studios and Shanaz Byrne on a project we called Night Garden, featuring a character called Tolla seen above. It was our first time experimenting with humanoid animations, inverse kinematics, enemies and a whole slew of other features. It was an on-rails runner featuring a character called Tolla, with some but limited control over where the character was positioned, and a single long take video of Timo’s mum’s garden. Ultimately we didn’t take it forward as we felt there was not sufficiently diverse gameplay or replayability, but we were slowly improving our capabilities and ambitions. It’s at this point that we’d fully committed to the vertical orientation as our preferred way of holding the phone, and using a single finger for most interactions. It was the right balance between interactions, comfort and screen visibility. Bear in mind this was before Tiktok, YouTube Stories or any other large scale proof that vertical video would be accepted by our audience.

In August 2017 we briefly went back to the idea of controlling small cars on a race track, similar to the original orange car demo, but now shot vertically with the single finger interaction. This featured a small but crucial new facility for us, cutting to a different camera as you approached the edge of the screen. Although this brief exploration of racing wasn’t taken forward, the idea of cutting between cameras would stay, allowing the player to explore the 3D space under their own control. This threw up all kinds of interesting questions about continuity and video time vs game time. That is, the player experiences each use of a clip of video in a strictly linear fashion, so we had to be careful with the clips mutating global state. If we use a video clip where someone places down a cup of coffee, then each subsequent clip we use must show the coffee cup on the table. Video clips that were designed for reuse must be neutral as much as possible.

The next stage of prototyping was unlocked by Jonathan Topf. Because of his work on Trickshot, he had a good feel for players using a touch screen as the primary control system. His insight was to allow players to draw an intended path of movement for the character, rather than manipulating indirect controls like a touchpad or virtual joystick. I remember being really impressed at the time, and that this method of input was right for our game, and I commented on the commit in our Slack.

We shot a demo called Sharpen which featured a player controlled inanimate object, not yet an Avocado, but a pencil sharpener. During the intro, our human in the scene would drop a magic droplet of liquid on to a regular pencil sharpener and eyes would suddenly spring out, and it would be brought to life, much to the delight of the human.

In this demo, we had a basic task for the player, and they’d have to find some items strewn around the play area, giving us our very first taste of player tasks in the form of these fetch quests. It also allowed us to get a better feel for drawing lines to navigate, cutting between different cameras, each of which would need to carefully frame the playspace, and we also explored the idea of cinematic cameras, which looked good, but did not invite immediate interaction, as it might be in motion, or it might not frame the table surface in a way to make line drawing easy.

We also shot it with a very shallow depth of field, and this taught us an important lesson, because the more extreme the DoF effect, the less stable the motion track we got. If we were going to make a lot of these cameras work correctly, we needed to be much more modest with the focal plane depth and position.

Sharpen was then followed by a more fleshed out idea named Tiny Frankenstein. It kept the idea of inanimate objects brought to life, but incorporated Jon’s new procedural walking system, to give them characters much more life. Alpha 1 was shot with Jack as the mad inventor with a makeshift set designed to test blocking, but it was becoming increasingly obvious that we needed to start looking for an actor that would help us tell the story properly, and a space which could be larger and more lavishly decorated. Our pencil sharpener hero had now become an avocado.

Alpha 2 featured a custom built set, Katie Reece as our hero inventor Billie, an early version of Avo with no arms, props with special effects and a much stronger narrative, with lines of dialogue. This would be influenced by Wallace and Grommit, and a desire to keep things simple, so our protagonist would be silent. As much of the editing would be done outside of the game engine, in order to minimise the amount of unused video in the application, but still allow us to tweak the timing on sequences dynamically. This meant keeping handles on the clips. This is a post production term where you give yourself extra footage at the front and back of each clip, with the expectation that you’d start playing them 1 or 2 seconds from the beginning, and stop playing them 1 or 2 seconds from the end.

Avo started out life with just his legs in Alpha 2

Alpha 2 got more and more polish, and it began to generate its own gravity. Exciting, fun, full of heart, and finally something we could scale up into a full game. Ryan North and Gemma Arrowsmith were brought in to help us create a fun story. It ended up being wildly ambitions and needed scaling back, but the bones of it were there. What followed was location scouting, set building, bespoke prop creation, full script development, table reads and all of the usual aspects of a full TV production, except done on a small budget, and very much in the guerilla film making school.

It’s at this point where we’d finished the raw invention phase, and tipped into production, polish and delivery. I cover a lot more of this in the technical post, but filming started in May 2018, and continued for approximately 10 weeks. I was split across supporting our data pipeline and writing systems in Unity. We used Black Magic’s Resolve, which used to corrupt timelines frequently, and I had to hook up a snapshot system for Postgres to allow us to roll back efficiently when this happened. I also had to work on scaling up the Python based processing script to cope with a vastly larger number of clips running through it. We were now under pressure to deliver a large volume of work quickly, so I needed to put in way more safeguards and cross checks to prevent human error from slipping by unnoticed.

From about September 2018 to January 2019 we implemented the 8 episodes seen in the main game. We added subtitle support, full music and sound effect support via Audiokinetic’s WWise, bluetooth audio support, the save checkpoint system, localisation, analytics, general UI, IAP integration, On-Demand Resource support, AR mode, low and high resolution videos, and a whole host of other things. We had no specific producer, so we took turns to run our weekly planning meetings. were crucial for establishing bottlenecks, and towards the end I was generally responsible for keeping the flow of work steady, as it became more and more technical. It was a remarkably intense time, and for the most part highly productive.

60%

We finally launched Avo at the end of January 2019, and it has gone on to have nearly 4 million downloads, and is regularly promoted in the App Store to this day. For our first title I consider it a huge success. While it may seem coherent and polished from the outside, it really was a hard won product from 3 years of inventive exploration in a brand new medium.

The ‘Making of’ video #

Timo and Jack organised a behind the scenes video of Avo’s production. I’ve uploaded to our instance of PeerTube running on recoil.org. It’s a lovely grand tour of the studio at that time.

]]>

About

Mon, 31 Jan 2022 18:09:53 UTC

You found the secret page! There isn’t a specific about page, the whole site is about me.

]]>

How MOO coupled product innovation with mass manufacturing

Wed, 21 Oct 2015 09:47:06 UTC

Hardware, as they say, is hard. In helping develop the concept of Business Cards+ into a fully realised, mass manufactured product that ships worldwide, we faced a series of engineering challenges of which needed some clever thinking to overcome. I’d like to share some of these with you in this post.

In the beginning #

To set the scene, in 2014 MOO restarted a project whose aim was to embed NFC chips into its business cards, transforming them into a smart, connected product. A similar initiative was attempted a few years earlier, but it proved to be a little too far ahead of its time and had to be put on hold. In the intervening years technology has finally caught up with the vision.

These new business cards would use cutting edge printed electronics coupled with MOO’s paper know-how. Inside each card is a tiny NFC chip bonded directly onto paper and programmed with a unique URL. This URL allows the owner to customise its behaviour whenever they like, even after those cards have been given out to prospective clients and businesses.

I came on board to help MOO design, develop and implement an end-to-end system which could produce the physical cards themselves, along with the online services that the cards would seamlessly interface with.

I knew this would be a difficult task, but this is where my history with connected products becomes important. Back in 2012 I worked at the design consultancy BERG on a product called Little Printer. It was an internet-connected thermal receipt printer with custom hardware, firmware and software that we designed and engineered entirely in-house.

I learned a lot in helping bring it to market, and one of the trickiest aspects to get right was the join between the physical and virtual. With connected products, you have something which not only exists in your hand, it also lives in the cloud, and the production process needs careful construction to ensure both are made successfully and are correctly linked together.

With Business Cards+, this means that the machinery and process we needed in order to program each card must be smart and fully connected to MOO’s backend systems, and far more sophisticated than anything that had been implemented to date. We went through a lot of early brainstorming to try to break down the problem into compartmentalised and separable areas of interest.

Mass production principles #

In order to produce these within MOO’s existing facilities, we needed to obey some core principals:

Be quick and easy to operate with minimal training
Be scalable should production demands increase
Introduce minimal disruption to the existing processes
Use inexpensive hardware to keep costs down

Existing NFC programming systems for factories can cost upwards of six figures, and the software integration costs can push that up even further, so using relatively inexpensive commodity hardware to program the NFC chips was going to be very important to us. Not only does it keep costs down, but it gives us flexibility in how we break down and arrange the individual production tasks.

MOO’s well-established production system makes business cards 25 at a time, grouped together on a large sheet of high quality card. To make the programming process as efficient as possible we designed a system which would use 25 separate USB NFC programmers to write to every single card on a sheet at once. This would keep efficiency and parallelism as high as possible while fitting within the constraints of the existing process.

We also use a barcode scanner to read specific information printed on each sheet. This information is used to make API calls back to the central servers so that we’re able to retrieve each customer’s specific data to embed on each card. The barcode scanner would ideally be the only means of input, and is robust and reliable while maintaining operational simplicity.

To manage all of these peripherals we use a small single board computer running a stripped-down Linux distribution, and along with some clever scripting it binds the separate components into a single highly streamlined appliance. Below you can see a picture of a very early prototype testing multiple NFC devices in parallel.

Induction and iteration #

One of the important aspects of our work was the physical form of the machine. The ability to communicate with any NFC chip is done through induction, and there are certain rules you need to follow with inductive coupling. In order to maximise the speed and reliability of any NFC data transfer, the gap between the NFC tag and the USB or phone device communicating with it must be as small as possible.

Given the dense arrangement of our 25 NFC chips, we needed to construct a bed to house each USB device which brought the circuit board flush with the surface where you rest the sheet of card.

I worked closely with Phil Thomas, one of MOO’s Product Designers, and he did an amazing job of modernising the original clear prototype from 2012 according to our newer hardware and stricter spacing requirements.

Phil worked on a number of iterations of the physical design. First to update the USB hardware, then a second to incorporate closer placement of the antenna flush with the surface, and third to accommodate a change in the antenna placement in the card themselves.

With each iteration, Phil designed the new layout in Solidworks, printed out a stencil on paper and then painstakingly cut through his stencil sheet into a piece of foam board below using a scalpel. In less than an hour we could test a new layout with all of the components to make it functional.

Once we had assembled our final iteration, Phil then arranged for the pieces to be prepared in thick laser-cut acrylic to form a robust case when put together. It needed to be strong enough to be sent as air freight and also survive in a busy warehouse environment.

Not only is laser-cut acrylic relatively quick and affordable to get made, it actually gives you a nice smooth surface finish. Phil had delivered a physical enclosure that was as well thought out and tailored as the software inside.

Copper Gremlins #

During the later stages of development we hit a particularly difficult issue when scaling up to running all 25 NFC devices at once. Whenever we ran our tests, we saw a low but consistent chance of permanently corrupting the NFC chips. This was highly unusual behaviour, and no matter what we did we could not find a reason why.

These sorts of bugs are some of the worst you can discover, being serious, difficult to reproduce and in turn awkward to correlate against possible sources. We lost a number of weeks writing rigorous testing processes, and repeatedly running our tests while being careful to only change one variable at a time. This was one of the lowest points of the project, and it put the whole idea of using off-the-shelf components into jeopardy.

After slowly ruling out much of the software, I eventually began to suspect this was a hardware issue, so we took the entire machine up to Cambridge. During the development of Little Printer in 2012 I was lucky enough to work with Alistair May who specialises in RF (Radio Frequency) electronics, and he has a lab in one of Cambridge’s many leafy science parks helping people with these sorts of issues. We gave ourselves a day to work on the problem, and if the source of this corruption couldn’t be found we would have to scrap all of our work done so far!

After a very tense session using some very expensive oscilloscopes and other elaborate bench equipment, we finally made a breakthrough. We had discovered a flaw in the design of the NFC readers which was compounded by very cheap USB cables where the manufacturer had skimped on the amount of copper in the wires of the cable itself, and it formed a sub-standard electrical connection.

This in itself is a fascinating lesson. It’s generally common knowledge that premium HiFi or HDMI cables advertised as being better through the use of oxygen or gold is nonsense, especially where a digital signal is concerned. These cables either work or not, surely? Digital by definition means something or nothing, so how does a lack of copper in our USB cables produce such strange analogue behaviour?

Well the detailed science behind this is something which deserves its own post, but the underlying issue here is one of resistance. You can think of a cable with normal amounts of copper as a fat straw, and one with reduced copper as a thin one.

USB devices need power to run, and NFC specifically takes this power in gulps. If the diameter of the straw is too small (i.e. not enough copper wire), then taking quick gulps becomes extremely difficult, and this is what the oscilloscope trace shows below. The bottom yellow area should be flat, and because of our cheap cables it cannot get power quickly enough, causing it to become wavey, and it’s this unwanted waviness which corrupts our chips.

In order to overcome this issue we now custom modify each and every USB device with an additional high speed smoothing capacitor to ensure it works reliably.

Returning from Cambridge with a fully working system, our thoughts now turned towards mechanisms which would allow us to give feedback on the chip programming process.

Simple chips, smart lamps #

As with any electronics there are occasionally some failures, and we see this with our sheets of NFC chips where some are non-communicative. In order to alert the operator to a chip which hasn’t been successfully programmed, we needed some kind of status display system for our NFC programmer.

Not too long ago I was lucky enough to work on a research project for Google around the concept of Smart Lamps, and I knew it would be a great fit in this instance.

Lamps: 24 Rules for smart light from Berg on Vimeo.

Due to the thickness of the cards themselves, it’s impossible to illuminate any cards from underneath without an extremely bright light source, and we have already discounted the idea of a monitor on which we’d display any errors; there would be too many opportunities for transcription errors looking between the screen and the cards, especially in a busy environment. This leaves us the idea of direct illumination from above.

To achieve this we use a pico projector connected directly to the HDMI port of the single board computer, and the output image is calibrated to match the surface of the machine itself. The hardware is inexpensive and, after some calibration, each of the 25 cards can be individually highlighted with any shape and colour we want.

Some of the designs on the cards can be very detailed and colourful, and coupled with bright ambient illumination in the warehouse, we needed to ensure maximum contrast for the projected light. We ended up using colour, flashing and motion to achieve this aim. Below you can see cards being highlighted with a “good” status after programming.

This projection mapping technique has not only reduced the chance of human error, it’s simplified overall operation of the machine, reducing the amount of training needed. This was the last piece of the puzzle, and with operator feedback implemented, we had a finished system.

Conclusion #

With a relatively modest capital outlay, we’ve introduced the means to produce a technically demanding product into MOO’s existing production line with minimal disruption. It’s flexible and scalable to match our needs, while being simple to use and maintain.

I hope you’ve enjoyed this look at some of the challenges we’ve faced in Business Cards+ to market. It’s been a challenging project on many levels, and getting to see this work as a key part of the worldwide launch is incredibly satisfying.

]]>