Playdeo Part 1 - Material Exploration

Back in 2016, Jack Schulze, Timo Armall and I set up Playdeo; a studio to develop an original idea of games and interactive experiences using full screen video.

This was the first time any of us had been directly involved with attempting to build a game, although we’d successfully worked with each other at the design studio BERG on a series of projects involving complex design and cutting edge tech. Jack brought his unique vision and design sense, Timo added his in-depth camera wrangling and cinematic skills, and I had direct experience in native mobile apps, video, and a history working in visual effects houses.

This article covers the period of time from the very start of our idea through to the end of the material exploration phase, where we tipped into producing our first game, Avo.

The spark, 2015 #

Jack started thinking about the possibilities of touchable video after seeing his young kids reach out and touch the screen of their iPads while watching a movie in 2015. Everything else you do with an iPad involves screen interaction, and in their minds video shouldn’t be any different, so of course it would be touchable! Instead, all they saw were the playback controls appearing. This made Jack wonder how true touch interaction with the video picture might actually work. He started making a prototype, working with Greg Borenstein to get things off the ground.

Some time later, Jack invited me for a coffee and showed me his first working prototype running on a laptop. It integrated three key elements; the full screen video, a 3D scene and camera motion tracking. This prototype just about held together. There was no audio, the file sizes were huge, playback wasn’t smooth, rendering was inefficient; the list went on. However, in spite of all that, there was something undeniably magical about it. You could render 3D objects into the video with rock solid believability, plus you could interact with anything on the screen. It worked better than the performance of AR at the time, and was under the player’s control, unlike something pre-rendered.

All the core elements were present, but to unlock a viable future for the tech, we knew we had to get a prototype working on mobile devices. This would immediately derisk the project by a huge amount, and since I’d spent time developing native iOS apps before, Jack wanted to know if this was something I’d be interested in working on. There was clearly enough here to show the idea’s enormous potential, so I eagerly hopped on board.

The first thing I worked on was video playback. There’s no way you could ship a product with JPEG flip-books, so we’d need a way of decoding video frames in real time. After three weeks of work, starting in October, I had a very crude iOS prototype working where we could decode true MP4 video frames, and pass that into Unity as a standard texture. Crucially, this also included frame timing information, so we could look up accompanying camera position data when rendering the video frames.

This gave us the confidence to begin scaling up our work, and start working with someone who knew the Unity engine well.

Tooling up, 2015-2016 #

Aubrey Hesselgren joined us over the winter to work on the very first prototypes, bringing his Unity expertise to the team, and helping with many of the core animation and rendering tasks. Our prototypes were crude, and relied on very ad-hoc processes of ingest and data manipulation, but each one gave us insight into what looked good, and what gameplay opportunities were available.

We had to work blind in the Unity Editor, as video decoding only worked on mobile devices. We’d frequently encounter issues such as playback synchronisation problems, involving a lot of trial and error debugging. We did our debugging with high speed video of the phone screens to figure out timing issues. It was crude, but nothing existed within Unity’s standard tools that was geared up to debug this sort of process, so we had to roll everything ourselves. Progress during this period was slow and frustrating.

As time went on, I began to bring formality and automation to our data pipeline and build process. The ingest process consisted of an ever growing Python script that formed the spine of multi stage data wrangling. Because we wanted to have near instantaneous access to any video frame, we used ffmpeg to concatenate all the videos together into a single seekable file. We started to use Autodesk’s FBX Python library to allow us to programmatically manipulate keyframe data from the camera track, rather than relying on Unity’s systems (which always wanted to smooth this motion out).

For the build process we used Fastlane. I disliked Unity’s automatically generated Xcode projects, as it would frequently be out of step with iOS releases, and I wanted a way to manipulate the generated project files independently. I’d seen Fastlane put to excellent work by Tom Taylor in a previous job, and knew it represented the perfect Swiss Army knife for manipulating, building and distributing our prototypes.

After a few months, we produced the Orange Car demo, some clips from which you can see in the images above. This was our most polished demo to date. It started with Jack speaking to camera, explaining what was going to happen, then opening a box. Out popped a little orange car which the player could control. The car headlights could even dynamically illuminate the objects on the table.

Investment, 2016 #

Armed with the Orange Car demo, we formed a company. Jack, Timo and I felt the demo showed the amazing potential of what we could make. The idea was easy to understand and, more importantly, it ran directly on a phone. Off the strength of the demo, we sought some initial investment to start scaling our work up. Chris Lee joined us as a fourth founder, bringing his important games industry knowledge.

We settled into a co-working space in Whitechapel, East London. Having spent much of the previous time camped out in Timo’s mother’s front room, with three or four of us packed in like sardines, it was a welcome step. It was a noisy and hectic space, and inexplicably our tele-sales entrepreneur neighbours would always love having their loudest conversations just outside our door. On the plus side we had one of the most important pieces of equipment, a huge whiteboard! I believe it’s the intellectual and spiritual hearth for people doing collaborative, inventive work.

In September 2016 one of our developers Yera Diaz completed one of the most important pieces of early work, a video plugin for the Unity Editor. We would finally be able to prototype by simply hitting the Editor’s Play button rather than requiring a whole iOS build to be made before we could see anything working. This would transform the experience of exploring this new medium, and accelerate the pace of our work enormously.

In October 2016, we worked with Jonny Hopper and Mike Green from Glowmade to smarten up some of the core code, and to start thinking about gameplay and interaction. We experimented with a platform game, and at this stage were still very much treating the phone like a TV, with landscape orientation and virtual joysticks for control. We were starting to mould the codebase into a space where we could experiment, and to derive a predictable and quick pipeline.

Soon after, we started working on long string of unique prototypes. Day/Night, Time Travelling, Kerbside, Physics Toy, Clean Up My Mess. These helped bring forward the idea that you were actually playing in video, not just watching it. While each prototype was its own separate concept, all of them explored touch interactions in various ways. Should the phone be horizontal or vertical? Was a virtual joystick really the best way for players to interact? How do you receive feedback?

For some of our experiments, we needed a way of transitioning from a 2D touch to a 3D drag, as can be seen in the images above. Although we made it work, it wasn’t as intuitive as we’d have liked, and we did not find any gameplay interactions that made it feel satisfying, so we kept moving forward with new prototypes.

New offices and a larger team, 2017 #

At the start of 2017 we moved to The Trampery Republic, a workspace in East India Docks in East London, and started to scale up our headcount. This was an exciting moment, but it also put pressure on everybody. We lacked sophisticated editor tools, so all design work had to be through sheer imagination first and foremost. It was particularly tough as we were working with a new medium that lacked a back catalogue of reference material. The traditional tools such as game design documents didn’t work, as there were so many technical limitations, and no obvious genre to aim for. This was probably the most difficult time, with far more experimental failures than successes.

By April 2017 we started working with Super Spline Studios and Shanaz Byrne on a project we called Night Garden, featuring a character called Tolla (seen in the image above). It was our first time experimenting with humanoid animations, inverse kinematics, enemies and a whole slew of other features.

It was an on-rails runner, with some (but limited) control over where the character was positioned, and a single long take of a camera’s path through a garden environment. Ultimately we didn’t take it forward as we felt there was not sufficiently diverse gameplay or replayability, but we were slowly improving our capabilities and ambitions. This was definitely the closest we’d come to an actual gameplay loop.

It’s at this point we’d fully committed to the vertical orientation as our preferred way of holding the phone, and using a single finger for most interactions. It was the right balance between interaction, comfort and screen visibility. This was before the rise of Tiktok, YouTube stories or any other large scale proof that vertical video would be accepted by our audience.

In August 2017 we briefly went back to the idea of controlling small cars on a race track, similar to the original Orange Car demo, but now shot vertically with single finger steering. This featured a small but crucial new facility for us; cutting to a different camera as you approached the edge of the screen. Although this brief exploration of racing wasn’t taken forward, the idea of cutting between cameras would stay, allowing the player to explore the 3D space under their own control.

This prompted all kinds of interesting questions about continuity and video time vs game time. That is, the player experiences each use of a video clip in a strictly linear fashion, so we had to be careful with anything that visually changed the world. For example, if we showed someone putting down a cup of coffee, then each subsequent clip we used had to show the coffee cup on the table, or we needed to show a clip of someone picking it back up. We learned that video clips that were designed for reuse must try to refrain from changing the set in any way. This was a big progression in understanding which impacted on our later work.

Game prototypes, 2017 #

The next important step in our prototyping was made by Jonathan Topf. Because of his work on a previous game he created, Trickshot, he had a good feel for players using a touch screen as the primary control system. His insight was to allow players to draw an intended path of movement for the character, rather than manipulating indirect controls like a touchpad or virtual joystick. I remember being really impressed at the time, and knowing this method of input was right for our game.

It’s a fantastic mechanic for player control on mobile, but it wouldn’t have made as much sense without the ability to change camera angles when approaching different regions of the table. In creative companies doing good work, breakthroughs should happen all the time, in lots of different areas. The key is to communicate these breakthroughs, and allow new possibilities to be unlocked at all points up and down the technology stack.

With the new draw-to-move mechanic, we made our next prototype with it in mind. As lines were easiest to draw on large, flat planes, setting the game on a tabletop would allow for easy filming, good opportunities for interactions between human and virtual characters, and easy player navigability. We were already drawn to the idea of small worlds like The Borrowers, but we knew fully rigged and animated characters like Tolla were too complex for us to manage internally, so we settled on a more Yokai idea of inanimate objects brought to life. This led to Sharpen (shown below), in November 2017.

Players needed to find some items strewn around the play area, giving us our very first taste of tasks in the form of these fetch quests. It also allowed us to get a better feel for drawing lines to navigate, cutting between different cameras, each of which would need to carefully frame the playspace. We also explored the idea of cinematic cameras, which were visually interesting, but did not necessarily work well with touch interaction. We were finding the right balancing act between interactivity and narrative, and this led to us distinctly naming these shots as either Coverage or Sequences. Coverage was used for line drawing, while Sequences were used to tell the story.

We also shot it with a very shallow depth of field, and this taught us an important lesson. The more extreme the DOF effect, the less stable the motion track as a result, meaning some video clips in the demo were very low quality as a result. If we were going to make a lot of these camera angles work, we needed to be far more modest with the focal plane depth and position.

When we had a retrospective on Sharpen, it was clear we had all the core components in place for our first game. These components were: small tabletop worlds, mixed human and virtual character interaction, line drawing, multiple camera angles and basic fetch quests. It was now time to commit to our first game release, but we needed a more charismatic main character …

Continued in part two of the Playdeo story.