TWID #4: Building Visuals for OP-Z in Unity

This week I used unity to build an interactive environment that can be controlled with the op-z sequencer from teenage engineering or any other midi capable device.

Jan 29, 2023

When teenage engineering launched the op-z around 2018 I was completely hooked. Not because of it’s musical capabilities, but because they’ve teamed up with keijiro takahshi to release videolab.

What is videolab

Videolab is a package for unity that lets you use a node based interface to map midi messages from the op-z (or any other midi controller) to almost any parameter in unity. You can then build a regular game, or export it as a videopak that can be opened and played in the op-z app, which is just awesome.

I’ve done that before

When they released it around 2018, I did a few of those scenes. But never actually managed to build a proper pack. So I just recorded them.

But after that initial run I’ve never touched it again, since it took quite some time and effort to make those.

But when videolab creators announced the videopak contest, I thought it might be fun to participate and build another one. But this time as a proper pack, so that others could use it as well.

Things I’ve learned so far

Build your scene in blender first

While you could assemble your scene in unity, I found it useful to do all that in blender. It gives me more tools to focus und the composition and visuality.

Using blender to create the scene and decide on composition.

Add materials and lights in unity

Since the engine to a huge part shapes the look of a scene, I did all the materials and lighting in there. While in blender it looked pretty nice, it did not do that in unity, which is somehow related to the fact that you have to use a 2018 version to run it properly.

To make it look better, I’ve added an hdri to have proper lights and reflections on metallic surfaces. It is worth mentioning that to make this work in the videopak it is necessary to add a reflection probe to the scene.

Define interactions to shape the scene

The most interesting part is of course hooking it up the midi controller. There is a node based interface called klak, which comes with videolab. But you can use it dependently as well and integrate it into other unity projects.

To use it you can either select a note or a cc message as an input. Convert the value to something else, like a rotation, gradient or a vector. And then use that value to control different aspects of a game object.

Using a note message to change the color of a material.

Conclusion

To get the interactions right takes a lot of experimentation and I am still not quite happy with what I have so far. But for me it worked best to have the drum tracks do a thing when played. And to have a separate channel just for adjusting the camera and overall look of the scene.

There is still some work needed and I’ll post more when I did that.