Code of Music
Final
Making Music with Tinkamo
Synthesizer controlled by Swappable Bluetooth Sensor Blocks.
Project Overview
I built a framework for interfacing with a children’s toykit alongside a computer music composition. There are three main components to this final project:
1. Tinkamo: The hardware component was not developed by me, but rather a Shanghai-based startup I met this past summer while teaching a course at NYU Shanghai. Tinkamo is a Lego-compatible wireless sensor kit meant for kids to engage with introductory computer science topics through constructionist games, activities, and challenges. There exists a core module which holds a low-energy Bluetooth component and a battery, and sensors that are powered only once they connect with the core. Perhaps the most compelling innovation lies in the fact that once the core module has formed a Bluetooth connection, new sensors can be snapped on and off seamlessly enabling the rapid prototyping of physical controllers without any rewiring. More information can be found on Tinkamo’s Kickstarter campaign.
2. Custom Bluetooth Library: As of this writing, Tinkamo only connects to a closed source iPad App. It was up to me to build a connection to my computer for making music. I went with Node as my programming language, Noble, a library for connecting to Bluetooth Low Energy (BLE) devices, and Open Sound Control (OSC) for connecting to other software and programming languages (Max in this case). The code largely matches the blocks themselves. There is a TinkaCore class in which new instances are created once the physical cores are connected, and a TinkaSensor class with subclasses that correspond to the various sensor types. After early versions of the work, Jam, the founder of Tinkamo, gave me a couple keys for understanding what the messages mean which sped up development. A longer term goal would be to connect to the Chrome Bluetooth API for utilizing with p5.js. (Imagine working in the p5.js editor, including the Tinkamo library, and walla - you now have all kinds of sensors at your disposal to control your code wirelessly!)
3. Max Patch: In addition to the musical instrument itself (described in more detail below), I built a connection patch in Max, a visual programming language I love with the motto “Connect Everything” that applies well to music making and physical computing. The connection patch receives messages from the Node server and forwards them to any open Max patches with helpful names, also via OSC. In my instrument, there are receive objects that then use the sensor readings to control elements of the patch itself. The benefit of this modular design is that the instrument functions independently of Tinkamo and can be built/played with entirely on my laptop.
Ideas Driving the Piece
The piece primarily revolves around the distance sensor. I wanted to come up with an instrument that is controlled essentially as a simple theremin but utilizes a more interesting mapping than height -> pitch. I came up with the idea of controlling the number of chord tones with height. As the performer raises his/her hand up in the air up to six notes in the chord fade in. This idea of chord tones opened up the challenge of how to change chords with only one sensor input. From there, I came up with the idea of using the sensor as a switch - once the distance sensor reads values below a certain threshold, it acts as a trigger resetting the chord, the order in which chord tones are outputted, and the timbre of each of the tones. Chords are not drawn from any lists, but are generated randomly by stacking notes on top of each other. This works based on three user parameters: lowest possible note in which the root with fall within an octave above, stack, the number of half steps each note will be above the previous (e.g. if stack is 5 and multiple is 3, the available intervals between each note is 5, 10, and 15.) For this performance I used a low-bound of 50, a stack amount of 7 (perfect fifth), and a multiple of 1. I found that the open fifths were pretty rather than dissonant, and the higher pitches sounded pure like augmented musical glasses with different water levels.
While this was the first Max patch I started working on, I experimented in six or seven smaller patches with different sounds and inputs before honing in on this one. In particular, I built a fully-fleshed out drum sequencer, but some of the feedback I got from the class during the User Testing Session was that it felt like a completely separate music composition. So, I took some of the control and rhythmic components from that patch and applied them to my “chord theremin.” The color sensor controls distortion level via a simple bit crusher. (Certain distortions resonate especially well.) The button toggles a series of pulses to play. There are two sets of rhythmic pulses, one panned to each side that interlock. Finally, the knob controls the tempo of the pulses. Each of these components make for an instrument that is pretty fun to improvise and interact with.
View the Project Presentation.
A blog for displaying Willie Payne’s progress in the NYU ITP Course “The Code of Music.”