Press play below and drag Cutoff slowly up and back down. A dark, muffled pad opens into something bright and airy, then closes again. Now imagine doing that by raising your hand in the air. That's this project: an effect you play with your body, like a theremin player shaping sound without touching anything, or Imogen Heap conducting her music with sensor gloves on stage. By the end you'll have a short performance, filmed on your phone, of a sound opening and closing as your hand rises and falls.
Sweep Cutoff up and down. Which part of the range sounds most expressive, and where does it get too dark or too harsh to be useful?
Play it by hand first
Synth Texture: Komiku, CC0 1.0
Before you add a camera: what should the sound do when your hand leaves the frame? Stay where it was, slowly return to dark, or jump? Hold on to your answer; you'll test it in Stage 5.
From hand to sound
Stage 1: Find the useful range
In the lab above, find two cutoff positions: a low one that's dark but still has a body (not silent), and a high one that's bright but not piercing. Those are your hand-down and hand-up positions. Add a touch of Resonance if you want the sweep to sing. Now you know exactly what your hand should control, before any technology gets involved.
Keep this mouse version handy throughout the project. It's your backup instrument, and it tells you whether a problem later is in the sound or in the tracking.
Checkpoint: two cutoff values you like, written down. Filters explains why a low-pass sweep sounds like opening a door.
Stage 2: Let the camera see your hand
Open HandMod, Maru's hand-tracking controller, and allow camera access. Set the first lane's Source to Left Hand Y (your hand's height) and turn the other lanes off for now. Raise and lower your left hand and watch the lane's value follow.
Check which hand is which. With the preview mirrored, like a selfie camera, your physical left hand appears on the right of the screen. Move only your left hand and make sure the lane moves. Now you know the camera sees what you think it sees.
Checkpoint: one lane that follows one hand. Camera and gesture control explains how a tracker turns pictures into positions.
Stage 3: Connect hand to filter
HandMod sends its lane out as a MIDI control change, CC 1 on channel 1 by default. In your music software, put a filter on a pad or drone, then use its MIDI-learn or mapping feature to connect CC 1 to the filter's cutoff. Raise your hand. If the knob moves and the sound opens, you're playing an effect with your body.
If nothing moves, check the CC number and channel match on both sides before changing anything else. If the knob moves but the sound doesn't change, check you mapped the cutoff of the filter you're actually hearing.
Checkpoint: your hand opens the filter. MIDI helps you follow the message from one side to the other.
Stage 4: Make it comfortable and steady
Set the lane's Min and Max so that your comfortable hand-low and hand-high positions land on the two cutoff values from Stage 1. Now your whole range of movement is useful range; no reaching for the ceiling.
Then hold your hand still in the middle and listen. You'll probably hear the sound flutter slightly, because no tracker is perfectly steady. Raise Smoothing until it stops. But move fast and you'll notice the trade: smoothing also makes the sound lag behind your hand.
Smoothing trades jitter for lag
Choose the trade for your music. A slow, dreamy sweep over a pad can take plenty of smoothing. A sharp rhythmic gesture needs less.
Checkpoint: a steady hold and a sweep you can repeat. Controls, macros and ranges covers ranges and response curves in depth.
Stage 5: Test the moment your hand disappears
Raise your hand to fully open, then drop it out of the camera's view. What does the sound do? Come back. Cross your hands. Turn the camera off.
What does the filter receive when the tracker jitters or loses your hand?
When the tracker loses you
Not playable yet.
Draw a line from hand low to hand high and mark your two cutoff values on it. Then note what the sound does at four moments: hand still, fast move, small jitter, hand gone. Decide which of those you want to change.
Write down what happens at each moment and decide whether you like it. A sudden jump to full brightness when you step away is a nasty surprise in front of an audience. If you get one, narrow the range or choose a behavior where the value holds. And make sure you can always stop the sound without the camera, with the mouse or a key. Now your instrument fails gracefully.
Checkpoint: a note of what happens when tracking is lost, and a way to stop the sound that doesn't depend on it. Live performance and looping has more on recovering on stage.
Stage 6: Perform it and film it
Pick a short loop: a held chord, a drone, a simple synth pattern. Plan a 30-second performance: start dark, open slowly over one phrase, close before the next, leave a moment of stillness, then one last big sweep. Practice it twice.
Then record the audio in your music software, and film yourself on your phone at the same time. The video is the part to show a friend: hand rises, sound opens, and the connection is obvious and a bit magical. Watch it back and listen: does the sweep land with the phrase? Does it flutter when your hand is still?
Checkpoint: a recorded performance and a short video of it.
Stage 7: Map something else
Save your setup, then try one new mapping alongside it: hand height to reverb amount, or your other hand's pinch to delay feedback (gently; feedback can run away). Notice that each destination wants a different speed and range. Now you have two expressive controls, and a feel for how to choose what a gesture should do.
Quick check
- The camera preview shows your hand perfectly, but the filter doesn't move. What do you check first? The connection, not the camera: that the CC number and channel HandMod sends match what your filter is mapped to. A working preview only proves the tracker can see you.
- Your sweep sounds smooth but feels sluggish when you move fast. What changed? Too much smoothing. It removes jitter by averaging, and averaging takes time. Lower it until the lag stops bothering you, and accept a little flutter.
- You step away to adjust something and the filter jumps to full brightness. How do you avoid that on stage? Test hand loss before performing, narrow the range so the extreme is never painful, choose a behavior where the value holds, and keep a non-camera way to stop the sound.
Next steps
- Keep it: Presets, state and automation. Save the gesture ranges and check what comes back after reloading.
- Sideways: LFOs and modulation. Your hand is doing the job an LFO does, only with intention.
