Training on your own sounds

Here's one learned instrument with two models inside it. Feed it a sound and listen. Then switch Model from Dawn chorus to Strengjavera and play the same thing again.

The first time you open it, each model downloads into your browser, about 50 MB for the birds and 125 MB for the piano, then runs on your own machine.

Before you switch: what do you think makes the two sound different? A different design? Different settings? Mostly, it's neither. Both are RAVE models, the same kind of model built the same way. One spent its training listening to recordings of birds at dawn. The other listened to a magnetic resonator piano, whose strings are set ringing by magnets. Everything you hear, every chirp and every shimmer, comes from those recordings.

The dataset is the instrument. If you trained this model on your own sounds, you'd get an instrument that sounds like your sounds, in the range you recorded, with the habits you recorded. Choosing those recordings is a musical decision, and it matters more than any setting.

What does training actually do?

Training nudges a model's internal numbers, its weights, millions of times, until its guesses match the examples it was shown. What you have at the end has absorbed the patterns in those examples, and nothing else. What an audio model learns follows that process, and the different ways the examples are gathered, in more detail.

So "train a model on my cello" really means: show it enough cello that it learns what cello does. What it learns depends on what you show it.

The jobs come in two broad shapes, and the shape decides how much work you're in for:

• Paired jobs. You give the model an input and the exact output you want for it. An amp capture is the classic case: you play a test signal through your pedal, record what comes out, and the model learns the mapping from one to the other. The answer is right there in the data, so the job is narrow and the model can be small. Neural amps and captured gear goes into captures in depth.
• Unpaired jobs. You give the model a pile of recordings and ask it to learn the whole world they come from. That's what the two models above did. Nobody told them "this input should become this chirp"; they learned to describe birdsong, or magnetic piano, from the inside, so they could turn any input into their own vocabulary. That's a much bigger job, and it needs much more data.

How much, how varied, how clean?

If you've ever planned a sample library, you already know most of this. Every question you'd ask before recording a sampled instrument is a question about a training set.

How much. For a paired capture, a few minutes of test signal and its recording is often enough, because the job is narrow. For an unpaired model of a whole sound, think in hours, not minutes. Projects that train timbre models like the ones above tend to ask for an hour or more of material, and more helps. Five minutes of your cello won't make a cello model; it'll make a model of those five minutes.

How varied. A model can only move within the space its recordings covered. Record only soft, low notes and it will struggle with a loud high one. Record only one bowing style and it has no idea what pizzicato is. Before you record, list the range you want to play later: lowest and highest notes, quietest and loudest dynamics, every articulation, a few awkward transitions. Anything missing from that list is something the model will have to guess at.

How clean. Whatever is in the recordings becomes part of the instrument. Room sound, a hum, the click of a pedal, your chair creaking, a bandmate bleeding into the mic: the model doesn't know these aren't "the sound", so it learns them too. That can be lovely. A model trained in a stairwell carries the stairwell. But choose it, don't inherit it. Clean, consistent recordings in one space give the model one thing to learn.

Same idea, different place: planning a training set is the same work as planning a sampled instrument: velocity layers, round robins, the range you cover and the noise you allow. Planning and recording samples is a practical checklist for both.

From scratch, or starting from someone else's model?

There are three routes to a personal model, and they cost very different amounts.

Training from scratch starts with random weights and learns everything from your recordings. You get a model that is entirely about your sound. You also pay full price: lots of audio, and a long time on a powerful computer. For timbre models of the kind above, that's typically days on a strong graphics card, not an afternoon on a laptop.

Fine-tuning starts from a model someone else already trained on lots of general audio, and continues training it on your recordings for a while. It already knows what sound in general does, so it needs far less of yours to learn what's special about it. The catch: it keeps some of its old habits, and it comes with its maker's terms. Not every model family offers it: RAVE, the family behind the two models above, has no documented fine-tuning workflow, so it's usually trained from scratch.

Capturing is the paired route, and it's the one most musicians will try first, because it's quick and the result is easy to judge: does it sound like my amp or not?

Too little training, too much

Training is a balance, and both ends have a sound.

Underfitting sounds like mush. Stop too early, or give the model too little to learn from, and it only knows the broad strokes: a vague, smeared average of your recordings. A cello model at this stage might give you a bowed-sounding hum with no grain and no attack.

Overfitting sounds like a tape recorder. Train too long on too few recordings and the model stops learning what your sound does and starts memorising what your recordings were. Feed it something new and it answers with fragments of the training takes: the same phrase, the same note, the same breath, whatever you play.

The way out is an old one: keep some recordings back. Before training, set aside a few clips the model never sees. People in machine learning call this a validation set or test set. While training runs, check how the model does on those clips. When it keeps getting better on its training material but worse on the held-back clips, it has started memorising. That's your cue to stop, or to record more variety.

Your ears are the best check of all. Play the model something that sounds nothing like your training recordings, a hummed tune, a drum loop, a door creak, and listen. A healthy model answers in your sound, following the new input. An overfitted one answers with your old takes.

What a personal model can and can't do

It can respond to new playing in your sound: hum into a model of your cello and get a cello-ish phrase back. It can capture the feel of a pedal at one setting so you can carry it anywhere. It can blend and morph in ways a sample library can't, because it isn't playing back recordings, it's estimating new sound from what it learned.

It can't play what it never heard. It doesn't know your intentions, only your recordings. It won't be as precise as a well-made sampler for exact notes and repeatable dynamics, and it may add a little delay while it listens and answers. And like every model on this site, what comes out is an estimate: the model's best guess at what your instrument would do, not a recording of it.

Before you record anything, decide which of those matters for your music. If you need exact, repeatable notes, sample it. If you want an instrument that responds and surprises, train it.

Keeping the rights to your data

A model trained on recordings carries those recordings inside it, in a form you can't easily take back out. So be sure of what goes in:

• Your own playing on your own instrument is the simplest case.
• Bandmates, session players and singers in the recordings need to agree that their performances will train a model, not just appear on a track. A voice in particular is part of someone's identity; voices, cloning and consent covers why.
• Samples, loops and other people's songs in the data bring their own permissions with them, or lack of them.
• The code and any starting model have their own terms too. The two models on this page, for example, are shared by the Intelligent Instruments Lab for non-commercial use only, so anything made with them stays in sketches and personal pieces unless you get permission. A model you fine-tune from someone else's may inherit limits like that.

Keep a short note with your dataset: what's in it, who played it, and what everyone agreed to. It takes five minutes now and saves a difficult conversation later.

Make it yours

Pick one sound you'd love a model of: your voice, your guitar through your favourite pedal, a toy piano, the creak of your house. Then plan it like a recording session:

1. Name the job. "A model that turns my humming into my cello" is a different dataset from "a capture of my fuzz pedal on its loudest setting".
2. Write the recording list. Lowest and highest notes, softest and loudest, each articulation, a few transitions, and how long in total. Be honest about how much time you have to record.
3. Choose the room and the noise. One space, one mic position, and a decision about whether the room is part of the instrument.
4. Record a short sketch of the plan, a few minutes, to hear whether the list covers what you'd want to play.
5. Hold one clip back. Label it "never train", and keep it. It's how you'll catch your model memorising later.

While you plan, try a listening exercise with LiveTransfer: feed each model something its training recordings never contained, and listen to how it fills the gap. That gap is exactly what your recording list is trying to close.

Quick check

1. You train a model on twenty minutes of your soft, low cello playing. It sounds great on slow low melodies and falls apart on a loud high note. Was the training broken? No. It learned what it was shown. Loud, high cello wasn't in the data, so the model has to guess. Record that range and train again, or keep the model for the music it's good at.
2. Your model sounds amazing on your training takes, but whatever you feed it, it plays back phrases you recognise from them. What's going on, and what would you try? It's overfitting: memorising the recordings instead of learning the sound. Stop training earlier (check the held-back clips), or add more varied recordings so there's more to learn than to memorise.
3. You want a personal model of your fuzz pedal by the weekend. Capture, train from scratch or fine-tune? Capture. It's a paired job (your input and the pedal's output), it needs only a few minutes of audio, and you can judge it directly against the pedal. A from-scratch timbre model would need far more audio and time for a job a capture does well.

Next steps

• Use it: An instrument trained on your sound. Take your plan through the whole cycle: record, train, and play a model of your own source.
• Go deeper: Neural instruments and timbre transfer. What happens inside a trained model when you play it: the encoder, the code and the decoder that only knows its training sound.