Generating notes

Below is a model that suggests chord progressions. It starts at a low Temperature. Press play and listen to eight chords.

Before you touch anything: if you turn Temperature up, will the chords get better, worse, or just different?

Now try it. Generate a few times at 0.3, then at 1.6. At the low setting you'll probably hear progressions you could swear you've heard before. At the high one you'll hear chords you wouldn't have picked, some lovely, some baffling. Change Seed and you get another roll of the dice at the same temperature.

If you guessed "different", that's the heart of it. The model didn't get smarter or dumber. You changed how adventurous it's allowed to be. Whatever it gives you is a list of chords you can change one at a time.

Notes, not sound

There are two very different ways a model can "make music", and they give you very different things to hold.

An audio generator gives you a recording: it can sound finished, but there are no separate notes inside it to move. Audio generation covers that kind of model and what you can still do with its clips.

A note generator gives you symbols: chords, notes, drum hits, as MIDI. It doesn't sound like anything until you play it through an instrument. But every single decision is still yours to change. Wrong third chord? Change the third chord. Hi-hat too busy? Delete four hits. The field calls this symbolic generation, because the model writes symbols rather than sound.

You can also chain them. Accompanist, a module we're building, goes the other way round: it takes notes you play and generates audio from them. Notes in, sound out. A note model can write a part, and an audio model can perform it.

Our take: for anything you'll want to tweak (a bass line, a chord bed, a drum part), start with notes. Reach for audio when the sound itself is what you're missing.

How a model writes music, one step at a time

A note model reads and writes music as a sequence of tokens: small symbols from a fixed list, a bit like words from a dictionary. For chords, a token can be a whole chord. Chord Seq, the model above, has a list of about a thousand chord shapes and picks one at a time.

For melodies and drums, a token is usually smaller. A phrase of MIDI might be written something like this:

BAR · NOTE_ON C4 · WAIT 1/8 · NOTE_ON E4 · WAIT 1/8 · NOTE_OFF C4 · …

Every model has its own token list, but the idea is the same. Music becomes a line of symbols, and the model's whole job is to answer one question, over and over: given everything so far, what comes next?

1. It reads the tokens so far.
2. It gives every possible next token a score: how likely, from what it learned, is each one to come next?
3. It picks one, partly by chance, guided by those scores.
4. It adds that token to the line and goes back to step 1.

This is called next-token prediction, and the model learned its scores from thousands of pieces of music in training. Chord Seq learned from tens of thousands of chord charts of popular songs, which is why its favourite moves sound like pop and rock. A model trained on jazz standards, or on Carnatic compositions, would have entirely different favourites.

Same idea, different place: this is the transition table from Music as data, grown up. A transition table remembers only the last note. A modern note model looks back over many bars at once, so it can notice that the chorus started on F and that you've already used that chord twice.

These tokens are not the same as the audio tokens in How models see audio. An audio token is a code for a sliver of sound with no musical name. A note token is a musical event: you can say which note it is, and you can change it.

Temperature is a musical control

Step 3 is where you come in. After C, Am and F, the model might rank its options like this:

Temperature reshapes those chances before the pick. Turn it down and the favourite gets even more likely: the model plays it safe and sounds like the average of everything it heard. Turn it up and the long shots get a real chance: you get the B♭ that makes a progression feel wistful, and also the E that makes it feel lost.

That's why the randomness isn't a flaw to be engineered away. A model that always picked its top choice would write the same progression every time, and it would be the most average progression in its training data. The randomness is where variety comes from, and temperature is the dial between cliché and chaos.

Two more controls often sit beside it. Top K says "only ever choose from the K highest-ranked options", which cuts off the wildest long shots while keeping some surprise. Seed fixes the dice: the same seed, settings and model give you the same result again, so you can come back to a progression you liked.

In a melody and drum tool we're testing, built on the open research model MIDI-GPT, the temperature knob is labelled Creativity, and it only turns across the range where the results still sound like music. Outside that range, low settings repeat the same loop and high settings stop sounding like the style at all. Where that useful range sits depends on the model and the job, so it's always worth a quick sweep before you settle.

Optional: what temperature does to the numbers. Each chance is raised to the power 1 ÷ temperature, then all of them are scaled back so they add up to 1. At temperature 0.5 that's squaring: 0.40 becomes 0.16 and 0.04 becomes 0.0016, so the gap between them grows from ten times to a hundred times. At 1.5 the gap shrinks to under five times.

Four jobs for a note model

The same "what comes next?" engine does several different musical jobs, depending on what you give it and what you ask for.

Continuation. You play two bars; the model writes the next two. This is the most natural job for a model that reads left to right, and the easiest to steer: the better your opening, the better its guess.

Infilling. You keep the first and last bars and ask for the middle. That's harder than it sounds, because the new bars have to lead into something that already exists.

A model that only reads left to right has a blind spot here: when it chooses bar 2, it can't see bar 4. Some models are trained specifically to fill gaps; others are given the end as extra input. If a tool can only continue, you'll often get a middle that wanders off and never comes back. That's true of Chord Seq too: it's built to write forwards, so to replace one chord in the middle, it needs help that looks at the chords on both sides.

Drum patterns. Here the tokens are drum hits and their timing. What makes a generated beat good isn't just which hits, it's when and how hard: the snare a hair late, the ghost notes quiet. Models learn that from recordings of real drummers played as MIDI. Google's Magenta team recorded drummers playing electronic kits for the Groove MIDI Dataset and, in a 2019 paper ("Learning to Groove"), trained models to turn a rhythm someone tapped into a full drum part with a drummer's feel. Groove MIDI lets you browse that dataset: real players, with their own timing and accents left in. That placement is musical, not random, which is why a groove is more than wobble.

Chords and accompaniment. Given a melody, suggest chords under it. Given chords, write a bass line. Here the model has to fit what's already there as well as continue it, which is closer to what a session player does. The field calls this conditional generation: the output is shaped by another part you supplied.

Fix it, don't reroll it

Here's where notes really pay off. Say a model gave you this line, and you like everything except the last note.

One move, and the line lands. With a generated audio clip you'd be slicing, pitch-shifting and hoping the edit doesn't smear. With notes you change the one thing you didn't like and keep everything you did.

That changes how to work with a note model. Don't reroll until you get a perfect answer; you'll throw away good bars to fix bad ones. Generate a few, keep the one with the best idea in it, then edit. The model is quick at proposing; you're better at choosing.

Where it breaks

Note models are good at short, local decisions and much weaker at the shape of a whole piece.

• Short memory. Most tools work with a few bars of context; the melody and drum tool we're testing works with four. Inside that window the model can be convincing. Ask it for a whole song and the chorus may never come back the way a chorus should, because the model can't hold the plan.
• Loops at low temperature. Turn it right down and the model can get stuck repeating its favourite bar, because each repeat makes the next one even more likely.
• Wandering at high temperature. Turn it right up and it stops sounding like any style at all. Each choice is plausible on its own; together they go nowhere.
• Echoes of the training data. A model can produce a phrase very close to something it learned from, especially a famous one. If a generated hook sounds suspiciously familiar, trust that feeling and check.
• Only what it learned. Ask a model trained on pop chord charts for a raga, a maqam or a gamelan pattern and it will answer in pop, because that's the only vocabulary it has. Its token list may not even contain the notes you need.

If your idea is audio (a hum, a riff you recorded), you'll need a listener first to turn it into notes, and Audio to MIDI shows what that step gets wrong. For where note models sit among all the other jobs models do, see What AI can do in music.

Make it yours

Take a song you're working on. In the chord generator at the top of this page, set Starting root and Starting quality to your song's key. Then:

1. Generate at a low temperature, around 0.3. Note what you hear: probably safe and familiar.
2. Generate at the default, then at around 1.4. Listen for one chord in each that you wouldn't have chosen.
3. Keep the progression with the best idea in it, even if it's only one chord.
4. Drag the MIDI into your track, play it with your own instrument and change at least one chord by hand.

Press Save on the generator and name the preset after the song, so the seed and temperature that gave you the idea are one click away. The progression in your track is now yours: the model proposed it, you chose and shaped it.

Quick check

1. Your generated progressions all sound like the same four chords. What do you change first? Temperature, upwards. At a low temperature the model sticks to its favourite moves, which are the most common moves in its training. A little more heat lets the second and third choices through.
2. You need a bass line you'll keep tweaking for weeks. Should you generate audio or notes? Notes. Every tweak to a MIDI bass line is one edit, and you can change the sound later. An audio bass line would need slicing and pitch editing for every change.
3. You ask a model to fill in bars 2 and 3 between two bars you love, and the result wanders off and never meets bar 4. Is the model bad? Not necessarily. A model that only reads left to right can't see bar 4 while it writes bar 2. Try a tool built for infilling, or generate bar 3 yourself and let the model fill the shorter gap.

Next steps

• Go deeper: Playing with a model live. A note model waits for you to press Generate. What changes when it has to answer while you're still playing?
• Use it: Jam with a model. Play a part, let a model answer it, and keep the bits worth keeping.
• Sideways: Chords and voicing. A model picks which chord; you still decide how to voice it. Same notes, very different feelings.