Voices, cloning and consent

Press play. A voice is singing a long "aah". Move Tongue Pos slowly and the vowel slides toward "ooh" or "eee". Raise Breath and it turns airy and close. Push Tenseness and it presses forward, like a singer reaching for a high note.

Here's the question to sit with: whose voice is this?

Nobody's. Mouth Synth is a model of a throat and mouth built from tubes and numbers: a buzzing source for the vocal folds, a shaped tube for the tract, a little turbulence for consonants. It was never trained on anyone. Its tract model carries the names of John Kelly and Carol Lochbaum, whose work at Bell Labs made an IBM computer sing "Daisy Bell" in 1961, one of the first times a machine sang at all.

The voices that make headlines today are different. They sound like someone: a well-known rapper, a pop star, your favourite singer. That's because they learned from someone's recordings. That difference matters when you decide whose permission and credit the result needs.

What makes a voice someone's?

When you recognise a singer on the radio after half a word, you're hearing two things at once.

The performance is what they choose to do: the notes, the timing, the words, where they breathe, how they slide into a note and how long they hold it. A great cover singer can copy a lot of this.

The instrument is the body doing it: the length of the throat, the shape of the mouth and nose, the way their vocal folds buzz. It gives a voice its timbre, the colour that stays the same whatever they sing. Nobody can change their instrument much, which is why it works like a signature.

You can move the instrument with ordinary signal processing. Here's a sung phrase rebuilt by a vocoder, with control over its formants, the resonant peaks the throat and mouth carve into a voice.

Lower Formant Stretch and the voice sounds like it comes from a bigger body. Raise it and the singer shrinks. Yet it never turns into a particular someone else. The words, the tune and the timing are untouched, and the new body is nobody's body: just a stretched version of the original. (How moving formants differs from moving notes is in pitch shifting and correction.)

A DSP effect reshapes the voice you give it. To sound like a specific other person, a system has to know what that person sounds like. That knowledge has to come from somewhere, and it comes from their recordings.

Three jobs with technical names

The research world splits voice generation into a few jobs. Here they are in a musician's terms.

Text to speech (TTS) turns written words into a spoken voice. It's the oldest of the three and the one most of the technology grew up in: phone menus, screen readers, sat-navs. Musicians have borrowed it for decades, from robot narrators to the spoken samples in electronic tracks.

Singing voice synthesis takes notes and lyrics, like a score, and sings them. Yamaha's Vocaloid, released in 2004, made this mainstream, and its best-known voice, Hatsune Miku (2007), was built from recordings of a voice actor, Saki Fujita, who agreed to lend her voice to the product. Voices like hers are built from singers and voice actors who said yes, and whole communities of producers write for those characters as if they were session singers.

Voice conversion keeps a real performance and swaps the body. You sing; the model re-renders your take in another voice's timbre. When it learns a particular person's voice from their recordings, people call it voice cloning, and when it works on singing rather than speech, papers call it singing voice conversion. Speech research came first here too, and singing models inherit its habits: a voice trained mostly on speaking can struggle with a sustained high note.

What survives a conversion?

Because voice conversion starts from your take, most of the performance comes through. Before you read the list, guess: of pitch, timing, words, breaths and vibrato, which would you expect a conversion to carry, and which to lose?

• Pitch and melody: usually kept. Many systems track your pitch and hand it to the new voice, so the tune is yours.
• Timing and phrasing: kept. The model works moment by moment, so where you push and pull against the beat stays where you put it.
• Words: mostly kept, though consonants can soften and accents can drift toward whatever the training recordings sounded like.
• Breaths: often kept, sometimes turned into an odd, slightly voiced whisper, because breaths are rare in the recordings these models learn from.
• Vibrato and slides: your shape, their colour. A singer whose vibrato is part of their sound won't arrive with it.

Out of range is where it gets strange. Sing far below or above anything in the training recordings and the model has to guess. The voice may crack, go hoarse, or snap into a range it knows. That's the same rule as every model on this site: its estimate is only as good as the examples it learned from. Neural instruments shows the same split with a model trained on birdsong instead of a person, where the "voice" you're borrowing is a flock of birds.

So here's the honest description of a cloned vocal. It's two people at once. The performance belongs to whoever sang into it. The timbre belongs to whoever the model learned from. If you sang a melody and a model made it sound like a famous singer, you're the performer, and they're an uncredited instrument.

Same idea, different place: a vocoder makes the same split with no learning at all: your voice supplies the words and shape, a synth supplies the buzz. The difference is that the synth isn't anyone. Vocoders shows how much of a voice survives that split.

Whose voice is it?

That's why consent sits inside the craft of using voices, not just in the paperwork afterwards. A voice is part of someone's identity. Hearing yourself say or sing something you never did is unsettling. For a working singer, an unauthorised model of their voice competes with them for their own job.

Musicians have already found different answers:

• In 2021 the composer Holly Herndon released Holly+, a model of her own voice that other people could perform through, on her terms.
• In 2023 Grimes invited people to release songs made with an AI model of her voice, and offered a 50 percent royalty split.
• That same year, "Heart on My Sleeve", a track by an anonymous creator imitating Drake and The Weeknd, went viral and was pulled from the major streaming services after their label objected. Neither artist had agreed to it.

What separates the first two from the third isn't the technology. It's permission.

Laws are catching up unevenly, and they differ from country to country. Some places now protect a person's voice specifically, alongside their name and likeness. This guide isn't legal advice. But you don't need a lawyer to practise the habits that keep you on the right side of the people involved.

Our take: treat a voice you didn't make like a sample you didn't record. Ask first. Write down what you were allowed to do. Credit everybody. Rights and maintenance shows how to keep that permission beside the file, so it travels with the track.

Habits for working with voices

• Ask, and ask for the specific use. "Can I use a model of your voice on my EP?" is a different question from "Can I sell a model of your voice?". Agreement to one isn't agreement to the other.
• Credit the performer and the voice. If you sang into a model of your friend's voice, the honest credit names you both: your performance, their voice.
• Label it. Tell listeners when a vocal is synthetic or converted. Most people don't mind the tool. They mind finding out later that they were fooled.
• Keep the original take. Your dry performance is proof of what you sang, and it lets you re-render later if the model or the permission changes.
• Check the model's own terms. Even a model you're allowed to use may come with limits, such as non-commercial use only, that decide what you can release.
• Don't train on a voice you don't have permission to use. That includes ripping an acapella from a record you love. Separation makes it easy; it doesn't make it yours.

Where it breaks

Even with every permission in place, converted voices have tell-tale weak spots. Listen for them before you release:

• Harmonies and doubles. Run three stacked parts through the same voice and they can lose the small differences that make a real double sound wide. Sing each part differently.
• Screams, growls and whispers. Extreme sounds are rare in training recordings, so the model smooths them toward an ordinary sung tone.
• Other languages. A voice trained on English may sing French with English vowels.
• Long held notes. The timbre can drift or wobble over a long sustain, where a real singer would stay steady.

And one that isn't a flaw: sometimes the most interesting voice is one that isn't anyone. A character voice, like the one at the top of this page, carries no one's identity, needs no one's permission, and can still sing your melody with feeling.

Make it yours

Pick one vocal idea you've been wanting to try: a backing choir, a robot narrator, a duet partner, a character who sings one line. Decide which kind of voice it needs:

1. Your own voice, reshaped. Formants, autotune, a vocoder. Nobody else is involved.
2. A voice that belongs to nobody. A synthesised character, like Mouth Synth.
3. Someone's voice. A converted or cloned voice, which needs that person's permission.

Then build a character with Mouth Synth. Find a vowel, a breath and a tension that feel like a personality: a weary narrator, a tiny bright chorus, a low drone monk. Press Save and name the preset after the character, so you can cast it in another track.

Finally, write the credit line your idea would carry, even if it's only for you: "Lead vocal: me. Choir: Mouth Synth character 'Lantern'." or "Lead performance: me. Voice model: Sam, used with permission for this single only." If you can't fill in that second kind of line yet, you know what to ask before you go further. And if the voice you'd most like to borrow turns out to be your own, that's a different project: a model trained on recordings you made and own.

Quick check

1. You sing a verse and a conversion model makes it sound like a well-known singer. Who performed it? You did. The timing, notes, phrasing and words come from your take. The model supplied the timbre it learned from that singer's recordings, which is why their permission matters as well as your credit.
2. A friend lowers Formant Stretch until their voice sounds huge and dark, and asks whether that's a "clone" of a bass singer. Is it? No. The formant change reshapes their own voice; it doesn't know what any other person sounds like. A clone needs a model that learned a specific person's timbre from their recordings.
3. You convert a whispered intro and the result sounds like a slightly hoarse sung tone. What went wrong? Probably nothing broke. Whispers are rare in the recordings these models learn from, so the model pulled your whisper toward the sung voice it knows. Keep the whisper dry, or convert only the sung parts.

Next steps

• Go deeper: Training on your own sounds. The one voice you never need permission for is your own. What would it take to train a model of it, and what would it really learn?
• Sideways: Identity and story. An artist's sound turned into something others can play, and why the artist should be the one to decide what that sound is.