Inside the un-mixing machine

To a computer, a song isn't notes and words — it can be drawn as a picture called a spectrogram: time runs left to right, low sounds sit at the bottom, high sounds at the top, and darker means louder. Once sound is a picture, “separate the voice” becomes something much more familiar: find the shapes that belong to the voice, and copy them onto a new canvas.

And here's the trick that makes it feel possible: each instrument leaves a different visual signature. Click through the stems below — you'll learn to read this picture in about twenty seconds.

A recording studio mixes: four separate performances go in, one song comes out. Demucs is that console run in reverse — from the finished song, it rebuilds all four channels. Here they are. Mute and un-mute them while the song plays; with all four on, you're hearing the machine's four guesses added back together.

Demucs listens to the song in two ways at once. One branch reads the spectrogram — the picture you just learned. The other reads the raw waveform — the sound wave itself, which keeps fine timing details the picture blurs. Two views of the same song, like reading the sheet music and hearing the performance.

Each branch is a funnel: a stack of layers that squeezes the input into a smaller, smarter summary. Early layers see raw detail; deeper layers see concepts — “this region behaves like a voice.” In the middle the two branches compare notes, then everything runs backwards through a mirror-image funnel that paints four clean stems.

Don't take the funnel on faith — look inside it. While Demucs separated this song's chorus, we photographed what its picture-ear layers actually passed forward. These are real network activations, not illustrations. Watch the image get shorter (frequencies squeezed: 512 rows → 8) while the network's private notebook gets thicker (48 → 384 channels of “what I noticed”).

Nobody programmed rules like “voices are wavy lines.” Demucs learned by practicing on songs where the answer was known — studio multitracks where the isolated voice, drums and bass already exist as separate files. The recipe:

Why it still isn't perfect: instruments that share the voice's territory (a lead guitar, backing harmonies) can smear across stems, and anything the training data rarely contained — rare instruments, heavy effects — confuses it. That's why “everything else” is a catch-all bucket, and why a little vocal bleed remains. As you heard in the overview, though: clean enough to cut the lyric error rate meaningfully.