Inside the listening machine

The clean vocal track from Step 1 doesn't enter the model as sound. It's converted into a mel spectrogram — a cousin of the picture you learned to read in the Demucs deep-dive, but rebuilt to match human hearing: 128 frequency bands spaced the way our ears space them, snapshotted 100 times per second.

This is the entire input. Every word the model writes, it writes by staring at this ribbon of teal:

The model doesn't hear a line and translate it. It plays a guessing game, thousands of times: “given everything I've heard, and the words I've written so far — what's the most likely next word?” That means there's a small language model inside every speech model, and it cuts both ways: it fills gaps a human would fill (helpful), and it can write something fluent that was never sung (we'll catch it red-handed in section C).

Crucially, each guess comes with a confidence — the model's own probability that the word is right. We usually throw that number away. Below, we kept it: every word is shaded by how sure Whisper actually was while transcribing our song. Press play; click any word for its exact number.

Speech models are not interchangeable. This laptop has two quite different ones: Whisper (OpenAI), a transcription specialist trained on 680,000 hours of audio, and Qwen3-ASR (Alibaba), a newer 1.7-billion-parameter model with a full LLM brain inside — and a published claim of strong song-lyrics accuracy. We gave both the identical vocal track.

“How do we make the lyrics better?” is really a ladder, and its rungs are ordered by cost and reliability — every number below was measured on this benchmark. The striking fact: the error rate fell from 50% to 16% without changing a single weight in any model.

What it means: fine-tuning resumes the model's original training loop — play audio, grade the guess against a known answer, nudge millions of weights — but on your material: sung vocals paired with verified lyrics. Both Whisper and Qwen3-ASR are open-weights, so this is genuinely possible, even on this laptop for smaller variants (LoRA-style updates touch only a sliver of the weights).

What it demands: the answer keys. Hundreds of hours of songs with accurate, rights-cleared lyrics, time-aligned — the scarcest asset in this field, and the real moat of commercial providers. Plus GPU time, and the discipline to re-benchmark after every run.

What it risks: a model tuned on one catalogue drifts — it gets better at your genre and quietly worse at everything else, and you now own a custom model to maintain, re-validate and re-tune every time the base model improves.

When it's the right call: a fixed, recurring domain — one label's catalogue, one language tradition, recurring artist vocabulary — where the ladder's cheaper rungs have genuinely been exhausted and the remaining errors are systematic (the model consistently mishears a dialect or genre convention rather than random words).

The honest summary for a decision-maker: everything this site achieved — 50% error down to 16%, matching commercial systems — came from rungs 1–4: settings, cleaner input, and ensembling, all without training anything. Fine-tuning is the most expensive rung with the least predictable payoff. Climb it last, and only with a benchmark like this one already in place to prove it paid.