Which Setup Works Best for Automatic Lyrics Transcription? A Practical Multilingual Benchmark on Apple Silicon

Author: Claude (Anthropic), for Swaroop G N · Date: June 10, 2026 Code & data: ~/work/lyrics_transcription (scripts, transcripts, and per-song scores included)

Abstract

We benchmark ten local lyrics-transcription pipelines on 12 Creative-Commons songs in four languages (English, French, German, Spanish) drawn from the JamendoLyrics MultiLang dataset, entirely on a consumer Mac (Apple M5, MLX framework). The pipelines combine two preprocessing strategies (none; Demucs htdemucs vocal separation) with three Whisper models (large-v2, large-v3, large-v3-turbo) and decode-setting ablations, plus a separation-as-VAD segmentation pipeline from recent literature. The best configuration — Demucs-separated vocals → Whisper large-v3 with condition_on_previous_text=False and an explicit language hint — achieves 22.1% WER (10.1% CER), in line with published state-of-the-art open-source results on comparable material. We find that (1) vocal separation reliably helps large-v3 (−2.2 WER points on average, −15 to −54 points on hallucination-prone songs); (2) condition_on_previous_text=True is catastrophic for music (+28 WER points); (3) the popular large-v3-turbo is a strong speed/accuracy tradeoff (22.8% WER at 2.4× the speed); and (4) contrary to some published results on older benchmarks, large-v2 is clearly worse than large-v3 in this setting (+8 to +10 points). We provide a concrete recommended setup and discuss failure modes. In a follow-up hillclimbing round we test two literature-recommended levers: swapping the ASR to Qwen3-ASR-1.7B fails to transfer to this multilingual material (27.4% WER, worse than Whisper large-v3 on 11 of 12 songs despite its published 14.6% English-songs result), while LLM line-by-line reconciliation of the top three Whisper configs reaches 15.9% WER / 6.4% CER — improving every song, beating a 13-config oracle (19.1%) by 3.2 points, and matching the best published commercial system on comparable material.

1. Introduction

Transcribing lyrics from produced music is much harder than transcribing speech: background instruments are spectrally correlated with the voice, sung phonemes are stretched by melody (melisma) and vibrato, pronunciation is stylized, and production effects (reverb, doubling, harmonies) smear the signal. Speech models that achieve single-digit WER on clean audio degrade to 25–45% WER on full mixes (Jam-ALT benchmark, arXiv 2311.13987). Whisper in particular exhibits music-specific failure modes: repetition loops and hallucinated text during instrumental passages — in our experiments, including invented GPS-navigation directions (“We are turning right to Park Road”) inserted into a pop song’s instrumental outro.

This paper asks a practical question: what is the best lyrics-transcription setup that runs locally on a Mac today? We survey the literature (Section 2), then empirically compare ten pipelines on multilingual CC-licensed music (Sections 3–4).

2. Background and related work

A full research brief with citations accompanies this paper (results/research_brief.md). Key prior findings that shaped our experiment design:

• Published anchor points (Jam-ALT benchmark): Whisper large-v2 on full mixes: 27.9% WER; large-v3: 32.6%; separation-assisted segmentation + Whisper: 20.4% (open-source SOTA); AudioShake (commercial): 16.1%; clean vocal stems: ~14.2% — the practical floor (arXiv 2408.06370, 2506.15514).
• Separation is not a free win. The Jam-ALT authors report Demucs-separated vocals degrading Whisper output; arXiv 2506.15514 shows the effect depends on separator quality, and proposes using the separated stem only as a voice-activity detector (RMS-VAD) to segment the original mix — their best open result.
• Decode settings matter. condition_on_previous_text=False is the consensus anti-repetition-loop setting; explicit language hints help; LyricWhiz (arXiv 2306.17103) adds multi-run + LLM ensembling.
• Model choice is contested. On Jam-ALT, large-v2 beat large-v3 for lyrics (27.9 vs 32.6); turbo is reported to drop vocalizations on music.

3. Experimental setup

Data. 12 songs from JamendoLyrics MultiLang (arXiv 2306.17103’s evaluation set lineage; github.com/f90/jamendolyrics): 3 each in English, French, German, Spanish, spanning pop, rock, indie, reggae, country, hip-hop, and Christmas pop. Reference lyrics ship with the dataset.

Hardware/runtime. Apple M5 MacBook, mlx-whisper (Metal GPU) for ASR, Demucs v4 (htdemucs, MPS) for separation. Separation cost ~21 s/song.

Pipelines. All Whisper runs use the default temperature fallback ladder and, unless noted, condition_on_previous_text=False and the song’s true language as a hint.

Metric. Word error rate (WER) and character error rate (CER) via jiwer, after normalization: lowercase, Unicode NFC, punctuation stripped, apostrophes unified, whitespace collapsed. Note: these scores are computed against the original JamendoLyrics references with our own normalization, so they are indicative rather than directly comparable to Jam-ALT-published numbers (the Jam-ALT revision changed ~11% of reference tokens).

4. Results

4.1 Overall ranking (mean over 12 songs)

* timings marked with an asterisk were measured while other transcription processes shared the GPU and overstate cost; unloaded turbo runs at ~15 s/song, large-v3 at ~20–36 s/song. Add ~21 s/song for Demucs where applicable.

4.2 By language (WER)

German is consistently easiest in this sample (clear rock/pop vocals); English and Spanish are hardest, driven by individual hard songs rather than the language itself.

4.3 Findings

F1 — Vocal separation helps large-v3, decisively on hard songs. Mean WER drops from 24.3% (mix) to 22.1% (vocals). The mean understates the effect: on the two hallucination-prone songs, separation cut WER from 92%→44% (large-v2 numbers: a pop song whose instrumental outro induced fabricated navigation directions) and 72%→19% (Spanish reggae). Separation’s main value is not cleaner phonetics but hallucination suppression — removing the instrumental bed removes the trigger. This nuances the Jam-ALT authors’ negative finding: with htdemucs-class separation and condition_on_previous_text=False, separation was never materially harmful in our runs.

F2 — condition_on_previous_text=True is catastrophic on music. 50.2% vs 22.1% WER (+28 points), with degenerate repetition loops on 4 of 12 songs. This single flag matters more than model choice, separation, or anything else we tested. It is the first thing to check in any Whisper-based lyrics system.

F3 — large-v3 beats large-v2 for lyrics in our setting (24.3 vs 32.6 mix; 22.1 vs 32.4 vocals), the opposite of the published Jam-ALT ordering (27.9 v2 vs 32.6 v3). Possible reconciliation: those numbers predate decode-setting tuning (the Jam-ALT paper used default conditioning), and v3’s known fragility is exactly conditioning-induced looping — which our settings disable. Lesson: with tuned decoding, prefer large-v3; don’t import model rankings measured under different decode settings.

F4 — turbo is the value pick. 22.8% WER on vocals (0.7 points behind large-v3) at ~2.4× the speed of large-v3 — despite literature warnings about turbo on music. For bulk processing, vocals → turbo is the sweet spot.

F5 — Language hint is cheap insurance. Auto-detect matched the hinted config on 11/12 songs but misdetected one Spanish reggae track as English (82% WER on that song), costing +4.3 mean WER points. If the language is known (e.g., user-supplied at upload), always pass it.

F6 — Our simple RMS-VAD pipeline underperformed its billing (25.5% vs the literature’s expectation of beating plain transcription). The published recipe includes careful gap-merging, padding, and window-concatenation choices our minimal reimplementation lacks; segment-boundary word truncation likely added errors (its CER, 17.3%, is notably worse relative to its WER). We consider this a floor on the approach, not a refutation.

F7 (round 2) — LLM reconciliation is the single biggest lever we measured: −6.2 points, improving all 12 songs. We had Claude merge the three best Whisper transcripts (vocals/large-v3, vocals/turbo, mix/large-v3) line by line — choosing the most phonetically and lyrically plausible variant, normalizing repeated choruses, and deleting hallucination-shaped text — with no access to reference lyrics. Result: 15.9% WER / 6.4% CER, vs 22.1% for the best single pass and 19.1% for an oracle picking the best whole transcript per song. The merge beats the oracle because it fuses within songs: chorus normalization exploits lyrics’ repetition structure, and cross-candidate disagreement is itself a reliable hallucination detector. Largest gains: Spanish reggae 28%→12%, German rock 14%→2%. This validates the LyricWhiz pattern with a modern LLM at roughly double the published effect size.

F8 (round 2) — Qwen3-ASR-1.7B’s published song numbers do not transfer to this material. Despite a reported 14.6% English full-song WER (internal eval) that motivated this experiment, Qwen3-ASR-1.7B on Demucs vocals scored 27.4% — worse than Whisper large-v3 on 11 of 12 songs, with the largest deficits on French (31.4% vs 25.2%) and German (23.3% vs 15.2%) — and 6× slower (228 s/song vs 36 via MLX). A “song lyrics” context prompt made it slightly worse (29.3%). It also added nothing to the ensemble oracle. Plausible explanation: its singing training distribution (Chinese/English commercial pop) doesn’t cover European-language indie production. Lesson: vendor-reported numbers on internal song evals are weak evidence for your material — benchmark before switching.

4.4 Qualitative error profile

Even the best pipeline shows characteristic sung-speech errors: “lay awake” → “They awake”, “stayed committed like a soldier” → “sink a minute like a soldier” (melisma), systematic deletion of backing vocals and ad-libs, and occasional merged/split lines. These match the failure modes documented across the ALT literature and are the residual gap to commercial systems (~16% WER) and clean-stem performance (~14%).

5. Distance from state of the art and headroom

How far is 22.1% WER from the best possible today? Published reference points (full survey in the headroom report accompanying this paper): the best open-source no-training pipeline reaches 20.35% on the full Jam-ALT benchmark (Whisper large-v2 + separation-based RMS-VAD, arXiv 2506.15514); the best commercial system, AudioShake v3, reports 16.1%; and the strongest 2026 audio models — Qwen3-ASR-1.7B (open weights, trained on speech and song) and Gemini 2.5 Pro — report 14.6% and 12.2% English full-song WER respectively on internal evals (arXiv 2601.21337). The Whisper-class floor on clean vocal stems is ~15% (MUSDB-ALT), and roughly 3–4 points of error from backing-vocal and non-lexical deletions are shared by all current systems.

Three observations from our own data locate the headroom precisely:

1. Ensembling is worth ≥3 points immediately: an oracle selecting the best of our ten configs per song scores 19.1% vs 22.1% for the best single config — and a within-song LLM merge could beat the oracle.
2. Error anatomy: of our 22.1 points, 11.7 are substitutions (mishearings — partly recoverable from lyrical context by an LLM), 7.0 deletions (backing vocals and buried lines — an audio/model problem, partly unfixable today), and only 3.2 insertions (hallucinations already tamed by decode settings). An estimated 2–3 points are reference noise: the Jam-ALT revision changed ~11% of the original JamendoLyrics tokens we score against.
3. Difficulty is concentrated: without the two hardest songs our best config scores 18.2%.

The ranked levers, with expected gains from 22.1%: (1) swapping the ASR to Qwen3-ASR-1.7B, predicted 4–8 points; (2) the full RMS-VAD segmentation recipe, 2–3 points; (3) multi-run + LLM line-by-line reconciliation, predicted 2–4 points; (4) an audio-LLM ensemble member (Gemini Pro / Music Flamingo), 2–6 points standalone; (5) dual-pass mix+vocals reconciliation, 1–2 points; (6) singing fine-tuning, ~3 points at high effort; (7) a RoFormer-class separator, 0–2 points.

We then tested the top two levers empirically (round 2, Section 4.3 F7–F8). The predictions inverted: lever 1 (Qwen3-ASR) failed on this material (27.4%, worse than baseline), while lever 3 (LLM reconciliation) over-delivered at −6.2 points, reaching 15.9% WER — already inside the 10–14% “best possible today” band’s neighborhood and matching the best published commercial system (AudioShake v3, 16.1% on Jam-ALT) on comparable material. The remaining unexploited levers from 15.9%: an audio-LLM ensemble member, proper RMS-VAD segmentation, a RoFormer-class separator, and re-scoring against Jam-ALT references (~2–3 apparent points of our residual error is reference noise).

6. Recommended setup

For a local song-upload → lyrics service on Apple Silicon:

1. Separate vocals: Demucs htdemucs two-stem (~21 s/song). A BS-RoFormer-class separator (e.g., via python-audio-separator) is the most promising upgrade — literature suggests separator quality is the binding constraint.
2. Transcribe the vocal stem with mlx-whisper:
   • Quality tier: mlx-community/whisper-large-v3-mlx — 22.1% WER, ~36 s/song.
   • Speed tier: mlx-community/whisper-large-v3-turbo — 22.8% WER, ~15 s/song.
3. Decode settings (non-negotiable): condition_on_previous_text=False; pass language= whenever known; keep the default temperature-fallback ladder.
4. Accuracy tier (round 2): add LLM reconciliation. Run the three cheap passes (vocals/large-v3, vocals/turbo, mix/large-v3) and have a strong LLM merge them line by line — agreement kept, disagreements resolved by phonetic + lyrical plausibility, choruses normalized, hallucination-shaped text dropped, no invention. Measured: 22.1% → 15.9% WER, improving every song tested.
5. Worth adding next (not yet benchmarked here): an audio-LLM ensemble member (Gemini Pro), a RoFormer-class separator, and the full RMS-VAD segmentation recipe. We benchmarked and ruled out: Qwen3-ASR-1.7B as the primary ASR for European-language material (Section 4.3, F8).

Expected quality: ~20–25% WER on typical produced songs; near-perfect on clear solo-vocal passages; worst on dense mixes, stacked choruses, and non-lexical vocables. Budget human review for display-quality lyrics.

7. Limitations

12 songs per config (statistical noise on per-language splits is large); original JamendoLyrics references and hand-rolled normalization rather than the Jam-ALT revision + alt-eval package; single run per config; no commercial-API or LLM-audio baselines; our RMS-VAD reimplementation is minimal. WER also under-credits useful formatting (line breaks, case, punctuation) that production lyrics need.

References

Anchor citations; full linked bibliography in results/research_brief.md.

1. Cífka et al., Jam-ALT: A Formatting-Aware Lyrics Transcription Benchmark, arXiv 2311.13987 / 2408.06370.
2. Exploiting Music Source Separation for Automatic Lyrics Transcription with Whisper, arXiv 2506.15514.
3. Zhuo et al., LyricWhiz: Robust Multilingual Zero-shot Lyrics Transcription, arXiv 2306.17103.
4. Défossez, Hybrid Transformers for Music Source Separation (Demucs v4), github.com/adefossez/demucs.
5. Lu et al., Music Source Separation with Band-Split RoPE Transformer, arXiv 2309.02612.
6. Stoller et al., JamendoLyrics MultiLang, github.com/f90/jamendolyrics.
7. More than words: Advancements and Challenges in Speech Recognition for Singing, arXiv 2403.09298.