Automatic Lyrics Transcription (ALT): 2025–2026 Engineering Research Brief

Scope: practical best-in-class setups for transcribing lyrics from full music mixes, with published benchmark numbers and a concrete Apple Silicon recommendation.

Headline findings (TL;DR): - Naive Whisper-large on a full mix gets ~28–36% WER on the standard Jam-ALT benchmark; the best published open pipeline (source-separation-assisted segmentation + Whisper) gets ~20.4%; the best published commercial system (AudioShake) gets ~16% (Jam-ALT/ICME 2025, arXiv 2506.15514). - Source separation is not a free win for Whisper: feeding separated vocals directly can degrade results unless the separator is high quality; the most robust use of separation is as a vocal-activity detector for segmentation, transcribing segments of the original mix. - Whisper large-v2 often beats large-v3 on lyrics, and large-v3-turbo is measurably worse on music. Use condition_on_previous_text=False to kill repetition loops.

1. Why lyrics transcription is harder than speech ASR

Singing breaks most assumptions baked into speech-trained ASR models. The survey “More than words: Advancements and Challenges in Speech Recognition for Singing” (arXiv 2403.09298) and the foundational “Automatic Lyrics Alignment and Transcription in Polyphonic Music: Does Background Music Help?” (arXiv 1909.10200) identify the core problems:

• Background accompaniment: harmonic and percussive instruments add spectral components that overlap and correlate with the vocal (they’re in the same key and rhythm), unlike typical uncorrelated speech noise. This is the single largest degradation factor (arXiv 1909.10200).
• Pitch and duration distortion: singing spans far wider pitch ranges with vibrato; phonemes—especially vowels—are elongated or compressed to fit melody and rhythm (melisma stretches one syllable across many notes), confirmed by phoneme-duration analysis in the survey (arXiv 2403.09298). The same text sung to different melodies yields different acoustics, making training data effectively sparse (PDAugment, arXiv 2109.07940).
• Pronunciation and lexicon shift: singers deliberately alter pronunciation; lyrics use vocabulary, repetition structures, and non-lexical vocables (“oh-oh-oh”, “na na”) rare in speech corpora.
• Production effects: reverb, delay, doubling, backing vocals/harmonies, and compression smear the signal. Whisper systematically deletes backing vocals and non-lexical vocables regardless of preprocessing (arXiv 2506.15514).
• Foundation-model degradation is measurable: Whisper that achieves single-digit WER on clean speech lands at 25–45% WER on full mixes (Jam-ALT, arXiv 2311.13987).

2. Vocal source separation as preprocessing — does it help?

Separator quality landscape (MUSDB18-HQ vocals SDR)

• Spleeter (Deezer, 2019): obsolete for quality (~6.9 dB vocals); only relevant for speed.
• Hybrid Demucs / htdemucs (Meta, github.com/adefossez/demucs): mdx ≈ 7.97 dB, mdx_extra ≈ 8.76 dB vocals SDR (as measured in arXiv 2506.15514); htdemucs_ft is the fine-tuned v4 default.
• MDX-Net / KUIELab (arXiv 2111.12203): SDX-challenge-era frequency-domain models; many checkpoints live on in the UVR ecosystem.
• BS-RoFormer (ByteDance, arXiv 2309.02612): current SOTA — 9.80 dB average SDR on MUSDB18-HQ without extra data, ~+1.3 dB over htdemucs; won SDX’23. Community checkpoints (e.g., model_bs_roformer_ep_317_sdr_12.97) reach ~12.9 dB vocals SDR and are downloadable via python-audio-separator. Mel-Band RoFormer (arXiv 2310.01809) variants are similar; MVSEP’s leaderboard tracks the practical SOTA.

Does separating before ASR reduce WER? The evidence is nuanced.

• Pre-Whisper era: arXiv 1909.10200 found that training acoustic models on polyphonic mixtures beat extraction-based pipelines — separation artifacts hurt more than music helps confuse.
• Against (Whisper, naive use): The Jam-ALT authors found that HTDemucs-isolated vocals substantially degraded Whisper results, especially v3 — on separated vocals “Whisper has a general tendency to omit parts of the lyrics (often the entire song) and instead produce generic or irrelevant text” (arXiv 2408.06370). Whisper was trained on noisy real-world audio; sparse, artifact-laden solo vocals are out-of-distribution.
• For (with care): The systematic study “Exploiting Music Source Separation for ALT with Whisper” (arXiv 2506.15514, Whisper large-v2, beam 5, language given) found:
  • Short-form, Jam-ALT: mix 20.99% vs separated 21.08–21.17% WER — no real gain.
  • Short-form, MUSDB-ALT: mix 23.59% → 20.00% with the better mdx_extra separator — separation quality is the deciding variable.
  • Clean vocal stems: 14.19% WER — the ~6-point gap between separated vocals and true stems shows separation artifacts (not the concept) are the bottleneck.
  • Best use = separation as VAD: their “RMS-VAD” — compute energy on the separated vocal to find sung segments, then transcribe — gives consistent gains and 20.35% WER on Jam-ALT long-form, SOTA for open-source, with zero training.
• For (with fine-tuned ASR): SongPrep reports Whisper improving from ~47% on mixtures to ~28% on separated vocals on their SSLD-200 set, and pipelines like VietLyrics separate first then fine-tune.

Engineering takeaway: with stock Whisper, (a) always use separation to segment (VAD); (b) only feed separated vocals to Whisper if your separator is BS-RoFormer-class (≥~9–12 dB vocals SDR); (c) consider transcribing both mix and separated vocals and reconciling — that’s effectively what commercial systems and LyricWhiz-style ensembles exploit.

3. ASR models for singing voice

Whisper family behavior on sung vocals

• Hallucinations/loops: instrumental intros, solos, and outros trigger ghost text, repetition loops (“burning through the sky” repeated), spurious “Thank you”/“(singing in foreign language)” tokens. Documented across openai/whisper #679, whisper.cpp #2286, and a music-specific comparison on Queen’s “Don’t Stop Me Now” (whisper.cpp #3074).
• large-v2 vs large-v3: counterintuitively, v3 is not better for lyrics. On Jam-ALT with language hints, v2 = 27.9% WER vs v3 = 32.6% (arXiv 2408.06370); the earlier benchmark run had them near-tied (35.7 vs 35.5, arXiv 2311.13987). v3 is also more fragile on separated vocals. Benchmark both; default to large-v2 for lyrics.
• large-v3-turbo: 4 decoder layers (vs 32) — fine for speech (within ~0.4 pt of v3, HF model card), but on music it misses vocalizations and flubs endings; the Queen test ranked large-v3 best for music fidelity (whisper.cpp #3074).
• Runtimes: faster-whisper (CTranslate2, built-in Silero VAD, ~4x faster, default beam 5); whisper.cpp (portable, GGML quantization); mlx-whisper (Apple MLX, Metal GPU) — ~2.0x faster than whisper.cpp on Apple Silicon with large-v3-turbo (billmill benchmark, Jan 2026); lightning-whisper-mlx claims further speedups; mac-whisper-speedtest lets you benchmark locally.

Singing-specific systems

• LyricWhiz (arXiv 2306.17103, HTML v4): training-free — Whisper-large run 3–5×, prompt "lyrics:", drop outputs with no-speech prob > 0.9 (plus PANNs audio-event filtering), then GPT-4 ensembles/corrects the candidates. Jamendo 24.25% WER (prior SOTA W2V2-ALT: 33.13%), Hansen 7.85%, MUSDB18 26.29%. No source separation used. The “Whisper-ear + LLM-brain, multi-run voting” pattern remains the best zero-training accuracy trick.
• Fine-tunes: no widely-adopted general-purpose singing Whisper checkpoint exists on HuggingFace; published fine-tunes are language-specific (e.g., VietLyrics fine-tuned large-v2: 20.5% lowercase WER on Vietnamese songs). Several papers deliberately skip fine-tuning because Whisper generalizes well (arXiv 2506.15514).
• End-to-end research models: SongPrep/SongPrepE2E (arXiv 2509.17404) — full-song structure + lyrics, 24.3% WER on SSLD-200 (vs Whisper 27.7%), dataset released but not weights. OWSM v3.1 does badly on lyrics (69.3% WER, arXiv 2408.06370).
• Commercial: AudioShake is the only vendor publishing lyrics WER: v3 = 16.1% WER on Jam-ALT (case-sensitive 20.1%), a 57% reduction vs Whisper v2, best on every formatting metric (AudioShake blog, arXiv 2408.06370). AssemblyAI’s own blog testing 15 songs found WERs of 0.47–0.88 — i.e., not usable for lyrics (AssemblyAI blog). Deepgram publishes nothing on music. Gemini 2.5 Pro / GPT-4o-audio have no published Jam-ALT numbers; NVIDIA’s Music Flamingo (arXiv 2511.10289) reports beating both on lyrics transcription (19.6% WER English, 12.9% Chinese on its eval). Anecdotally Gemini handles full songs well, but treat it as unbenchmarked; it’s worth including in your own bake-off.

4. Known best practices (decode-time)

1. condition_on_previous_text=False — the single most important anti-repetition-loop setting for music; instrumental gaps otherwise poison subsequent windows (whisper discussion #679, Memo AI writeup). Cost: loses cross-segment context (slightly worse consistency).
2. Language hint — always pass language=; Jam-ALT shows +lang improves Whisper and avoids per-segment misdetection on sung vowels (arXiv 2408.06370).
3. VAD before Whisper — don’t let Whisper see instrumental-only audio. Either faster-whisper’s built-in Silero vad_filter=True (faster-whisper) or, better for music, energy/RMS-VAD computed on the separated vocal stem, then cut the original mix at those boundaries — this is the published SOTA open recipe (20.35% Jam-ALT WER, arXiv 2506.15514). Note Silero is speech-trained and can be unreliable on singing; vocal-stem RMS sidesteps that.
4. Temperature fallback — keep the default ladder (0.0, 0.2, …, 1.0) with compression_ratio_threshold=2.4 and logprob_threshold=-1.0; these gate the retry that escapes degenerate beams. Greedy/beam at T=0 plus fallback beats fixed nonzero temperature.
5. no_speech_threshold — lower it (e.g., 0.4–0.6) and drop high no-speech-probability segments; LyricWhiz dropped predictions with no-speech prob > 0.9 (arXiv 2306.17103v4).
6. initial_prompt — LyricWhiz’s "lyrics:" prompt (localized: “paroles”, “Liedtext”) nudges the decoder toward lyric register; you can also seed artist/title or known chorus lines. Beware: prompts can also seed hallucinations on silent stretches.
7. Chunking — prefer segment boundaries at vocal-activity gaps (from VAD above) padded to ≤30 s windows; for short-form scoring, the concatenation approach in arXiv 2506.15514 reduced WER consistently. hallucination_silence_threshold (openai/mlx implementations) helps skip long silences when using word timestamps.
8. Multi-run + LLM rerank (optional, biggest accuracy lever after separation): run 3–5 decodes (vary temperature/seed), have an LLM merge — LyricWhiz pattern, worth ~5–10 WER points on hard genres (arXiv 2306.17103).

5. Evaluation methodology

• Metrics: word-level WER is standard; CER for logographic languages (Mandarin lyrics work uses CER). Lyrics raise normalization landmines: case, punctuation, line/section breaks, parenthesized backing vocals, and non-lexical vocables are all meaningful content for display but destroy naive WER comparability.
• Jam-ALT (arXiv 2311.13987, project page, HF dataset) is the de-facto 2024–2026 benchmark: a revision of JamendoLyrics MultiLang with an industry-style annotation guide (Apple/Musixmatch/LyricFind conventions) and an eval package (alt-eval) that defines WER, case-sensitive WER’, case error rate, punctuation-F, parentheses-F (backing vocals), line-break-F and section-break-F. The revision alone amounts to 11.1% WER-worth of changes vs the original JamendoLyrics — i.e., numbers on “JamendoLyrics” vs “Jam-ALT” are not comparable.
• JamendoLyrics MultiLang (github f90/jamendolyrics, now deprecated in favor of HuggingFace hosting): 79–80 songs, EN/FR/DE/ES, word-level timing — primarily an alignment benchmark; Jam-ALT is its transcription-focused descendant.
• MUSDB-ALT (arXiv 2506.15514): new long-form lyrics benchmark where true vocal stems are public — uniquely lets you measure separation-artifact cost.
• Published WER anchor points (Jam-ALT, overall unless noted):

Per-language (Whisper v2 +lang): EN 39.7, ES 21.9, DE 19.9, FR 27.1 — English is paradoxically hardest in this set (genre mix). Expect 20–25% WER from a good open pipeline, 35–50%+ on rock/metal/heavy mixes, near-15% only with clean stems or commercial systems.

6. Concrete recommendation: Mac (Apple Silicon) pipeline

Primary recommendation (best published-evidence open pipeline):

1. Separate vocals with a RoFormer-class model via python-audio-separator (pip install "audio-separator[cpu]" — uses CoreML execution provider on macOS; e.g. model_bs_roformer_ep_317_sdr_12.97.ckpt), or the MLX-native fork mlx-audio-separator. Fallback: demucs -n htdemucs_ft (demucs).

2. Segment using the separated vocal as a VAD: RMS-energy gate on the vocal stem → sung-region boundaries, merge gaps < ~1 s, pad ~0.3 s, cap windows at 30 s (recipe per arXiv 2506.15514).

3. Transcribe with mlx-whisper, model mlx-community/whisper-large-v2-mlx (benchmark v3 too; v2 is the safer lyrics default), per segment:

   mlx_whisper.transcribe(
       seg_audio,
       path_or_hf_repo="mlx-community/whisper-large-v2-mlx",
       language="en",                      # always hint
       condition_on_previous_text=False,   # kills repetition loops
       temperature=(0.0, 0.2, 0.4, 0.6, 0.8, 1.0),  # fallback ladder
       compression_ratio_threshold=2.4,
       logprob_threshold=-1.0,
       no_speech_threshold=0.5,
       initial_prompt="lyrics:",           # LyricWhiz trick; A/B it
   )

4. Feed segments from the original mix by default; if your separator is BS-RoFormer-class, also transcribe the separated-vocal segments and reconcile the two passes with an LLM (LyricWhiz pattern: present both candidates, ask for the most plausible lyric line). Expected: ~20% WER territory on Jam-ALT-like material; the LLM merge is your path below that.

Runner-up configurations to benchmark against: - faster-whisper large-v2/v3 with vad_filter=True, beam_size=5, condition_on_previous_text=False on the mix — simplest credible baseline (faster-whisper). - whisper.cpp large-v3 (not turbo — turbo demonstrably drops vocalizations on music, whisper.cpp #3074) if you want a zero-Python binary; expect ~2x slower than MLX on M-series (benchmark). - Gemini 2.5/3 Pro full-song audio prompt — unbenchmarked on Jam-ALT but cheap to test and strong anecdotally; include in your bake-off. - AudioShake API if budget allows — only system with published ~16% Jam-ALT WER plus correct line breaks/casing (AudioShake).

Evaluate with alt-eval on the jam-alt dataset (and MUSDB-ALT if you care about long-form), so your numbers are comparable to the literature; don’t hand-roll WER normalization.

Known residual failure modes no pipeline fixes today: backing vocals/harmonies (systematically deleted), non-lexical vocables, heavily distorted genres (rock/metal), and choruses with stacked vocals — budget for human review there.