Date: June 10, 2026 · Companion to
paper.md (our result: 22.1% WER, htdemucs → Whisper
large-v3)
Our 22.1% WER is ~2 points off the best published open-source no-training result (20.35%, Whisper large-v2 + separation-based RMS-VAD on the full 79-song Jam-ALT set), ~6 points off the best commercial system (AudioShake v3, 16.1% Jam-ALT), and ~8–10 points off the best audio-LLM numbers on full songs (Gemini 2.5 Pro 12.2% EN; Qwen3-ASR 14.6% EN, internal evals). Roughly 2–4 of our 22.1 points are likely not model error at all: reference noise in the original JamendoLyrics annotations plus systematic backing-vocal/non-lexical deletions that all current systems share. A realistic “best possible today” on our material is ~12–16% WER, with ~10% as the practical floor for full mixes.
Combining our own error decomposition with published analyses:
| Error bucket | Points (approx.) | Fixable by |
|---|---|---|
| Substitutions (mishearings) | 11.7 | better acoustic model, LLM correction |
| Deletions (backing vocals, buried/non-lexical lines) | 7.0 | partly unfixable (~3–4 pts systematic across all systems); rest: better separation/model |
| Insertions (residual hallucination) | 3.2 | ensembling, VAD |
| — of which reference noise (original JamendoLyrics vs Jam-ALT revision changed ~11% of tokens) | ~2–3 | re-scoring against Jam-ALT refs with alt-eval |
Two more facts from our own runs sharpen this: an oracle picking the best of our 10 configs per song scores 19.1% (so pass-ensembling is worth ≥3 points before any new model), and our best config without the two hardest songs scores 18.2% — the headroom is concentrated in a minority of hard songs.
| System | WER | Where |
|---|---|---|
| Whisper large-v2 + RMS-VAD (open, no training) | 20.35% | Jam-ALT, arXiv 2506.15514 |
| AudioShake v3 (commercial API, separation-first) | 16.1% | Jam-ALT, audioshake.ai |
| Qwen3-ASR-1.7B (open weights Apache 2.0, Jan 2026; trained on speech+song, 52 languages) | 14.6% EN full songs | internal eval, arXiv 2601.21337 |
| Gemini 2.5 Pro (full-song audio prompt) | 12.2% EN | same eval |
| NVIDIA Music Flamingo (8B, open weights) | 19.6% EN | MUSDB18, arXiv 2511.10289 |
| GPT-4o-Transcribe | 30.7% EN | same eval (poor on music) |
| Whisper-class floor on clean vocal stems | ~15.0% | MUSDB-ALT, arXiv 2506.15514 |
Notable: no published paper has beaten 20.35% on Jam-ALT itself yet, but Qwen3-ASR has very likely leapfrogged it un-benchmarked; and generic 2026 speech models (Parakeet, Canary, Voxtral) top speech leaderboards but are not better on singing.
| # | Lever | Expected gain | Effort |
|---|---|---|---|
| 1 | Swap ASR to Qwen3-ASR-1.7B (local, Apache 2.0) for vocals + mix passes | 4–8 pts → ~14–18% | Low |
| 2 | Proper RMS-VAD segmentation (full recipe from arXiv 2506.15514, not our minimal version) | 2–3 pts | Low |
| 3 | Multi-run ensemble + LLM line-by-line reconciliation (LyricWhiz pattern, modern LLM) | 2–4 pts | Medium |
| 4 | Audio-LLM pass (Gemini 2.5/3 Pro API, or local Music Flamingo) as ensemble member | 2–6 pts standalone | Low–Medium |
| 5 | Dual-pass mix+vocals reconciliation (our data: per-song winners differ) | 1–2 pts | Low |
| 6 | Fine-tune Whisper on singing data (DALI v2, MulJam) | ~3 pts | High |
| 7 | RoFormer-class separator instead of htdemucs | 0–2 pts | Low |
| 8 | Re-score against Jam-ALT references with alt-eval
(measurement fix, not model fix) |
~2–3 pts apparent | Trivial |
Levers 1+2+5 are roughly additive at first; 3 and 4 overlap with each other and with 1.
API allowed (estimated 10–14% WER): AudioShake API + Gemini 2.5/3 Pro on the mix + Qwen3-ASR on RoFormer-separated vocals, merged line-by-line by an LLM with phonetic-plausibility instructions and explicit “keep backing vocals and non-lexical vocables” prompting.
Fully local (estimated 13–17%): Mel-RoFormer vocals → Qwen3-ASR-1.7B dual-pass (vocals + mix) with RMS-VAD segmentation → local LLM tie-breaker/corrector.
Single highest-value next experiment: run Qwen3-ASR-1.7B on our existing 12 songs (both the htdemucs vocals and raw mixes) — one afternoon of work, expected landing zone 14–18% WER, the largest single lever available.
We tested the top two levers from the table above. The predictions inverted:
| Lever | Predicted | Measured |
|---|---|---|
| #1 Qwen3-ASR-1.7B (vocals input, mlx, language hint) | 4–8 pts gain → 14–18% | −5.3 pts (WORSE): 27.4%, behind Whisper large-v3 on 11/12 songs; FR/DE hit hardest; 6× slower; “song lyrics” context prompt made it worse still (29.3%) |
| #3 LLM reconciliation (Claude merges vocals/large-v3 + vocals/turbo + mix/large-v3, line-by-line, no reference access) | 2–4 pts gain | +6.2 pts gain: 15.9% WER / 6.4% CER, improving all 12 songs, beating the 13-config oracle (19.1%) and matching AudioShake v3’s published 16.1% on comparable material |
Two lessons: (1) vendor-published internal song WERs are weak evidence for your own material — Qwen3-ASR’s singing distribution apparently does not cover European-language indie production; benchmark before switching. (2) Lyrics’ repetition structure makes LLM merging unusually effective — chorus normalization plus disagreement-as-hallucination-detector delivers roughly double the literature’s reported ensemble gain.
Updated best local recipe: Demucs vocals → Whisper large-v3 (+turbo, +mix passes) → LLM line-by-line merge = 15.9% WER. Remaining unexploited levers from here: audio-LLM ensemble member (Gemini Pro), proper RMS-VAD segmentation, RoFormer-class separator, Jam-ALT re-scoring (~2–3 apparent points).
Jam-ALT benchmark: arXiv 2408.06370, audioshake.github.io/jam-alt · Separation/VAD study and MUSDB-ALT: arXiv 2506.15514 · Qwen3-ASR: arXiv 2601.21337, github.com/QwenLM/Qwen3-ASR · Music Flamingo: arXiv 2511.10289 · LyricWhiz: arXiv 2306.17103 · Singing fine-tuning: arXiv 2506.02339, 2510.22295 · AudioShake: audioshake.ai benchmark posts · SongPrep/SSLD-200: arXiv 2509.17404 · LLM over-correction caution: arXiv 2505.17410.