We Used Benchmaxxers' Favourite Trick to Climb 10 Places on the Hugging Face Open ASR Leaderboard

TL;DR: There is a well-known trick for climbing ASR leaderboards: fine-tune on data that closely resembles a testset, submit, watch your rank improve. We did exactly that with AMI, the one testset that consistently separates serious models from optimised ones. We went from rank 26 to rank 16. Here is what actually happened, what broke along the way, and what it means for how you read anyone's leaderboard score. Everyone knows leaderboards get gamed Public benchmarks have a problem. The moment a testset becomes the standard measure of model quality, it also becomes the optimisation target. Teams fine-tune on training data that resembles the testset, run inference configurations tuned specifically for evaluation conditions, and submit. The number goes up. The leaderboard moves. Whether the model has actually improved at the underlying task is a separate question. This is not a secret. It is just rarely talked about openly, because talking about it requires admitting you did it. We did it. We used AMI the testset that every serious benchmaxxer on the HF Open ASR Leaderboard targets to climb 10 places in the rankings. We went from rank 26 (WER 6.90) to rank 16 (WER 6.38), sitting between NVIDIA's Parakeet and Kyutai's 2.6B model. The work is still in progress and there are likely more gains available. But the process surfaced enough interesting findings, including a few things that failed spectacularly, that it is worth writing up now. The more useful story here is not the rank. It is what the AMI testset reveals about real-world ASR quality, why it is the one testset vendors should not be able to fake, and what happens when you push the optimisation too far. If you are a PM or technical buyer evaluating speech APIs, the practical takeaway is in the final section: leaderboard rank is a useful signal, but only if you know which testsets drove the score, whether the model was tuned to the benchmark, and whether the same setup holds up in production. The technical detail in between is the proof of why those questions matter. Why AMI is the testset everyone targets The HF Open ASR Leaderboard scores models across eight testsets under a single normalised WER metric. Most of them are tractable. LibriSpeech is read speech in a quiet room. TED-LIUM is prepared talks. SPGISpeech is financial calls with relatively clean audio. Models that have been through standard training pipelines score reasonably well on these without any special effort. AMI is different. AMI is the ICSI Meeting Corpus: spontaneous conversations, overlapping speakers, informal register, variable microphone quality, people talking over each other. It is the kind of audio that shows up in real meeting transcription products, contact centre recordings, and any application where the speaker is not reading from a script and does not know they are being evaluated. It is also the testset with the highest variance between models on the leaderboard. Clean-audio scores cluster tightly. AMI scores spread wide. A model that scores competitively on LibriSpeech but has not been exposed to conversational audio will show it immediately on AMI. This is why it is both the most gamed testset and the most meaningful one. Improving your AMI score either means your model genuinely handles hard conversational audio better, or you found training data close enough to the testset to inflate the number. The two are not always easy to distinguish from outside. From inside, it is obvious. What we did Our starting point was the publicly submitted speechmatics/enhanced model at rank 26, WER 6.90. Before any new fine-tuning, we worked on the inference path, specifically how long audio gets split and processed. The standard approach is hard splitting: cut audio into fixed chunks, transcribe each independently, concatenate. It is simple and it loses words at every boundary. We implemented overlap correction using the Triton orchestrator: overlap consecutive chunks and merge transcripts by finding the region of agreement between passes. Combined with properly-plumbed language hints (more on that shortly), this moved the score to 6.76 , rank 25. Then came the AMI fine-tune. The approach was direct: take the current AED model and fine-tune it for approximately 6,000 steps on training data from the AMI corpus the same corpus the testset is drawn from. This is the benchmaxxer's move. The training data is not the testset itself, but it is close enough that the model learns the acoustic and linguistic patterns specific to that domain. AMI WER dropped from 13.48 to 9.58 . Combined with overlap correction and language hints, the final score was 6.38, rank 16 . For context, that AMI improvement (from 13.48 to 9.58) is roughly one in three errors eliminated on spontaneous conversational audio. If you are building meeting transcription or anything involving real spoken dialogue, that number matters. Whether it reflects a genuinely better model or a domain-adapted one is a fair question, and we will come back to it. \ What broke along the way The fine-tune kept improving on AMI past the 6,000-step checkpoint. By step 15,070, hard-splitting AMI WER had reached 8.94, a real gain over 9.51 at step 6,028. So we tried applying overlap correction and language hints to the later checkpoint. AMI WER jumped to 15.55 . The cause took time to diagnose. The Triton orchestrator hardcodes the production body: an int8-quantized build. The fine-tuned head was trained against a floating-point variant of the same body. At early checkpoints, the head stays close enough to the base model that the quantization calibration holds. As fine-tuning continues and the head drifts further from base, the calibration breaks. The overlap-correction path, which relies on consistent decoder behaviour across consecutive passes, starts producing cascading duplications. | Checkpoint | Hard splitting AMI WER | Overlap correction + langhint AMI WER | Delta | |----|----|----|----| | step 6,028 | 9.51 | 9.58 | +0.07 | | step 9,042 | 9.56 | 9.92 | +0.36 | | step 15,070 | 8.94 | 15.55 | +6.61 | The pattern is clear: push the fine-tune too far and the inference path breaks. The fix requires re-calibrating the production body against the new checkpoint, not just training a better head. This is the kind of thing that does not show up in a benchmark submission but would absolutely show up in production. The language hint story followed the same pattern. The first attempt at plumbing language hints through the inference path scored 6.91 , marginally worse than no hint at all (6.90). The hint existed in the configuration but was not reaching the decoder. The fix required plumbing JobConfig.language_hints through aed_pipeline.py to the orchestrator and on to the liteasrengine . After that, the gain on AMI was 13.48 to 12.60. Before the fix: nothing. The feature was not a feature until it was actually propagated. What this means for reading anyone's leaderboard score If you are evaluating speech API vendors, this section is the one that matters most. A leaderboard rank is a snapshot of a specific inference configuration on a specific set of testsets. It is not a general statement about model quality. The questions worth asking when you see a leaderboard position are: which testsets does the headline WER come from, and what does the score look like on AMI and Earnings22 specifically? A vendor who scores well on clean testsets but has not published AMI scores is telling you something, even if they do not mean to. The AMI testset is hard to fake sustainably. You can fine-tune on AMI-adjacent data to inflate the leaderboard score, but if the model then breaks when you push the fine-tune further, or if the inference path does not hold up outside evaluation conditions, that shows up in production even if it does not show up in the submission. The leaderboard is a useful signal. It is not a guarantee. We will go deeper on how to run your own evaluation and what questions to ask any speech API vendor in a follow-up post. The full results All scores are %WER under whisper.normalizers.EnglishTextNormalizer across eight testsets. CommonVoice was removed from the headline leaderboard. | Variant | clean | other | vox | ted | gig | spgi | e22 | ami | Unwgh WER | Micro WER | Rank | |----|----|----|----|----|----|----|----|----|----|----|----| | gpu enh baseline (decode-rescore prod) | 2.51 | 4.81 | 6.90 | 3.32 | 9.88 | 3.21 | 12.81 | 13.39 | 7.10 | 5.74 | 27 | | Current public submission (speechmatics/enhanced) | 2.23 | 4.56 | 6.45 | 3.05 | 9.78 | 3.06 | 12.56 | 13.48 | 6.90 | 5.60 | 26 | | Overlap correction + langhint v2 | 2.27 | 4.56 | 6.42 | 3.04 | 9.70 | 3.05 | 12.43 | 12.60 | 6.76 | 5.53 | 25 | | FT ami hf step 6,028, hard splitting | 2.66 | 5.11 | 6.62 | 3.00 | 9.52 | 3.19 | 11.98 | 9.51 | 6.45 | 5.42 | 19 | | FT ami_hf step 6,028, overlap correction + langhint | 2.29 | 4.82 | 6.61 | 3.16 | 9.59 | 3.11 | 11.92 | 9.58 | 6.38 | 5.37 | 16 | Neighbouring positions: #15 is nvidia/parakeet-tdt-0.6b-v3 at 6.32, #17 is kyutai/stt-2.6b-en at 6.40. What did not move the needle Beam size (tested on 3 of 8 testsets): 11.97 at beam=4, 11.96 at beam=8, 11.94 at beam=16. No meaningful improvement past beam=4. EOS bias (tested on 3 of 8 testsets, -2 to +2): 11.51 across the range. No effect. Chunk size: Larger chunks help up to around 15 seconds. The orchestrator sweep from 12,800 to 224,000 tokens produced scores between 6.75 and 6.77. Flat. What is next Three directions: model averaging (SWA across steps 20k–40k) to recover the clean-set regression introduced by AMI specialisation; ROVER across multiple decode passes; and addressing the streaming gap at the model timing head rather than the post-processing path. There are probably more gains available. We will publish when there is something worth publishing. Try it yourself The model behind these results is available through the Speechmatics API , including language hints and domain configurations. Free tier, no credit card required. The leaderboard testsets are public and downloadable. If you want to evaluate how any speech API handles the audio that actually matters in production, AMI and Earnings22 are the ones to start with. The Speechmatics GitHub Academy has runnable examples for both batch and real-time transcription.