Model scores no longer reflect voice AI capabilities

Outstanding scores are no longer sufficient proof

Public tests for voice AI increasingly suggest that models are reaching human-level capabilities. But these leaderboards sometimes lie. Because public benchmarks like VoxPopuli or LibriSpeech are open, models can optimize directly for them. A new study shows that scores rise due to pattern recognition in the test, not due to genuine improvement in the underlying speech transcription task.

AI prefers repeating someone else's mistake

Researchers tested 11 popular open-source ASR models, focusing on how they react to errors. For example, a recording clearly says "Thank you, Mr. President," but the reference transcript omits "Thank you." Up to six of the tested models obediently reproduced the erroneous transcript without the missing word. The models ignore what is actually in the audio and instead follow the answer the benchmark expects from them.

Models notice cues that humans miss

When the study's authors silenced numbers in the test recordings, something surprising emerged. On the LibriSpeech test, some top models correctly filled in the missing numbers in 30 to 40% of cases, even though they were completely absent from the audio. The models rely on surrounding acoustic cues and sentence context to guess or recall the reference text they optimized on.

Separating training and test data makes cheating harder

When researchers tested the same models on freshly collected data with new voices recorded after their training cutoff, the benchmark optimization vanished entirely. Models stopped reproducing old errors and went back to faithfully transcribing the actual audio. This shows that testing on new, completely held-out data sets is becoming crucial for model selection, rather than relying on datasets the community has circulated for years.

Lilith's verdict

When a metric is public, it becomes the target, and models will find a way to hack it. Transcribing silence from a cheat sheet is not intelligence; it is excellent optimization for the wrong thing.