# Echo ASR benchmark source audit

Retrieved: 2026-07-29

## Research question

Which existing speech-to-text benchmarks are credible enough to contextualize Echo before paying to run additional frontier APIs?

## Bottom line

Use three separate evidence lanes:

1. Hugging Face Open ASR Leaderboard for reproducible public short-form English and model efficiency.
2. Hugging Face Open ASR long-form track for meetings, earnings calls, talks, and interviews.
3. Artificial Analysis AA-WER v2 for current closed API models tested independently.

Do not merge scores across these lanes. They use different audio, normalization, weighting, serving paths, and hardware.

## Evidence quality

### Tier A — strongest for publication

- Hugging Face Open ASR Leaderboard paper, code, datasets, predictions, and current result datasets.
- Artificial Analysis AA-WER v2 methodology and current independently measured results.
- Echo's own same-audio, versioned artifacts when the complete manifest, hypotheses, normalization, timing boundary, hardware, and hashes are retained.

### Tier B — useful corroboration

- Peer-reviewed or academic task-specific comparisons such as the ICML 2025 WhisperKit streaming study.
- New held-out leaderboards such as FFASR for far-field conditions.

### Tier C — context only

- Vendor-authored benchmark charts from OpenAI, ElevenLabs, Deepgram, AssemblyAI, Google, and others.
- Model-card scores without independently retained predictions.

## 1. Current Hugging Face short-form English results

Source dataset: https://huggingface.co/datasets/hf-audio/open-asr-leaderboard-results

Method/code: https://github.com/huggingface/open_asr_leaderboard

Paper: https://arxiv.org/abs/2510.06961

The current result dataset averages normalized WER across seven cleaned/public English columns: AMI, Earnings22, GigaSpeech, LibriSpeech test-clean, LibriSpeech test-other, SPGISpeech, and VoxPopuli. Lower WER is better. Open-model RTFx is measured on the leaderboard's standardized server hardware and is not comparable to Echo's Core ML timing.

Selected current rows:

| Model | Type | Average cleaned WER | RTFx |
|---|---:|---:|---:|
| Modulate VFast | Proprietary | 4.43% | not reported |
| Microsoft Azure Speech (06-2026) | Proprietary | 4.51% | not reported |
| ElevenLabs Scribe v2 | Proprietary | 4.65% | not reported |
| OpenMOSS MOSS-Transcribe preview 2B | Open | 4.66% | 148.51x |
| Qwen3-ASR 1.7B HF | Open | 4.99% | 796.19x |
| AssemblyAI Universal 3.5 Pro | Proprietary | 5.03% | not reported |
| Qwen3-ASR 0.6B | Open | 5.61% | 438.74x |
| NVIDIA Parakeet TDT 0.6B v3 | Open | 5.66% | 6098.20x |
| Whisper large v3 | Open | 6.55% | 462.22x |
| Whisper large v3 turbo | Open | 7.01% | 782.56x |

Interpretation:

- Parakeet v3 is exceptionally strong on the accuracy/throughput tradeoff, not the absolute lowest WER.
- Qwen3-ASR 0.6B is close to Parakeet in the upstream leaderboard, but that does not contradict Echo's weaker INT8 Core ML challenger result. Conversion, quantization, decoder behavior, and application post-processing are part of the shipped system.
- The leaderboard's full multi-domain average cannot be compared directly with Echo's 100-utterance LibriSpeech sample or Echo's generated dictation protocol.

## 2. Current Hugging Face long-form results

Source dataset: https://huggingface.co/datasets/hf-audio/leaderboard_longform

Underlying audio track: https://huggingface.co/datasets/hf-audio/asr-leaderboard-longform

The average covers Earnings21, Earnings22, TED-LIUM, and CORAAL. Lower WER is better.

| Model | Type | Long-form average WER | RTFx |
|---|---:|---:|---:|
| ElevenLabs Scribe v2 | Proprietary | 7.32% | not reported |
| AssemblyAI Universal 3 Pro | Proprietary | 8.34% | not reported |
| Speechmatics Enhanced | Proprietary | 8.80% | not reported |
| Cohere Transcribe 03-2026 | Open weights | 9.73% | 417.63x |
| NVIDIA Parakeet TDT 0.6B v3 | Open | 10.72% | 1002.91x |
| Whisper large v3 turbo | Open | 11.01% | 148.26x |
| Whisper large v3 | Open | 11.23% | 68.56x |
| Google Chirp | Proprietary | 13.05% | not reported |

Interpretation:

- This lane is the best published match for Echo's meeting-transcription story.
- It measures transcription only. It does not test Echo's Qwen3.5 meeting summarization, evidence validation, action items, or grounded structured output.

## 3. Artificial Analysis AA-WER v2

Leaderboard: https://artificialanalysis.ai/speech-to-text/non-streaming

Methodology: https://artificialanalysis.ai/speech-to-text/methodology

AA-WER v2 directly calls model APIs. It uses roughly eight hours of audio: a private held-out AgentTalk set at 50% weight, cleaned VoxPopuli at 25%, and cleaned Earnings22 at 25%. It applies a documented normalizer and includes network time in batch speed.

Selected current overall AA-WER rows:

| Model | AA-WER v2 |
|---|---:|
| ElevenLabs Scribe v2 | 2.18% |
| Microsoft MAI-Transcribe-1.5 | 2.38% |
| Mistral Voxtral Small | 2.77% |
| Gemini 3.1 Pro Preview (High) | 2.82% |
| AssemblyAI Universal-3.5 Pro | 3.02% |
| OpenAI GPT Transcribe | 3.31% |
| Qwen3.5 Omni Plus | 3.55% |
| OpenAI GPT-4o Transcribe | 3.96% |
| Speechmatics Enhanced | 4.05% |
| Whisper large v3 | 4.07% |

Selected AgentTalk WER:

| Model | AgentTalk WER |
|---|---:|
| ElevenLabs Scribe v2 | 1.51% |
| AssemblyAI Universal-3.5 Pro | 1.94% |
| OpenAI GPT Transcribe | 2.22% |
| Google Chirp 3 | 2.49% |
| OpenAI GPT-4o Transcribe | 2.68% |

Selected cleaned VoxPopuli WER:

| Model | Cleaned VoxPopuli WER |
|---|---:|
| ElevenLabs Scribe v2 | 1.74% |
| OpenAI GPT-4o Mini Transcribe | 2.40% |
| Whisper large v3 | 2.41% |
| OpenAI GPT-4o Transcribe | 2.76% |
| Speechmatics Enhanced | 2.86% |
| Deepgram Nova-3 | 2.95% |
| Google Chirp 3 | 3.66% |
| Qwen3 ASR Flash | 4.06% |

Important:

- "GPT Transcribe" and "GPT-4o Transcribe" are distinct products in the current table.
- Qwen3 ASR Flash is not Echo's local Qwen3-ASR 0.6B INT8 build.
- Artificial Analysis does not publish the private AgentTalk audio, so its exact end-to-end run cannot be independently reproduced.

## 4. Academic streaming comparison

Paper: https://arxiv.org/abs/2507.10860

Venue: ICML 2025.

The WhisperKit study compares WhisperKit, Deepgram Nova-3, Fireworks Whisper large-v3-turbo, and OpenAI GPT-4o Transcribe in a streaming setup. On its TIMIT-based evaluation, the paper reports WhisperKit and Deepgram at about 2% confirmed WER, Fireworks at 4.72%, and OpenAI as the highest-WER system. It also reports approximately 0.45 seconds mean hypothesis latency for WhisperKit/Fireworks and 0.83 seconds for Deepgram.

This is useful corroboration, not a universal ranking. The authors build WhisperKit, the task is streaming, and the audio/protocol differs from Echo's benchmarks.

## 5. Far-field benchmark to watch

FFASR announcement: https://huggingface.co/blog/ffasr-leaderboard

FFASR uses held-out far-field audio, simulated rooms validated against real measurements, and standardized NVIDIA L4 hardware. It reports WER and RTFx. This is more relevant to room microphones than LibriSpeech, but it is not a replacement for a real personal-microphone release protocol.

## What is already covered without buying API runs

- Parakeet, Whisper, Qwen3-ASR, Canary, Moonshine, and many other open models: Hugging Face short-form.
- ElevenLabs, AssemblyAI, Microsoft Azure, Speechmatics, AquaVoice, and other proprietary systems: Hugging Face short-form.
- ElevenLabs, AssemblyAI, Speechmatics, Google Chirp, Parakeet, and Whisper: Hugging Face long-form.
- OpenAI transcription models, Gemini, ElevenLabs, AssemblyAI, Google Chirp, Deepgram, Speechmatics, Mistral, and others: Artificial Analysis.
- OpenAI GPT-4o Transcribe and Deepgram Nova-3 in streaming: WhisperKit/ICML.

Anthropic has no native speech-to-text model and therefore has no legitimate ASR WER lane.

## Publication guardrails

- Do not put Echo's 3.38% mean utterance WER on the same unlabeled axis as the Hugging Face 5.66% seven-dataset average.
- Do not describe Parakeet v3 as the most accurate ASR model overall. Describe it as a strong accuracy/efficiency production choice.
- Do not use Hugging Face H200 RTFx as Echo's Mac speed.
- Do not use hosted Parakeet results as proof of Echo's Core ML conversion quality.
- Do not describe vendor-reported results as independent.
- Do not compare API network latency with Echo's model-only latency.
- Do not treat WER as proof of diarization, punctuation, named-entity accuracy, command execution, meeting intelligence, or hallucination resistance.
- Do not claim frontier parity until Echo and the named frontier endpoint are run on the exact same audio, normalizer, and scoring script.

## Recommended benchmark visuals

1. **Public short-form accuracy:** selected Hugging Face normalized WER rows.
2. **Accuracy versus throughput:** Parakeet, Qwen3-ASR, Whisper, and two leading open models on the Hugging Face RTFx/WER plane.
3. **Long-form meetings:** Hugging Face long-form WER for Scribe, AssemblyAI, Speechmatics, Parakeet, Whisper, and Chirp.
4. **Independent cloud context:** Artificial Analysis AA-WER v2, clearly labeled as a different protocol.
5. **Echo same-track evidence:** Echo Parakeet, Echo Qwen INT8, and local Whisper on the existing 100-clip manifest, with sample size and audio duration displayed.
6. **Evidence boundary:** a final card stating what is public-standard, independently tested, Echo-measured, and not yet proven.

## Recommended next experiment

Do not run every API. First make the visuals from existing published evidence. If an exact frontier comparison is still necessary, run only:

- OpenAI GPT Transcribe or GPT-4o Transcribe;
- ElevenLabs Scribe v2;
- optionally Deepgram Nova-3 for a latency-oriented cloud baseline.

Run them on the identical frozen Echo manifest and retain raw hypotheses, endpoint/model version, request parameters, timestamps, normalization code, failures, cost, and wall-clock timing. This will answer a different question from the public leaderboards: how the shipped Echo build compares on Echo's exact test material.
