# Echo common-track benchmark v1

This benchmark replays one immutable public audio sample through every eligible
ASR route.

## Protocol

- Dataset: `openslr/librispeech_asr` through the Hugging Face Dataset Viewer.
- Splits: `clean/test` and `other/test`.
- Sampling: deterministic rows 0–49 from each split, for 100 utterances total.
  This is deliberately simple and reproducible; it is not a random or
  speaker-stratified sample.
- Accuracy: both mean utterance WER and corpus WER after the same
  case-insensitive Unicode word normalization.
- Timing: report model timing and whole-run wall timing separately.
- Privacy: only public benchmark audio can be sent to external APIs.
- Promotion: sampled results are stronger than a 20-row smoke test but are not
  full-split leaderboard results.

## Planned lanes

1. Echo production route — Parakeet TDT 0.6B v3 Core ML.
2. Echo challenger — Qwen3-ASR 0.6B INT8 Core ML.
3. Open-weight benchmark — Whisper large-v3-turbo Q5 through whisper.cpp.
4. Frontier API lane — only providers with configured credentials and a native
   speech-to-text endpoint.

Anthropic is not an ASR lane because it does not currently offer a native
speech-transcription model or audio-input API.
