# Echo Benchmark Truth Set

Eleven evidence-first 1080 × 1350 benchmark cards for LinkedIn.

## What is directly comparable

Cards 02–04 use the same frozen 100-utterance public-audio manifest, references,
normalization, and scoring:

- Echo Parakeet TDT 0.6B v3 Core ML.
- Whisper large-v3-turbo Q5 through whisper.cpp.
- Echo Qwen3-ASR 0.6B INT8 Core ML.

## What is contextual

- Card 06: Hugging Face seven-domain short-form average.
- Card 07: Hugging Face long-form average.
- Card 08: Artificial Analysis AA-WER v2 cloud API benchmark plus Echo's
  separately labeled public-subset reconstruction.

Echo is never assigned an inferred score. Its 6.19% public-subset result was
measured on all 634 released VoxPopuli and Earnings22 rows, but it is not an
official AA-WER score because the private AgentTalk half and complete current
AA normalizer are unavailable.

## Card order

1. Benchmark methodology.
2. Common-track accuracy.
3. Clean versus difficult speech.
4. Model-only and whole-run speed.
5. Generated dictation, vocabulary, and non-speech.
6. Published short-form model context.
7. Published long-form meeting context.
8. Independent cloud context.
9. Privacy, offline use, fees, and measurement coverage.
10. Meeting intelligence.
11. Current release evidence boundary.

## Publication-safe summary

On Echo's deterministic 100-clip public sample, the shipped Parakeet Core ML
route measured 3.38% mean utterance WER, compared with 3.50% for local Whisper
large-v3-turbo Q5 and 7.10% for the rejected Qwen3-ASR INT8 route. The sample
is not a full leaderboard evaluation. Published Hugging Face and Artificial
Analysis results are shown separately and must not be merged into the direct
Echo ranking.

On the released public half of AA-WER v2, Echo measured 3.25% on VoxPopuli and
9.13% on Earnings22, producing a 6.19% equal-weight public-subset composite.
This measurement is contextual and ineligible for the official AA-WER
leaderboard.

The current release remains unaccepted because the current-build personal
microphone protocol has zero rows and the automatic-delivery evidence has zero
successful traces.

## Source files

- Composition: `src/BenchmarkTruthCards.tsx`
- Detailed claim ledger: `BENCHMARK_TRUTH_CLAIMS.md`
- Common-track protocol: `benchmarks/common-track-v1/README.md`
- AA-WER public-subset reconstruction: `benchmarks/aa-public-v2/README.md`
- Research source audit:
  `research-evidence/echo/asr-benchmark-source-audit-2026-07-29.md`
