# Echo Benchmark Truth Set — claim ledger

Audited 29 July 2026. Echo application source was not modified. The existing
production Core ML benchmark binary was built and run against frozen public
benchmark audio.

## Evidence classes

- **Direct comparison:** identical audio, references, normalization, and scoring.
- **Published context:** trustworthy external benchmark using a different
  protocol. It contextualizes the model family but is not an Echo app score.
- **Product property:** architectural fact such as local execution. It is not
  an accuracy benchmark.
- **Unproven:** no score is inferred or substituted.

## Cards 01–04 — Echo common track v1

Protocol:

- Dataset: `openslr/librispeech_asr`.
- Deterministic rows 0–49 from `clean/test` and rows 0–49 from `other/test`.
- 100 utterances and 585.350063 seconds (9.755834 minutes) of audio.
- Shared case-insensitive Unicode word normalization.
- Mean utterance WER is the arithmetic mean of per-row WER.
- Corpus WER pools substitutions, deletions, insertions, and reference words.

Artifacts:

- `benchmarks/common-track-v1/manifest.json`
- `benchmarks/common-track-v1/sources.json`
- `benchmarks/common-track-v1/results/audio-summary.json`
- `benchmarks/common-track-v1/results/parakeet-v3.scored.json`
- `benchmarks/common-track-v1/results/qwen3-asr-int8.scored.json`
- `benchmarks/common-track-v1/results/whisper-large-v3-turbo-q5.scored.json`

### Accuracy

| Route | Mean utterance WER | Corpus WER | Clean mean | Other mean |
|---|---:|---:|---:|---:|
| Echo Parakeet v3 Core ML | 3.3765% | 2.7672% | 2.8341% | 3.9189% |
| Whisper large-v3-turbo Q5 | 3.5014% | 3.2556% | 3.0012% | 4.0017% |
| Echo Qwen3-ASR 0.6B INT8 | 7.1040% | 5.9685% | 3.1577% | 11.0504% |

Derived comparisons:

- Parakeet mean-WER edge versus Whisper:
  `(3.501443378925107 - 3.376474816301742) / 3.501443378925107 = 3.57%`.
- Qwen-to-Parakeet mean error-rate ratio:
  `7.10402927337116 / 3.376474816301742 = 2.104×`.
- Qwen-to-Parakeet `test-other` ratio:
  `11.050377611347209 / 3.918855717314797 = 2.82×`.

Qualifications:

- This is a deterministic 100-row sample, not the full LibriSpeech split.
- It is not speaker-stratified or randomly sampled.
- Whisper uses `whisper.cpp` and a Q5 model; Echo routes use the current app
  benchmark binary. The system implementations differ intentionally because
  the comparison is of usable local routes, not raw checkpoints.

### Speed

Model-only timing:

- Echo Parakeet: 87.7718 ms mean per row.
- Echo Qwen INT8: 1380.2163 ms mean per row.
- Qwen/Parakeet model-only ratio:
  `1380.2163362503052 / 87.77183890342712 = 15.725×`.
- Whisper model-only timing was not emitted by the runner and is deliberately
  shown as `not captured`.

Whole-run timing:

| Route | Wall seconds | Whole-run realtime multiple |
|---|---:|---:|
| Echo Parakeet | 39.30 | 585.350063 / 39.30 = 14.89× |
| Whisper Q5 | 95.2693 | 585.350063 / 95.2693 = 6.14× |
| Echo Qwen INT8 | 171.94 | 585.350063 / 171.94 = 3.40× |

Whole-run time includes model loading and runner overhead. It is not described
as model-only latency.

## Card 05 — generated dictation and vocabulary

Artifacts:

- `echo-clean/reports/standard-v1-result.json`
- `echo-clean/reports/standard-v1-text-vocabulary-result.json`

Claims:

- Raw mean WER: 4.7364% across 11 generated speech fixtures.
- Text-vocabulary mean WER: 4.0871%.
- Non-speech false positives: 0 across two generated fixtures (silence and low
  background noise).
- The single technical fixture changes from 7.14% raw WER to 0% after the
  deterministic `Qwen` text-vocabulary substitution.

Qualifications:

- Generated macOS TTS regression evidence is not a personal-microphone result.
- One technical fixture does not establish universal technical-vocabulary
  accuracy.
- Deterministic vocabulary is text post-processing. No model-weight
  fine-tuning is claimed.

## Card 06 — Hugging Face short-form context

Source:

- https://huggingface.co/datasets/hf-audio/open-asr-leaderboard-results
- https://github.com/huggingface/open_asr_leaderboard
- https://arxiv.org/abs/2510.06961

Snapshot retrieved 29 July 2026. Selected `avg cleaned` values:

- Modulate VFast: 4.43%.
- Microsoft Azure Speech 06-2026: 4.5129%.
- ElevenLabs Scribe v2: 4.6471%.
- AssemblyAI Universal 3.5 Pro: 5.0286%.
- Qwen3-ASR 0.6B: 5.6114%.
- NVIDIA Parakeet TDT 0.6B v3: 5.6614%.
- Whisper large-v3-turbo: 7.0071%.

The average covers seven current cleaned/public English columns. These scores
do not measure Echo's Core ML conversion, quantization, post-processing, or
delivery path.

## Card 07 — Hugging Face long-form context

Source:

- https://huggingface.co/datasets/hf-audio/leaderboard_longform
- https://huggingface.co/datasets/hf-audio/asr-leaderboard-longform

Current average WER across Earnings21, Earnings22, TED-LIUM, and CORAAL:

- ElevenLabs Scribe v2: 7.315%.
- AssemblyAI Universal 3 Pro: 8.34%.
- Speechmatics Enhanced: 8.80%.
- NVIDIA Parakeet TDT 0.6B v3: 10.72%.
- Whisper large-v3-turbo: 11.01%.
- Google Chirp: 13.045%.

Echo has not run this complete long-form track through the Core ML app route.
The chart is explicitly upstream context.

## Card 08 — independent cloud context

Sources:

- https://artificialanalysis.ai/speech-to-text/non-streaming
- https://artificialanalysis.ai/speech-to-text/methodology

Current AA-WER v2 selected rows:

- ElevenLabs Scribe v2: 2.18%.
- Microsoft MAI-Transcribe-1.5: 2.38%.
- Gemini 3.1 Pro Preview (High): 2.82%.
- AssemblyAI Universal-3.5 Pro: 3.02%.
- OpenAI GPT Transcribe: 3.31%.
- OpenAI GPT-4o Transcribe: 3.96%.

AA-WER v2 uses approximately eight hours weighted 50% AgentTalk, 25%
VoxPopuli-Cleaned-AA, and 25% Earnings22-Cleaned-AA. AgentTalk is private.
Echo therefore cannot receive an independently reproduced official AA-WER
score.

### Echo public-subset reconstruction

Echo's production Parakeet TDT 0.6B v3 Core ML route was run locally against
every released public AA-WER v2 row:

- VoxPopuli-Cleaned-AA v1: 628 samples, 7,122.378 seconds, 3.2465%
  duration-weighted WER.
- Earnings22-Cleaned-AA v1: six full parent calls, 6,895 seconds, 9.1278%
  duration-weighted WER.
- Equal-weight public-subset composite: 6.1871%.
- Total: 634 samples, 14,017.378 seconds (3.8937 hours).
- Whole run: 127.8 seconds on an Apple M2 Pro, or 109.68 audio seconds per wall
  second. This includes model loading and runner overhead.

Scoring uses `whisper-normalizer==0.1.15` and `jiwer==4.0.0`. Artificial
Analysis' complete current custom normalizer is not public. AA-AgentTalk—the
other 50% of the official benchmark—is also not public. The 6.19% result must
therefore be called **AA-WER v2 public-subset reconstruction**, not AA-WER,
and must not be treated as directly rank-comparable with the full cloud scores.

Artifacts:

- `benchmarks/aa-public-v2/manifest.json`
- `benchmarks/aa-public-v2/sources.json`
- `benchmarks/aa-public-v2/echo-coreml-results.json`
- `benchmarks/aa-public-v2/echo-coreml-aa-public-scored.json`
- `benchmarks/aa-public-v2/echo-coreml-aa-public-summary.json`
- `benchmarks/aa-public-v2/validation.json`

## Card 09 — system tradeoffs

- Echo Parakeet and local Whisper run locally and do not require sending audio
  to a transcription API.
- The compared frontier lane is a remote API lane and therefore transmits audio
  to its provider.
- `none` means no per-use transcription API fee. It does not mean hardware,
  electricity, development, or distribution has zero cost.
- Cloud accuracy, wall time, and model time remain `not run` on the frozen Echo
  common track.

## Card 10 — meeting intelligence

Sources:

- `echo-clean/reports/meeting-suite-v1-result.json`
- `echo-clean/Docs/meeting-intelligence-zenith.md`

Claims:

- Five synthetic meeting-domain contracts pass.
- The corrected private financial-call reference passes 9/9 checks.
- Qwen3.5-4B MLX 4-bit baseline is approximately 3.06 GB.
- Warm cached processing is approximately 3–4 seconds.
- Qwen3.5-2B was rejected because grounded structured output was unreliable.

Qualifications:

- The domain count and semantic checks are contract tests, not a general model
  accuracy percentage.
- Meeting-intelligence time is not transcription latency.
- Private reference content is not exposed in the visual.

## Card 11 — release evidence boundary

Authoritative current artifact:

- `echo-clean/reports/release-acceptance.json`
- Modified 29 July 2026 at 13:17 EDT.

Current facts:

- Overall release acceptance: false.
- Speech regression: pass at 4.74% generated WER.
- Non-speech rejection: pass with zero false positives.
- Current-build automatic delivery: fail; successful traces = 0.
- Ten-row current-build personal microphone protocol: fail; rows = 0.
- Current-build delivery latency: fail; records = 0.
- Meeting transcription and meeting-intelligence categories pass their current
  contracts.

The publication must not describe Echo as production-proven personal dictation
until the personal microphone and delivery gates are completed.
