# Echo AA-WER v2 public-subset reconstruction

Run date: 29 July 2026  
Status: measured and validated; not eligible as an official AA-WER score

## Result

Echo's production Parakeet TDT 0.6B v3 Core ML route measured:

| Public component | Rows | Duration | Duration-weighted WER |
|---|---:|---:|---:|
| VoxPopuli-Cleaned-AA v1 | 628 | 7,122.378 s | 3.2465% |
| Earnings22-Cleaned-AA v1 | 6 full calls | 6,895.000 s | 9.1278% |
| Equal-weight public composite | 634 | 14,017.378 s | 6.1871% |

The complete 3.8937 hours ran in 127.80 wall seconds on an Apple M2 Pro,
including model loading and runner overhead.

## Why this is not an official AA-WER result

Artificial Analysis defines AA-WER v2 as 50% proprietary AA-AgentTalk, 25%
VoxPopuli-Cleaned-AA, and 25% Earnings22-Cleaned-AA. AA-AgentTalk contains 469
held-out samples and is not publicly available. Artificial Analysis also
describes a custom normalizer built on the Whisper English normalizer, but its
complete current implementation is not published.

The Echo reconstruction therefore:

- includes every released row from the two public components;
- reweights the two public components equally;
- uses `whisper-normalizer==0.1.15` and `jiwer==4.0.0`;
- reports `6.19% public-subset WER`, never `6.19% AA-WER`;
- does not claim an Artificial Analysis leaderboard rank.

## Sources

- Artificial Analysis methodology:
  https://artificialanalysis.ai/speech-to-text/methodology
- AA-WER v2 article:
  https://artificialanalysis.ai/articles/aa-wer-v2
- VoxPopuli-Cleaned-AA:
  https://huggingface.co/datasets/ArtificialAnalysis/VoxPopuli-Cleaned-AA
- Earnings22-Cleaned-AA:
  https://huggingface.co/datasets/ArtificialAnalysis/Earnings22-Cleaned-AA

Pinned dataset revisions:

- VoxPopuli-Cleaned-AA:
  `07adf4a242e2e5dd955a230909c177573c978ee3`
- Earnings22-Cleaned-AA:
  `5999936ee3d8d3cde160e3c65a8cc6ff5b7c5276`

## Local evidence

Root:

`benchmark-workspace/benchmarks/aa-public-v2`

Key artifacts:

- `manifest.json`: 634-item Echo suite manifest.
- `sources.json`: source revision, duration, reference, local path, byte count,
  and SHA-256 for every audio file.
- `echo-coreml-results.json`: raw production-run transcripts and timings.
- `echo-coreml-aa-public-scored.json`: normalized strings, edit counts, row WER,
  component aggregation, and composite.
- `echo-coreml-aa-public-summary.json`: compact result.
- `validation.json`: passed evidence-integrity assertions.
- `fetch_aa_public.py`, `score_aa_public.py`, and
  `validate_aa_public.py`: reproducible workflow.

Raw result SHA-256:

`cd8b62a161a41ab930d07e63b1f87449cde26713bca27c59e96328a1f9d0cc12`

## Validation

The validation pass confirmed:

- 634 source, manifest, result, and scored IDs match in order;
- 628 VoxPopuli rows and six Earnings22 parent calls;
- 634 unique IDs and 634 non-empty raw transcripts;
- production `multilingualV3` and `raw-asr` pipeline;
- every local audio SHA-256 matches its frozen source record;
- raw result SHA-256 matches the scored report;
- `official_aa_wer_eligible` is explicitly false.

## Publication language

Safe:

> On all 634 released public AA-WER v2 samples, Echo's local Parakeet Core ML
> route measured 6.19% public-subset WER: 3.25% on VoxPopuli and 9.13% on
> Earnings22.

Unsafe:

> Echo scores 6.19% AA-WER.

The unsafe version falsely implies that the private AgentTalk half and complete
official scoring pipeline were run.
