# AA-WER v2 public-subset reconstruction for Echo

This protocol runs Echo's production Parakeet TDT 0.6B v3 Core ML route against
the two public datasets in AA-WER v2:

- all 628 VoxPopuli-Cleaned-AA v1 samples;
- all 6 Earnings22-Cleaned-AA v1 parent recordings.

It cannot produce an official AA-WER score. AA-AgentTalk is proprietary and
accounts for 50% of the official score. Artificial Analysis' complete current
custom normalizer is also not published.

The result is therefore reported as **AA-WER v2 public-subset reconstruction**.
It uses duration-weighted WER within each dataset and an equal-weight mean of
the two public dataset scores. Scoring uses the open-source Whisper English
normalizer through `whisper-normalizer==0.1.15`.

## Reproduce

```sh
python3 fetch_aa_public.py

cd echo-clean
swift build --configuration release --product echo-benchmark --disable-index-store
.build/release/echo-benchmark \
  --suite /absolute/path/to/aa-public-v2/manifest.json \
  --engine v3 \
  --output /absolute/path/to/aa-public-v2/echo-coreml-results.json

python3 -m venv .venv
.venv/bin/pip install jiwer==4.0.0 whisper-normalizer==0.1.15
.venv/bin/python score_aa_public.py echo-coreml-results.json
```

Dataset revisions, per-file SHA-256 hashes, references, durations, raw model
transcripts, normalized strings, edit counts, WER values, and timings are saved
with the result.
