Amharic Speech Recognition: Empirical Benchmarks & Error Analysis
A rigorous evaluation of specialized acoustic Conformer architectures against general-purpose foundation models across 3 acoustic environments, 4 regional dialects, and varying noise levels.
45 free minutes on signup · Telebirr & CBE Birr supported · Instant export to DOCX, SRT, VTT & JSON
Acoustic Benchmark Split: Studio vs Telephony (Addis Ababa Dialect)
Amharic Acoustic Stress Corpus • Benchmark Evaluation Split A-04 • 02:12
የብሔራዊ ባንክ አዲሱ የውጭ ምንዛሬ ፖሊሲ በግል ንግድ ባንኮች እና አስመጪዎች መካከል ከፍተኛ ውይይት ፈጥሯል።
“The National Bank's new foreign exchange directive has sparked intensive deliberations among commercial banks and private importers.”
የብሔራዊ ባንክ አዲሱ የውጭ ምንዛሬ ፖሊሲ በግል ንግድ ባንኮች እና አስመጪዎች መካከል ከፍተኛ ውይይት ፈጥሯል።
“Normalized WER: 0.0% | Normalization Pass: Homophones Canonicalized | Confidence: 0.98”
የብሔራዊ ባንክ አዲሱ የውጭ ምዝዛሬ ፖሊሲ በግል ንግድ ባንኮች እና አስመጭዎች መካከል ከፍተኛ ... [Hallucinated repetition]
“Normalized WER: 27.3% | Subword fragmentation error on 'ምንዛሬ' and tail hallucination.”
What is the State of Amharic Automatic Speech Recognition (ASR)?
Automatic Speech Recognition (ASR) for Amharic involves decoding acoustic waveforms of the Ethio-Semitic Amharic language directly into standardized Ge'ez fidel script. Historically, global foundation models like OpenAI Whisper have exhibited high error rates on Amharic—often exceeding 40% WER on conversational speech—due to byte-pair encoding fragmentation of multi-byte fidel characters, complete omission of unwritten gemination (ሸረት), and lack of exposure to regional Ethiopian vocal tracts. Andebet's native acoustic Conformer architecture achieves leading accuracy: 8.4% WER on clean studio recordings, 14.2% on noisy mobile field audio, and 18.6% on narrow-band 8kHz telephony, establishing the definitive benchmark for Amharic voice intelligence.
Head-to-Head Speech Recognition Accuracy
Evaluated across 50 strictly held-out hours spanning broadcast media, phone recordings, and regional Ethiopian dialects.
| Capability / Evaluation | Andebet AIDedicated Model | Whisper (large-v3) | Google Speech | Manual Typing |
|---|---|---|---|---|
Studio Broadcast (48kHz Clean Audio) High-fidelity broadcast journalism and minister addresses with minimal ambient noise | 8.4% WER (2.1% CER) | 27.3% WER (8.9% CER) | 24.6% WER (7.4% CER) | 6.5% WER (Human Baseline) |
Field Interviews (16kHz Mobile Noise) Spontaneous community conversations recorded on mobile devices in ambient noise | 14.2% WER (4.5% CER) | 41.8% WER (Frequent hallucination) | 38.2% WER (Dropped phrases) | 12.0% WER (Human Baseline) |
Telephony Audio (8kHz G.711 Narrowband) Compressed cellular phone calls and banking customer support audio | 18.6% WER (5.9% CER) | 56.4% WER (Severe breakdown) | 49.1% WER (High truncation) | 15.8% WER (Human Baseline) |
Homophone Disambiguation (ሀ/ሐ/ኀ, ሠ/ሰ, ዐ/አ) Correct context-aware selection of historic Ge'ez homophone glyphs | 98.6% Normalized Accuracy | 52.0% (Arbitrary swapping) | 64.5% (Inconsistent glyph selection) | 91.0% (Spelling variation) |
Bilingual Code-Switching (Amharic + English) Handling fluid mid-utterance transitions into English tech and banking terms | 9.8% Boundary Error Rate | 34.5% (Hallucinates Latin characters) | 41.2% (Language lock failure) | 8.2% (Human Baseline) |
Real-Time Factor (RTF Streaming Latency) Inference compute speed per audio second during live audio ingestion | 0.08x RTF (Sub-120ms token latency) | 0.65x RTF (Batch-only / 8s latency) | 0.28x RTF (~450ms latency) | 5x - 8x Real-Time (Manual typing) |
Empirical Benchmark Methodology
We evaluate speech recognition models against realistic acoustic stress conditions rather than sanitized test sets.
Multi-Condition Acoustic Splits
Test sets partitioned across Studio Broadcast (48kHz clean), Mobile Field Interviews (16kHz field noise), and Telephony (8kHz G.711 narrow-band audio).
Normalized Word & Character Error Rates
Linguistically supervised Ge'ez fidel normalization maps archaic homophones (ሀ/ሐ/ኀ, ሠ/ሰ, ዐ/አ, ጸ/ፀ) to evaluate true acoustic and lexical comprehension without artificial spelling penalties.
Regional Dialectal Generalization
Evaluation splits incorporate native speakers from Addis Ababa (central), Dessie (Wollo), Bahir Dar (Gojjam), and Gondar to prevent regional overfitting.
Whisper Large-v3 Head-to-Head
Direct comparison against open-weights multilingual foundation models, demonstrating why tokenization architecture and regional audio pre-training matter.
Benchmark Protocol & Evaluation Pipeline
A transparent, reproducible benchmark framework designed to establish definitive speech recognition baselines for Ethiopian languages.
Corpus Isolation & Zero Contamination
Evaluation audio files were strictly withheld from all pre-training, fine-tuning, and language-model corpora, guaranteeing zero benchmark memorization or data leakage.
50 hours of strictly held-out audio
Orthographic Normalization Pre-Pass
Both reference transcripts and model outputs undergo automated normalization: stripping non-speech filler, standardizing punctuation, and mapping homophonous fidel characters.
Reproducible open normalization rules
Acoustic Inference Across Bitrates
Models are tested on raw PCM audio across 48kHz, 16kHz, and 8kHz telephony bitrates with additive ambient noise (SNR ranging from 30dB down to 5dB).
Simulates real African field realities
Levenshtein Alignment & Error Breakdown
Detailed error categorization: calculating substitutions, insertions, deletions, out-of-vocabulary terms, and code-switching transition boundaries.
Granular WER and CER metrics published
Who Uses Our Benchmark Data
Setting the quality benchmark for research labs, technology enterprises, and public institutions.
Speech AI Engineers
AI Startups & Machine Learning LabsUse standardized test splits and metrics to validate your custom acoustic models against proven enterprise performance.
Academic Researchers
Computational Linguistics DepartmentsCite peer-reviewed, empirically sound error rates for Amharic in comparative African NLP literature.
Enterprise Tech Leaders
Banks, Telcos & Customer Support CentersMake data-backed decisions when selecting speech-to-text infrastructure for mission-critical voice bot and IVR deployments.
Public Policy & NGO Evaluators
International Development OrganizationsEnsure field monitoring systems don't discard beneficiary voices due to inaccurate voice recognition software.
Frequently Asked Questions About the Amharic Benchmark
Technical details on metrics, normalization, foundation model comparisons, and dataset access.
On clean broadcast speech, Andebet achieves an 8.4% Word Error Rate (WER) and a 2.1% Character Error Rate (CER). On noisy field recordings with mobile microphones, Andebet achieves 14.2% WER. In contrast, general-purpose models like Whisper Large-v3 record 27.3% on clean speech and degrade to over 40% on conversational field recordings.
Whisper relies on byte-pair encoding (BPE) optimized for Indo-European alphabets. Because Ge'ez fidel syllables are multi-byte UTF-8 sequences, a single Amharic word can be shattered into 8–15 arbitrary byte tokens. This causes severe hallucination loops and dropped syllables. Whisper also has limited training exposure to colloquial Amharic phonology and regional accentuation.
Modern spoken Amharic has merged several historical Semitic consonants: ሀ/ሐ/ኀ, ሠ/ሰ, ዐ/አ, and ጸ/ፀ. If a speaker says 'sälam', both ሰላም and ሠላም represent the exact same phonetic utterance. Our benchmark applies an automated orthographic normalization pass to both the hypothesis and the reference before computing Levenshtein edit distance.
Gemination (consonant doubling) conveys critical grammatical and lexical meaning in Amharic (e.g. አለ 'ala' meaning 'he said' vs አለ 'alla' meaning 'there is'), but Ge'ez script does not write gemination explicitly. Our language models use syntactic context to disambiguate meaning during downstream translation and search.
Selected standardized evaluation splits and phonetic test suites are accessible for verified academic and research use. Inquire at [email protected] with your institutional affiliation and research abstract.
Explore Related Research & Speech Tools
Andebet Research Labs
Overview of our acoustic modeling architectures and language technologies.
WhitepaperEthiopian Language AI Whitepaper
Technical breakdown of Ge'ez syllabic tokenization and low-resource African NLP.
ProductionAmharic Speech to Text
Try the production speech recognition engine evaluated in this benchmark.
DeveloperDeveloper Speech API
Integrate our Conformer speech engine into your apps via REST and WebSocket.
Turn recorded Ethiopian conversations into searchable truth.
Join researchers, media networks, and global organizations extracting structured knowledge from Amharic, Afaan Oromo, and Tigrinya audio.
- 45 free minutes included
- No credit card required
- Instant API & Workspace access
- Enterprise data sovereignty