Conversational Finesse: Recovering Rare Entities in Voice AI

THE PROBLEM

‍

‍The Small Mishearings That Carry Enormous Costs Automatic Speech Recognition (ASR) engines have achieved impressive general word error rates, but they frequently degrade on domain-specific entity phrases like brand names, medical terms, callsigns, and financial tickers. In high-stakes fields like finance, healthcare, and air traffic control, a small near-miss—turning "Cytiva" into "citeva" or "Linezolid" into "linear zolid"—can be extremely costly. When an entity is deleted or severely corrupted in the initial ASR output, standard text-only post-correctors have zero acoustic signal to recover it.
‍

THE GAP

‍

‍Why Single-Hypothesis Post-Correction Falls Short Most post-ASR correction pipelines operate strictly on a single 1-best transcript. When a rare entity is omitted or hallucinated away at decode time, an LLM post-corrector cannot infer what was originally spoken without blindly guessing, leading to over-correction or hallucinated edits. While white-box decode-time contextual biasing helps, it requires deep access to the recognizer's internal decoder, making it unusable with commercial black-box ASR services.

‍

A single 1-best transcript that drops a rare entity leaves text-only LLMs blind. True recovery requires tapping into alternative ASR decoding hypotheses.

‍

OUR APPROACH RECOVER:

‍

Agentic Multi-Hypothesis Entity Correction RECOVER addresses this fundamental evidence gap by extracting multiple hypotheses from a black-box ASR via temperature sampling, surfacing complementary errors where an entity missed in one hypothesis survives in another. An agentic controller orchestrates a suite of specialized tools to retrieve, fuse, propose, and verify entity corrections safely:
‍

• Multi-Signal Retrieval (RETRIEVE): Scores entity lists using exact token matches, fuzzy Levenshtein similarity, and phonetic key prefixes to pull the top-$K$ candidates.
  ‍
• Hypothesis Fusion (FUSE): Evaluates variants via LLM-Select, ROVER ensemble voting, or entity-aware selection to build a unified base transcript.
  ‍
• Constrained Proposals & Guardrails (PROPOSE & VERIFY): Enforces strict closed-vocabulary rules where proposed replacements must match verified entity list phrases, filtered by deterministic similarity floors.
  ‍
• LLM Critic & Targeted Feedback: An LLM judge inspects surviving edits against all original audio hypotheses. If an actionable entity was missed, it triggers a targeted feedback round ($R=1$) to re-retrieve candidates and re-propose corrections.
  ‍

THE NUMBERS

‍

‍Production Performance Across High-Stakes Domains Evaluated across five benchmarks (Earnings-21, ATCO2, Eka-Medical, Common Voice, and ContextASR-Bench) on top of a Whisper-large-v3 baseline, RECOVER delivers substantial entity recovery:

‍

THE DIFFERENCE
‍

What Changes When Post-Correction Has Multi-Hypothesis Evidence
‍

‍

By combining multi-hypothesis sampling, constrained retrieval, deterministic safety guardrails, and an agentic LLM critic, RECOVER bridges the gap between black-box speech recognition and enterprise precision. Rather than forcing language models to blindly guess missing terms, RECOVER equips them with cross-hypothesis evidence—delivering the domain accuracy and conversational finesse required for mission-critical Voice AI.