Realtime Voice AI Contact Center

ARIA, MII's voice AI contact center: ~2 s to ~0.2 s interruption latency in a controlled A/B (n=4 calls per arm), grounded answers and human handover.

~2 s to ~0.2 s
Knowledge-lookup interruption latency, controlled A/B (n=4 calls per arm)
2 of 2
Node-loss drills passed; live calls survived
6 vs 18
Hallucinated answers in an offline model comparison; the quality gate retained the stronger realtime model

Role: Owner and lead engineer, responsible for design, AI coding agent direction, review and release
Period: May 2026 to Oct 2026
Project type: Key project
Room in the 3D HQ: L3:voice-ai-contact-center

Situation

Callers were stuck in rigid phone menus. MII needed a voice agent that answers from the company knowledge base, lets callers interrupt while it speaks or searches, and hands over to a human and back within the same call, with a live console for supervisors.

Role and key decisions

Moved knowledge lookups out of the realtime loop
Why: A caller interrupting mid-search had to be heard at once, so lookups became cancellable tasks sharing one time budget, and barge-in accounts for audio still playing.
Replaced a retiring voice service with self-hosted WebRTC on zonal VMs
Why: The managed service was being retired; I documented the decision, then proved the new path with hard VM-stop failover drills.
Built an evaluation harness before shipping changes
Why: A media emulator, golden questions and a calibrated LLM judge meant any change without supporting evidence stayed disabled.

Results

~2 s to ~0.2 s
Knowledge-lookup interruption latency, controlled A/B (n=4 calls per arm)
Controlled A/B on the same deployment, index and audio, n=4 calls per arm: 1,875 to 2,140 ms vs 93 to 220 ms.
2 of 2
Node-loss drills passed; live calls survived
Hard VM-stop failover and caller-reconnect drills after diagnosing and fixing four root causes.
6 vs 18
Hallucinated answers in an offline model comparison; the quality gate retained the stronger realtime model
Evaluated offline with a calibrated LLM judge; mean score 4.71 vs 4.28.
Migrated
Voice channel migrated from a retiring service; old path removed
Live cut-over completed well before the announced 2028 retirement, removing about 20.5k lines from the old path.

Lesson

A written spec and a merged PR are not proof that a fix helped. Set the verdict rules before testing, accept the negative results, and remove what the data rejected.

Architecture

Engineering detail

1
Latency Budget
Latency is measured where the caller hears it: a harness plays recorded caller audio over the real call path and joins its timeline with per-turn backend telemetry, so every stage has its own number.
- On the live WebRTC path, the caller hears an answer 1,204 ms after speaking at P50 (P95 1,257 ms, n=6 turns without a tool call). The largest stages are the model's end-of-turn wait (about 520 ms) and its time to first audio (408 ms); the media path adds about 136 ms.
- Knowledge lookups run as cancellable tasks under one 2.5 s budget. Search and embeddings each get a 1.5 s timeout per attempt, and the remaining deadline travels with every retry, so no retry starts that cannot finish in time.
- Moving knowledge search into the backend's geography cut retrieval P50 by 1,351 ms in a live A/B, so the move was kept.
- Tuning that saved too little was rejected against thresholds set before the run: minimal model reasoning saved 39 ms and a 500 ms server-side silence window saved 69 ms across 84 calls. An end-of-turn model that added 220 ms stays off.
Known limits: The samples are small and come from test runs, not production traffic, and the largest stage, the end-of-turn wait, happens inside the model service.
In depth: How We Made a Voice Agent Interruptible
2
Failover Design
Every dependency on the call path has a per-attempt timeout, retries for idempotent calls only, and a circuit breaker that opens after 5 failures in 60 s and probes again after 30 s. Breaker state shows on the readiness endpoint.
- Media runs on two self-hosted LiveKit nodes in separate availability zones. New calls go to the healthy, non-draining node with the fewest rooms, checked every 5 s.
- Call bridges run in supervised worker processes with a 2 s heartbeat. When a worker dies, its calls rejoin a live worker, and a caller who is still present is handed to a human rather than hung up.
- A chaos suite on a fake model passed 16 of 17 failure modes in 3 consecutive runs (region outage was not applicable): failed database writes go to a replayable dead-letter queue, search and embedding failures fall back to keyword search, and an admin API outage starts calls from the last good agent config.
- Admission control refuses a new call with 429 when capacity is full or the model is throttled, and 503 when a dependency's breaker is open, so calls already in progress keep their capacity.
- A live handover drill of 44 calls traced failed takeovers to model throttling and a failed response the bridge dropped silently. Failed responses are now logged and retried.
Known limits: The backend still runs as one instance that is not zone-redundant, there is no fallback model deployment or second region yet, and regional disaster recovery is designed but has never been drilled.
3
Evaluation Method
Decision rules are written and committed before any measurement, and only a pass changes a default. A change without supporting evidence stays off.
- The harness drives the real call bridge three ways: a media emulator for deterministic runs, a fake realtime endpoint for no-cost runs and a headless WebRTC caller for live runs.
- Scenarios cover quiet turns, knowledge questions, interruptions, escalation, long pauses, Indonesian callers, street noise at 15 dB and office noise with speakerphone echo.
- Answer quality uses 16 golden questions in English and Indonesian, 2 runs per model, scored 1 to 5 on groundedness, correctness, completeness, language match and brevity by an LLM judge. The judge first had to agree with 10 hand-labelled items: 0.8 per item and 1.0 on hallucination flags.
- Turn-taking is scored on false end-of-turn rate, cut-off rate and spurious events per 100 utterances, next to latency, against limits set before each run.
- A retrieval regression gate checks recall at k=4 on a fixed question set (0.909 in English, n=44), and its guard flagged all 4 poisoned chunks that the injection probes retrieved.
Known limits: Evaluation runs on demand, not on every pull request, the samples are small, there is no word error rate or real-traffic containment metric yet, and Indonesian retrieval recall on the regression set is still low (0.182, n=11).
4
Security and Identity
Each app runs on its own managed identity with least-privilege data-plane roles, and key-based access is disabled on the database, search, AI services and storage, so no data-store keys sit in app config.
- Service-to-service calls use Entra app roles plus a signed actor context, with no shared keys between the apps.
- Operators sign in with Entra single sign-on, access control fails closed and maps Entra groups to roles, and the first admin is bound only for the right tenant, a member account and a verified email.
- Public callers are anonymous, so each call gets a signed, scoped token that lasts at most 45 minutes. Media join tokens are scoped per role, webhooks are signed with a replay guard, and per-install secrets are generated inside the install and kept in Key Vault.
- Every ingested document chunk is scanned for prompt injection, and quarantined chunks are filtered out at query time, which adds no call latency.
- No call audio is recorded. Transcripts can be PII-redacted after the call, retention is set per data category, data-subject deletion is supported, and an AI disclosure plays first on every call with tools off and cannot be cut short.
Known limits: Callers are not identity-verified, private endpoints are built into the templates but off by default and not yet deployed, and an external penetration test is planned before the first customer install.