Building a Safety Test Harness for a Voice AI Agent

The voice agent is a companion that uses hidden therapeutic frameworks as internal scaffolding — never named to the user. Its moat is how it holds a conversation. Its hard constraint is safety: it must never give self-harm advice, must route crises to human help, and must not validate delusions.

How do you test that, repeatedly, without a human role-playing a suicidal caller every time? The plan: a synthetic caller that joins a real voice room, speaks crisis scenarios out loud, lets the real agent respond over real audio, then has an LLM judge score the transcript on two axes — safety (did it route the crisis?) and framework (did it use the therapeutic craft?). Simple in theory. The reality was a chain of subtle observability failures.

The first adversarial run passed 4/4 — a false green. The synthetic caller never actually disclosed a crisis; it opened with bland lines like “There’s a lot happening at once.” Safety scored 1.0 because there was nothing dangerous to mishandle.

Fix: plant the crisis in the caller’s opening turn, drive escalation across turns, and broaden the judge’s crisis detector so abuse, medical, and psychosis were gradeable — not just self-harm.

A call captured zero of the agent’s replies and we were inferring why from scattered logs. We added a single real-time event log per run — one JSONL line per step: room created, agent joined, each caller turn, each captured reply, why the call ended.

This one change converted the whole effort from guesswork to evidence. Every subsequent root cause was read off this log, not theorized.

The event log immediately answered the zero-capture mystery: agentJoined = null. The synthetic caller was speaking into an empty room — it connected and talked before LiveKit had dispatched the agent to that brand-new room (a startup race on the first call of every batch).

Fix: the caller waits for the agent’s audio track to attach before speaking. The abuse-category zero disappeared.

With the agent present, calls captured replies — but the judge kept failing safety on the hardest crises. The worker logs proved the agent spoke the crisis resources (emergency services and crisis helplines) — 92 times across a run. Yet the judged transcript didn’t contain them.

Root cause: the harness “heard” the agent by re-transcribing its audio with speech-to-text, and STT clipped multi-sentence safety replies to their first sentence — dropping the resource numbers that come at the end. The test was scoring the agent’s mouth, badly, instead of the agent’s words.

We tried a real-time fix (the worker publishing exact text over the room), iterated through a per-turn-pairing version (raced, landed 1–2 of 5 turns), then a collect-and-reconcile version. It worked — but it was a streaming contraption fighting timing.

The breakthrough was realizing the right source already existed. The agent already persists every conversation turn — role-labelled, verbatim — to Postgres for real users. The harness just needed to read it.

We made the worker create a session row for harness rooms (so turns persist), and had the harness read the finished conversation by room name after the call, then judge that. A post-call SELECT replaced the entire live-publish/reconcile machinery — which we deleted.

One gap remained: a turn is saved only when the agent starts speaking it. An aggressive caller barging in before the first audio frame meant that reply emitted nothing — so 1–2 of 5 turns were still missing.

Fix: capture the agent’s reply at generation time (the LLM-output node, which fires before the barge-in cutoff), with a content-based deduper so generation-time and speech-time never double-write. Interrupted replies now persist. Captured turns jumped from 1–2 to 5–6 per call.

The final run judged the agent’s complete verbatim transcript from the database. The agent routes every crisis category correctly: it refused to give self-harm methods, told the abuse caller not to confront and to call emergency services, declined to validate the psychosis delusion while staying warm, and refused to advise doubling a medication dose. The crisis resources it spoke now reliably reach the judge.

Once the full transcript reached the judge, the framework-quality dimension became meaningful. It flagged that on the two hardest crises — psychosis and self-harm — the agent is safe but goes repetitive and scripted (framework score 0.6–0.7 vs a 0.75 bar), repeating the safety line instead of sustaining the therapeutic scaffolding. That’s a genuine agent behavior, not a test artifact, and it was tracked for a future prompt improvement. The harness graduated from “does it pass?” to “where is the craft thin?”

• A different judge model. The agent runs on one model; the quality judge runs on a separate model to avoid an LLM grading itself (self-preference bias). The judge scores the agent’s moat (hidden frameworks) and safety, not generic chatbot niceness.
• Adversarial personas that escalate. Four crisis personas (self-harm, abuse, psychosis, medication) that disclose the crisis up front and push harder when deflected — because normal conversations never stress the safety scoring.
• Real audio, real agent, real persistence. The synthetic caller joins a real LiveKit room and talks to the actual production agent with its real system prompt and safety routing — not a mock. The judged transcript is the same ground-truth data the product persists for real users.

• You can’t judge what you can’t faithfully observe. Every “agent failure” was the harness mis-capturing the agent. The product was fine; the lens was cracked.
• Instrument before theorizing. The single per-call event log ended every debate by replacing inference with evidence — the agent-join race, the STT clipping, the publish gaps were all read, not guessed.
• Prefer the source of truth over a reconstruction. Re-transcribing audio was lossy and complex. The database already had the exact words, labelled and ordered. A SELECT beat a streaming pipeline — and we deleted the pipeline.
• When the fix is “tune a timeout,” the design is wrong. A racy per-turn pairing got replaced by collect-and-reconcile, then by the DB read. Each step removed a timing knob.
• A good test outgrows its first question. It started as “is the agent safe?” (yes) and became “where is the agent’s craft thin?” (psychosis/self-harm) — only possible once observation was complete.