How we test a voice agent before launch: 7 checks from real projects

In short

• Most voice AI agent problems are visible before launch if you test specific situations from your business, not just "does it talk".
• At a diagnostic centre, staff collected 37 remarks after the first test, and 12 of them were covered by data the clinic already had. The model had nothing to do with it.
• We run the seven checks below in every project: knowledge base, question order, phone numbers, product names, line pauses, voicemail detection, a pilot on a small group.

1. Audit the knowledge base as a separate step

After the first voice agent test at a diagnostic centre, the clinic team collected 37 remarks. We checked each against the documents the clinic had provided. 12 were covered by data the client already had: scanner names, directions, last patient hours, when results are ready. It simply never reached the agent. Several other questions had no answer in any document: sedation for children, weight limits, insurance, how results are delivered. And some documents contradicted each other.

Now we audit the knowledge base before any prompt changes: what the agent must know, where it is written, whether there are contradictions. Open questions go back to the client instead of being invented.

2. Question order and what the caller cannot know

The agent should not ask people things they cannot know. A patient does not choose which scanner the MRI is done on: that follows from the exam and the referral. Order matters too: implants and stents must be asked about before agreeing a time, otherwise the booking has to be cancelled. And the agent must not re-ask what the person already said, such as their weight.

We run typical scenarios and list every question the agent asks: can the caller answer it, does it come too late, is it repeated.

3. Phone numbers by voice

When the agent repeats a number digit by digit, the number gets mangled. In one test it collected more digits than any phone number has. Now the agent takes the number in groups and confirms it once at the end, and the system checks the digit count before saving.

4. Product and service names from live speech

People never name services and products the way the price list does. At the clinic the agent did not recognise an exam that was in the catalogue because a synonym was missing. At a bakery we tested search on 2,119 real phrasings across 267 price items: the old approach recognised 57%, similarity search with clarifying questions reached 93% in test dialogues. More in why prompts do not fix a voice agent.

Before launch we collect names from real call recordings, not from the price list, and test against them.

5. Pauses and silence on a phone line

An agent that sounds fine in the browser can stumble on the phone. On a call people read silence as a dropped line, and noisier audio makes the system decide more often that the speaker has finished. So we test real phone calls separately, and every action that takes time has a holding phrase.

6. Voicemail detection and long greetings

On outbound calls the voicemail detector mistook a person silently listening to a long greeting for voicemail and hung up. We shortened the greeting and switched detection to the transcript. Now we test the first seconds of the call before any outbound campaign.

7. A pilot on a small group and honest metrics

The first real calls go to a small group of loyal customers who were warned in person beforehand. The agent's first line says who is calling and why. We count how many requests the agent closed on its own, how many records a manager had to edit and how often customers hang up. Those numbers, not impressions from tests, decide whether to expand.

One more rule from the clinic: if the agent promises that an operator will confirm the time and call back, the system must not release that slot automatically. We never auto-cancel such holds; the front desk sees the ones waiting too long.

Pre-launch checklist

• Knowledge base checked against documents, open questions returned to the client.
• Every agent question: the caller can answer it, it comes at the right time, it is not repeated.
• Phone numbers taken in groups, digit count validated.
• Name search tested on phrasings from real calls.
• Tested on the phone, not only in the browser. Holding phrases on every slow action.
• For outbound: a short greeting and a voicemail detection test.
• A pilot on a small group with metrics.

How we build voice agents for clinics and businesses: AI receptionist and voice AI agent, AI for clinics, clinic case.