Why prompts do not fix a voice agent: what we changed after real calls

In short

• A voice AI agent took orders for a bakery that supplies retail outlets. It worked in tests and "said whatever it wanted" on real calls.
• Swapping models and rewriting the prompt did not help. Moving decisions from the prompt into code did: the tool decides the item, availability and price and returns a ready line.
• The prompt shrank from 19.7k to 7.9k characters, and item search from live speech went from 57% to 93% in test dialogues.
• Long silences, invented stock levels and dropped items had technical causes, not a "bad model".
• The honest result: on real calls the agent is still the manager's assistant, not a replacement. But we know exactly what to do next.

The job: phone orders every day

The bakery calls its retail outlets along delivery routes every day and collects orders. In a minute an operator gets 5 to 15 items named any way the customer likes: abbreviated, in mixed languages, with mistakes. We built a voice agent that calls the outlet itself, takes the order and creates a draft invoice in the CRM. Details in the bakery voice agent case.

In browser test calls the agent behaved well. The problems started once we connected the phone line and real customers.

The symptom: the agent "says whatever it wants"

The owner had tried several models and dozens of prompt versions. The agent still confirmed every item ("Got 10. Is that right?" up to eight times per call), dropped suggestions in the middle of dictation and lost items said while it was speaking. Each new prompt fixed one thing and broke another.

The cause was not the model. A 19.7k-character prompt was trying to make the model a state machine: when to ask, when to confirm, how to count, what to suggest. A model cannot reliably run that logic in every reply, especially in real time.

The fix: code decides, the model only speaks

We rebuilt the agent around a simple principle. Everything that can be checked is computed by tool code: find the item in the price list, check availability, fill in the price, decide whether a clarification is needed. The tool returns a ready line, and the model voices it and keeps the conversation going.

• The prompt went from 19.7k to 7.9k characters.
• Clarifications happen during dictation, not after the whole list is confirmed.
• Behaviour stopped depending on which model is plugged in. It can be verified by tests instead of guessed.

Long silences are rarely the database

Customers complained that after dictating, the agent stayed silent long enough that they had to say "hello?" several times. We checked the database: it computed an eleven-item order in a fraction of a second. In reality every item was a separate round trip of model, tool and database, and three tools had no holding phrase.

Now every agent action has a short phrase such as "one second, writing that down" and a second one if the lookup runs longer than a few seconds. We also tuned pauses for the phone: on a call people read silence as a dropped line, and noisier audio makes the system decide more often that the speaker has finished.

Invented stock levels and test data in the live prompt

The nastiest bug: the agent offered an outlet a pie it had just ordered and quoted a stock level nobody had entered for weeks. The cause was a forgotten "demo card" with a test customer left in the prompt from development.

The lesson for every project: quantities, prices and availability come only from a tool, never from prompt text. If there is no data, the tool returns a line such as "stock is not tracked for this item, do not quote quantities". And we reread the full prompt in the platform dashboard before every release.

People never name products the way the price list does

The agent did not recognise a hot dog item named the way customers say it, and logged a poppy-seed loaf as the raisin one because a synonym fired on any phrase containing that word. We ran a test: 267 price-list items, 8 phrasings each, 2,119 phrases in total with mixed languages, abbreviations, recognition errors and filler words.

• The old search recognised 57% of phrasings.
• Hand-written synonyms covered only a small share of real failures from the call log.
• The new similarity search with ranking and clarifying questions when the agent is unsure reached 93% in test dialogues.

We also rank first the items this customer ordered in the last 90 days: for a short word like "loaf" the shorter name always looks more similar, and without customer history the agent picked the wrong one. When two managers correct a heard word to the same item, it becomes a synonym automatically.

The honest result and what comes next

We measure quality on real calls: how many items were captured correctly, how many invoices the manager did not have to edit, how often customers hang up. Those numbers said honestly that for now the agent is the manager's assistant, not a replacement: it collects the order and a person checks the invoice.

They also showed the direction. In an experiment on 12 recorded calls the live agent captured 85% of items correctly, while a stronger model reviewing the recording after the call reached 96%. So the next step is not another prompt, but a post-call review of the recording to check the invoice.

What to take from this if you are planning a voice agent

• If the agent behaves unpredictably, do not swap the model first. Look at how much logic you are pushing into the prompt.
• Numbers, prices and availability come only from tools.
• Every action that takes time gets a holding phrase.
• Test item search on hundreds of real phrasings, not on price-list names.
• Measure results on real calls and be honest about where the agent already replaces a person and where it still assists.

How we build voice agents and what it costs: AI receptionist and voice AI agent. For outbound calls to customers see AI outbound calling.