Prompting AI voice agents that don’t sound robotic

Text prompt engineering and voice prompt engineering are different disciplines. Voice is real-time, unforgiving, and has no scroll-back. Here are the six patterns we’ve seen consistently produce natural, effective agents.

1. Prescribe pace, not just persona Instead of “Be warm and helpful”, write “Speak in short sentences. Pause after questions. Never say more than 20 words before checking in.” The model matches your instruction style.

2. Ban markdown Models trained on chat sometimes emit lists and headers. Add: “This is a voice call. Do not read bullet points, brackets, or markdown symbols aloud. If you need to enumerate, use ‘first’, ‘next’, ‘finally’.”

3. Explicit turn-taking “After you ask a question, stop and listen. Do not answer for the caller.” Without this, models sometimes finish their own thought and then answer *as* the caller — uncanny valley.

4. Silence handling “If the caller is silent for more than 4 seconds, gently prompt with ‘Are you still there?’ or ‘Take your time.’ Do not repeat the previous sentence.”

5. Disambiguate similar-sounding data “When collecting phone numbers, dates, or names with unusual spelling, read them back in NATO phonetic (Alpha, Bravo, Charlie) or digit-by-digit and confirm.”

6. Fail gracefully “If you don’t know the answer, do NOT invent one. Say ‘Let me flag that for a specialist to call you back — what’s the best number?’ and record the callback intent via the book_appointment tool.”

Bonus: the golden opener The first 5 seconds decide 40% of drop-offs. The pattern that works: > “Hi, this is [Name] at [Company]. I’m an AI assistant — I can help with [top 3 use cases]. How can I help?”

Being upfront about being AI, offering three concrete options, and asking a wide-open question outperforms “How may I direct your call?” by 25% in our A/B tests.

Try these patterns in Callora’s live builder — edit the instructions, hit “Talk live”, and iterate in real time.