Best LLMs for Voice Agents: Why Model Choice Is a Workflow Decision

Most people ask the wrong question first.

They ask, "What is the best LLM for voice agents?"

A better question is:

What kind of conversation are we trying to run, and what does the business need after the call?

That distinction matters.

A voice agent is not just a chatbot with a phone number attached. Real-time calling has a much smaller margin for error. A delayed response feels awkward. A vague answer breaks trust. A poor handoff makes the next person repeat the same questions. A cheap model can become expensive if it produces messy outcomes. A powerful model can become wasteful if it is used for simple routing.

At DialoGrove, we think about LLM selection as a workflow decision, not a leaderboard decision.

The right model depends on latency, reasoning depth, instruction following, cost, observability, geography, voice quality, tool use, and the operational value of the call.

For real estate, mortgage, home services, and other lead-heavy businesses, that choice directly affects customer experience and commercial outcome.

Voice agents are different from chatbots

A text chatbot can pause for a second and most users will tolerate it.

A voice agent cannot.

In a phone call, people expect rhythm. They interrupt. They hesitate. They change direction mid-sentence. They ask vague questions. They sometimes give half an answer and expect the other side to understand the intent.

That makes real-time voice AI harder than normal text automation.

A good voice agent needs to do several things at once:

• listen accurately
• respond quickly
• follow the playbook
• ask natural follow-up questions
• know when to stop talking
• handle interruptions
• capture structured information
• route the outcome correctly
• know when a human should take over

The LLM is only one part of that system. The full voice stack matters.

The two main architectures for voice agents

Most real-time voice agents use one of two architectures.

1. Cascaded voice pipeline

A cascaded pipeline uses separate components:

Speech-to-text -> LLM -> Text-to-speech

The customer speaks. Speech-to-text turns the audio into text. The LLM decides what to say. Text-to-speech turns the response back into audio.

This architecture is popular because it is flexible and observable. You can swap the speech-to-text provider, the LLM, or the text-to-speech provider. You can inspect transcripts. You can debug each stage. You can improve cost, latency, voice quality, or language support without rebuilding the whole system.

That observability matters for business workflows. If an agent qualifies a seller lead, books a callback, or flags a conversation for human review, the team needs to understand why.

The trade-off is latency. Every step adds time. Even when every component is fast, the total delay can still feel noticeable if the system is not engineered carefully.

2. Native speech-to-speech model

A native speech-to-speech model processes audio directly and produces audio directly.

Audio in -> Voice model -> Audio out

This can feel more natural because the model is built for real-time voice interaction. It can often handle interruptions and tone more fluidly than a stitched pipeline.

The trade-off is control. Native speech-to-speech systems can be less transparent because there may be fewer intermediate artifacts to inspect. They can also create more vendor lock-in because listening, reasoning, and speaking are bundled together.

For some calls, that is a good trade. For others, especially workflows where auditability and structured outputs matter, a cascaded pipeline may still be the better choice.

There is no single best LLM for voice agents

The best LLM for a voice agent depends on the job.

A top-of-funnel triage call does not need the same model as a complex seller qualification call. A post-call analytics task does not need the same model as a live conversation. A regulated workflow needs different controls from a casual FAQ call.

Here is the practical way to think about it.

The important point is this:

It should match the model to the workflow.

What matters when choosing an LLM for voice calling

When evaluating LLMs for voice agents, the headline benchmark is not enough.

You need to look at the operational requirements.

Latency

Can the model respond fast enough for natural conversation?

Voice is unforgiving. If there is too much silence, people talk over the agent, lose confidence, or hang up. Latency is not only about the model. It also depends on speech recognition, text-to-speech, telephony, orchestration, and geography.

Reasoning depth

Can the model handle vague answers?

This is especially important in real estate.

If a seller says, "We are thinking of selling, but it depends on the market," the agent should not jump straight to booking. It should ask a useful follow-up question.

For example:

That is not just conversation quality. That is qualification quality.

Instruction following

Can the model stay inside the approved playbook?

For real business workflows, this matters more than sounding clever. The agent must know what it can say, what it should avoid, what it needs to capture, and when to escalate.

Tool calling and integrations

Can the model reliably trigger the right action?

A voice agent may need to book a meeting, update a CRM, create a callback task, route a lead, or send a conversation to review. If tool calls are unreliable, the workflow breaks after the call.

Observability

Can the team inspect what happened?

For operational trust, teams need transcripts, summaries, structured fields, outcome labels, recordings, and review trails. A beautiful voice experience is not enough if the business cannot understand the outcome.

Cost per conversation

Can the model economics support real usage?

Voice agents run by the minute. Long prompts, verbose answers, expensive output tokens, speech synthesis, telephony, and orchestration all add up. The cheapest model is not always the best, but unit economics matter if the workflow is high volume.

Geographic infrastructure

Where does the audio travel?

For teams in Australia and other non-US regions, this is easy to underestimate. If audio moves across multiple overseas providers before coming back to the caller, the model may look fast on paper but still feel slow in the call.

The model stack matters more than the model name

A production voice agent is usually a stack, not a single model.

It may include:

• telephony
• speech-to-text
• real-time LLM reasoning
• text-to-speech
• interruption handling
• orchestration
• tool execution
• CRM or calendar integration
• post-call extraction
• human review

This is why model comparisons can be misleading when they only rank intelligence.

For a voice agent, a slightly less powerful model with better latency, cleaner tool use, and lower cost may outperform a more powerful model in the real workflow.

The opposite can also be true. For a complex seller qualification conversation, using a weak model to save money can create poor follow-up, weak notes, and bad handoff. That cost shows up later.

How this applies to real estate voice agents

Real estate is a good test case because the conversations are practical but not always simple.

A buyer may ask about the property.
A seller may be unsure about timing.
An open home attendee may like the property but think the guide is too high.
A past buyer may still be interested, but only in a different suburb.
A rental enquiry may need urgent human attention.

These conversations need more than a script.

They need a playbook that knows:

• what kind of lead this is
• what the agent should ask
• what context matters
• what data needs to be captured
• what outcome should be assigned
• what should happen next
• when a human should step in

That is where LLM choice becomes part of product design.

For example, an Open Home follow-up call may not require a heavyweight model for every turn. The value is consistency, speed, and structured buyer feedback across many attendees.

A seller lead qualification call may need more careful reasoning because the person may be weighing timing, price expectations, emotional factors, and agent trust.

Post-call analytics may not need real-time speed at all. It can happen asynchronously after the call, using a model that is accurate and cost-efficient for extraction and summarisation.

A practical model strategy for voice agents

A strong architecture can use different models for different jobs.

Frontline router

This model answers quickly, identifies intent, and sends the conversation down the right path.

It might decide whether the caller is:

• a buyer
• a seller
• a tenant
• an existing contact
• a wrong number
• someone who needs a human

This model should be fast and inexpensive.

Specialist conversation agent

This model handles the actual workflow.

It may qualify a seller, follow up with an open home attendee, screen a borrower, or help an inbound enquiry move to the right next action.

This model needs stronger reasoning, better instruction following, and reliable tool use.

Post-call extraction agent

This model works after the call.

It extracts structured information, checks outcomes, creates summaries, identifies next actions, and prepares data for CRM or review queues.

This model does not need to be real time. That means it can often be optimised differently for cost and accuracy.

Review and quality layer

Some calls should not move straight to automation.

If a conversation is ambiguous, sensitive, high value, or outside the playbook, it should be flagged for review.

This is part of the system design, not an exception to it.

DialoGrove's view: workflow first, model second

At DialoGrove, we do not think the most important question is which LLM is fashionable this month.

The more important question is whether the voice workflow is safe, useful, and operationally clear.

For us, that means:

• playbooks before prompts
• structured outputs before vague notes
• handoff rules before improvisation
• review queues before blind automation
• integrations before manual copy-paste
• post-call actions before transcripts alone

The model matters. A lot.

But the model is only useful when it sits inside a workflow the business can trust.

That is why an Open Home follow-up playbook, a seller lead qualification playbook, a rental management workflow, and a mortgage intake workflow should not all be treated as the same generic AI call.

They need different questions, different capture fields, different outcomes, and sometimes different model choices.

What to ask before choosing a voice LLM

Before choosing the best LLM for voice agents, ask these questions:

1. Is this a live conversation or post-call processing?
2. How fast does the agent need to respond?
3. Does the workflow need deep reasoning or simple routing?
4. Does the business need strict auditability?
5. Are calls high volume or high value?
6. What structured fields must be captured?
7. What tools or integrations must the agent call?
8. What happens when the lead gives a vague answer?
9. What should trigger human review?
10. Where are callers located, and where is the voice infrastructure hosted?

These questions will tell you more than a model leaderboard.

The best LLM for voice agents is a strategy

The voice AI market will keep changing. Model names, pricing, context windows, and latency claims will keep moving.

That is why businesses should avoid building their whole strategy around one model being "the best".

A better approach is to build a flexible voice architecture that can choose the right model for the job.

Use fast models where speed and cost matter.
Use stronger reasoning where judgement and qualification matter.
Use observable pipelines where compliance and review matter.
Use asynchronous models where real-time latency is not needed.
Use human review where trust matters.

That is how voice AI becomes more than a demo.

It becomes a real operating layer for lead follow-up, qualification, routing, and post-call action.

And that is the standard we are building toward with DialoGrove.