Why AI voice agents break in India

This article is authored by Sneha Roy, co-founder and COO, Murf.AI.

Over the past year, voice Artificial Intelligence (AI) agents crossed a threshold. What was once a buzzword is now fast becoming infrastructure, getting deployed across global markets for everything from booking health appointments to managing sales pipelines and handling support escalations.

AI (AFP)

This is happening because the problem it addresses is enormous. Voice remains one of the most preferred interfaces for customer engagement. Even today, 71% of Gen Z users prefer calling customer support, reinforcing that voice isn’t just an interface, but behaviour. Yet the system is under tremendous pressure. In India alone, customers spent 15 billion hours waiting on support calls in 2023, resulting in an estimated $55 billion economic loss. The inefficiency is increasingly difficult to ignore as nearly 50% of customer queries are repetitive and automatable, which is why deployment of GenAI-powered voicebots is accelerating.

But India is not a straightforward deployment. The foundational models powering today's voice agents, the LLMs, ASRs, and TTS systems, were built on English data, optimised for English speakers, and benchmarked against western use cases. Within that frame, they work exceptionally well. India breaks the frame.

Deploying in India doesn't mean adding one new language. It means navigating dozens, each carrying its own dialects, code-switching patterns, and acoustic textures. Add a distinct regulatory environment and the unpredictable texture of everyday Indian context, and you have a set of challenges that English-language models were simply never designed to meet.

The challenge, however, is not whether voice AI will scale in India, but whether existing systems are capable of handling the realities of the market. From multilingual and code-mixed conversations to noisy environments, regulatory constraints, and extreme cost sensitivity, nearly every layer of the voice stack behaves differently in India. These are not edge cases. They are baseline conditions that fundamentally shape how voice agents need to be built and deployed.

Most voice agent models were trained heavily on English and far less on Indian languages. Indic datasets are under-labelled, each language has its own script, rhythm, and pronunciation, and accuracy alone isn't enough as a model that misses local accent variation will transcribe correctly but sound wrong.

The deeper problem, however, is code-mixed speech. Indians rarely speak a single language--they mix in words from others, largely English. For example, a user may begin a sentence in Hindi, switch to English midway for certain phrases, and return to Hindi again within the same thought. Word error rates for monolingual Indic speech are already high, and code-switching pushes them 30 to 50% higher. Many STT solutions available for India are not optimised for this and are trained primarily on monolingual input. The underlying issue is one of data: Indic language speech datasets are thin relative to English, and the subset capturing spontaneous code-mixed conversation is smaller still. Errors don't stay contained; a mistake in STT cascades into the LLM and degrades the response. When evaluating models, test explicitly against code-mixed audio samples from your target language and region. Models trained on large Indic corpora perform significantly better here.

TTS has the mirror problem. When the language switches mid-sentence, the model's voice shifts audibly with it. This often results in speech that sounds unnatural or robotic to native speakers, even if the words themselves are technically correct. A possible fix is a linguistic layer that detects the language of each word and transliterates accordingly, keeping prosody consistent across switches.

In India, background noise is not an edge case. Urban sound levels average 70 to 85dB, consistently louder than a typical office or street in the US, and any voice agent deployed here will spend most of its life in that environment. A customer could be speaking from inside a moving auto-rickshaw, a crowded market, near traffic, or with television noise running in the background.

ASR accuracy degrades in noise, but interruption handling is also a problem worth unpacking. Before a voice agent can respond, it needs to know when you have finished speaking. That is the job of a Voice Activity Detection model, or VAD. In noisy environments, continuous background sound makes it genuinely difficult for a VAD to distinguish ambient noise from a speaker pausing mid-thought.

Solutions are still emerging here. A practical starting point is tuning your VAD thresholds against real call recordings from your target environment rather than leaving them at defaults.

Regulated industries in India, including financial services, telecom, and healthcare, require user data to be processed and stored within the country. For voice agents in these sectors, that is not a preference. It is a hard constraint.

Most global models were not built with Indian data residency in mind, and the options that do comply come with trade-offs, usually on model performance. Self-hosting and local hosting options exist but are typically gated behind enterprise plans, making compliance an expensive unlock for smaller deployments.

Before selecting any model for a regulated deployment, verify data residency, hosting, and inference location independently. Assume nothing is bundled. For LLMs, the decision will likely come down to how much performance you are willing to trade for compliance. On the TTS side, confirm that in-region storage and inference hold at every pricing tier, not just enterprise. It is also worth evaluating edge-deployed models, which are increasingly available across price points and can satisfy residency requirements without the cost ceiling of a full enterprise plan.

Based on customer conversations, significant cost pressure exists around voice-agent economics in India, and expectations are only moving lower as the market matures. Cost pressure here is not just a procurement problem. It shapes every architectural decision, from which models you choose to how you structure your infrastructure. Meeting those expectations requires deliberate trade-offs at every layer of the stack.

A possible starting point is lightweight STT and TTS models, combined with routing simpler queries to smaller, cheaper LLMs rather than running every call through a large model. Open source LLMs with targeted tuning could bring costs down significantly, though performance on your specific tasks should be benchmarked before committing. Where usage reaches sufficient scale, self-hosting models can reduce costs and meaningfully improve unit economics in the long run. Committed use agreements with vendors may unlock discounts that are otherwise unavailable.

None of these are silver bullets individually. Used together, they could make the number reachable.

English-language voice agents were built for a specific world defined by low ambient noise, monolingual speakers, and infrastructure costs that are not the primary constraint. India is a different world entirely. The same models that work exceptionally well there will stumble on code-mixed speech, mishear speakers in noisy environments, and run at cost structures that make deployment unviable.

Closing that gap requires purpose-built data, architecture designed for code-mixed and noisy speech, and infrastructure that treats cost as a first-class constraint. The teams willing to do that work will not just be building better voice agents. They will be building the voice layer for one of the largest underserved markets in the world.

(The views expressed are personal)

This article is authored by Sneha Roy, co-founder and COO, Murf.AI.