CALLVIO
A voice AI demo that sounds fluent is not a deployment test. A Pakistani business needs to know whether callers can be understood over real phone networks, whether the agent handles Urdu-English switching, whether it correctly reads numbers and local names, and whether it finishes tasks without inventing facts.
Why standard speech benchmarks are not enough
In the peer-reviewed 2025 paper “WER We Stand: Benchmarking Urdu ASR Models,” researchers evaluated Urdu automatic speech recognition on both read and conversational speech. Different model families performed best on different conditions, and the authors highlighted Urdu text normalisation and conversational evaluation challenges. A strong English-language demo therefore says very little about actual Urdu call performance.
The practical question is not merely “Can this system transcribe Urdu?” It is “Can this customer complete the intended action accurately, in their preferred language, through a real business telephone connection?”
What to test in Urdu and English AI phone calls
Speech understanding asks whether the input was recognised accurately: names, dates, amounts, addresses, intent and changes in language. Reasoning and tool use ask whether the agent selected correct business information or authorised actions. Conversation quality asks whether it responded in a natural time, handled interruptions and recovered gracefully from mistakes.
Score all three. A perfect transcript followed by a fabricated order status is a failed interaction. An imperfect transcript can still lead to a correct resolution, although errors involving amounts, identities or instructions require particular care.
Build a representative Pakistani test set
- Everyday Urdu at conversational speed, not only slow or scripted speech.
- Pakistani English from callers with different regional accents.
- Urdu-English code-switching within one sentence and across turns.
- Roman Urdu phrasings expressed aloud; Roman Urdu is a writing convention, so the audio tests target colloquial speech rather than written script.
- Personal names, product brands, Pakistani cities and neighbourhoods such as Gulberg, DHA and Faisalabad.
- Pakistani mobile numbers, order references, amounts in rupees, dates and time expressions.
- Audio from different mobile networks, headsets, speakerphones and noisy environments.
- Interrupted speech, repeated questions, corrections and explicit requests for a human.
- A caller who changes language, changes their decision or says an unexpected word.
- Low-confidence speech and cases where the correct action is to ask again.
Twelve starter test utterances
- “Mujhe apne order ka status maloom karna hai.”
- “Is there a booking at six this evening?”
- “Kal DHA wali branch khuli hogi?”
- “Meri appointment reschedule kar dein for Friday evening.”
- “Mera phone number zero three double zero…” followed by a realistic number.
- “Mujhe Faisalabad delivery charges bata dein.”
- “Can you repeat that a little slowly?”
- “No, I said fifteen, not fifty.”
- “Actually English mein baat karein please.”
- “Mujhe human agent se baat karni hai.”
- “Mera order number galat likha gaya hai.”
- “I want to cancel that request, not confirm it.”
Treat these as example test prompts, not representative benchmark evidence. Real testing should use consented recordings across multiple speakers, environments and business tasks.
Word Error Rate: useful but incomplete
Word Error Rate (WER) = (substitutions + deletions + insertions) / reference words. A lower WER is usually desirable, but punctuation, spelling normalisation and the reference transcript affect the result. Code-switched Urdu requires consistent transcription conventions before comparisons are meaningful.
Track slot accuracy separately for phone numbers, dates, prices, names, and order IDs. An error in a customer's order number may matter more than three harmless filler-word differences. Report task completion, false confirmations and escalation results alongside WER.
Measure latency and interruption recovery
Record time to first useful AI audio, end-of-turn detection, p50 and p95 delay, and whether the AI stops speaking when interrupted. Use full telephone-path tests, since network audio, voice activity detection, API calls and speech output all contribute to response time.
A fast but incorrect reply is not useful. Record failures where the agent begins talking too soon, speaks over the caller, misses a mid-sentence correction or repeats a previous answer after an interruption.
A transparent scoring method for a pilot
For each test case, predefine the correct intent, required entities, authorised next action and expected response. Score understanding, factual support, correct tool use, conversation behaviour and handoff. Make exposure of private data, fabricated transactions or unsafe actions automatic failures independent of the overall score.
Review a balanced sample manually with bilingual evaluators. Keep a separate error log for speech, reasoning, retrieval, tools and telephony so engineering work targets the actual failure source.
A call-quality test must include a completed business task
Include requests containing Urdu names, English product codes, Roman Urdu phrasing, repeated phone numbers and an interrupted appointment request. After each call, verify the captured fields against the transcript and the connected business record. Good speech recognition is insufficient when the agent mishears an address, books the wrong slot or fails to hand off an unclear request.
Use this evidence in the Pakistan vendor-selection scorecard and check peak-traffic call handling before production rollout.
How do you benchmark an Urdu AI voice agent on real Pakistani phone calls?
Use a defined set of consented or synthetic test utterances across Urdu, English and mixed-language speech, then test the same tasks over actual telephone connections. Record the test configuration, network, language mix, noise conditions, model version and expected outcome. Evaluate whether the system captures names, phone numbers, locations and amounts correctly, not only its overall transcription error rate.
Report separate measurements for critical entity accuracy, first-response latency, interruption recovery, successful transfers and verified task completion. For example, a correctly transcribed appointment date with a failed calendar write is still a failed booking. Publish the number of trials, the scoring rules and failure examples alongside any results so readers can judge what the benchmark does and does not prove.
Related guide: engineering challenges in building voice AI for Pakistan
Frequently asked questions
Can voice AI understand Urdu-English switching?
Some models support multilingual speech, but quality varies by accent, acoustics and conversation type. Test the exact deployed configuration using your own caller scenarios.
Is Roman Urdu a spoken language?
No. Roman Urdu is Urdu written using the Latin alphabet. For voice testing, evaluate colloquial Urdu and English speech; for WhatsApp text, separately evaluate Roman Urdu spelling variations.
What is an acceptable WER?
There is no universal percentage suitable for every business. Judge task-specific risk, especially the accuracy of identity, amounts, dates and instructions.
Should a vendor be evaluated using only recorded audio?
No. Use both repeatable recorded scenarios and live calls over real telephony; interruptions and network effects can change the result.
Related guidance and source
Use our guide to AI agent guardrails for handoff and action-safety requirements, and our article on choosing an AI voice agent in Pakistan for procurement evaluation. Contact Callvio to design a scenario-based pilot; no performance guarantee is implied without a measured test.
