Open-Source Voice AI Models Worth Testing for Your Voice Agent
Voice AI no longer needs a paid speech service billed by the minute. Open models for listening and speaking are now good enough for many business uses, and they run on your own server for a fixed cost. The catch is choosing well: licences, languages and speed vary a lot.
The short answer: for listening (speech-to-text), start your shortlist with Whisper-family models or NVIDIA Parakeet. For speaking (text-to-speech), start with Kokoro for a small, fast, commercially licensed voice, Chatterbox if you need voice cloning, and CosyVoice2 for many languages. We work with any open model, these or others, and pick by testing each on your own audio before you commit. Our Agent Assurance team runs that test before any voice agent goes live.
What makes a voice AI model dependable?
A model that tops a leaderboard can still be the wrong choice. We judge voice models on four tests:
- Licence. Can you use it in a commercial product, and on what conditions? MIT and Apache 2.0 are the simplest. Some licences add rules, such as attribution or user limits.
- Speed on your hardware. Does it keep up on the server you can afford, with or without a GPU?
- Accuracy on your audio. Accents, noise, phone-quality sound and your industry’s words.
- Languages. Does it handle every language your customers speak, including mixed-language speech?
Which open-source speech-to-text models are worth testing?
| Model | Licence | Strength | Watch out for |
|---|---|---|---|
| Whisper family (via whisper.cpp) | MIT | Many languages, runs on CPU, large community | Larger versions need more memory; can invent words in silence |
| NVIDIA Parakeet | Check model card | Fast and accurate, especially in English | Fewer languages than Whisper |
| IBM Granite speech models | Check model card | Part of the Granite family, built for business use | Newer, so test carefully |
If you want a ready-made desktop tool rather than a library, OpenWhispr is an open-source (MIT) app that runs Whisper and Parakeet models locally. We used it in our own laptop voice agent (see A Voice AI Agent on a 16 GB Laptop).
Which open-source text-to-speech models are worth testing?
| Model | Licence | Strength | Watch out for |
|---|---|---|---|
| Kokoro | Apache 2.0 | Tiny (82 million parameters), fast, about a dozen languages including Hindi | No voice cloning |
| Chatterbox | MIT | Natural voice cloning from a short sample, built-in audio watermark | Slower than Kokoro |
| CosyVoice2 | Apache 2.0 | Strong across many languages, supports cloning | Larger, needs more hardware |
| Orpheus | Llama licence | Wide emotional range | Requires “Built with Llama” attribution; check Llama terms |
We use Kokoro-82M as the voice of our own laptop voice agent (see A Voice AI Agent on a 16 GB Laptop). Licence and feature details from the 2026 open-source TTS guide by TTS.ai; confirm each on the model’s own page.
Each lane lists models worth shortlisting, with their licences; none goes into a product until it passes a test on your own audio.
How do you test a voice stack?
Voice agents fail in ways text bots don’t. We test the whole loop, not each model alone:
- Record a golden set. 50 to 100 real questions, spoken by different people, in real conditions: office noise, phone lines, accents.
- Score listening. How many words did speech-to-text get wrong? Pay most attention to names, numbers and product codes.
- Score speaking. Have people rate clarity and naturalness, and check that numbers, dates and abbreviations are read correctly.
- Check the end-to-end task. Did the caller get the right answer or action?
- Re-test on every change. A new model version, a new voice or new hardware can shift results.
A change that misses a target goes back to be fixed, and the failing case joins the test set so the same mistake can’t return. The same approach applies to every agent, voice or text (see How to Test AI Agents).
Common mistakes
- Choosing a model from a demo clip instead of your own audio.
- Ignoring the licence until the product is built.
- Cloning a real person’s voice without written consent.
- Testing in English only when customers speak several languages.
- Measuring each model alone and never the full conversation.
How SHVX can help
Our Agent Assurance team builds voice test sets, scores speech-to-text and text-to-speech models on your own audio, and checks licences before you build. The result is a voice stack you can run on one server at a fixed monthly cost.
Learn more about Agent Assurance.
FAQ
What is the best open-source speech-to-text model?
There is no single best. Whisper-family models are the safe default for many languages; Parakeet is a strong option for English. Test both on your own audio.
Which open-source text-to-speech models allow commercial use?
Kokoro and CosyVoice2 (Apache 2.0) and Chatterbox (MIT) are permissive. Always confirm on the model’s own page.
Can open-source voice models run without a GPU?
Many can, especially smaller ones such as Kokoro and smaller Whisper versions. A GPU helps when one server handles many calls.
Is voice cloning legal?
It depends on consent and local law. Only clone a voice with the speaker’s written permission.
Licence details checked in November 2026. Licences change; confirm each one on the model’s own page before you build.