TL;DR
NVIDIA published a how-to for building a fast voice agent using three of its open models: Nemotron Speech ASR, Nemotron 3 Nano LLM, and a preview Magpie TTS checkpoint. The project combines Pipecat streaming building blocks, a WebSocket transcription server, and optimizations for sub-24ms final ASR latency to enable responsive multi-user and local deployments.
What happened
NVIDIA and collaborators demonstrated a voice agent built from three NVIDIA open models: the recently released Nemotron Speech ASR (now on Hugging Face), the 30B-parameter Nemotron 3 Nano LLM, and a preview checkpoint of the Magpie text-to-speech model. The example integrates Pipecat’s low-latency agent components and a WebSocket transcription server that runs a cache-aware streaming ASR pipeline. Nemotron Speech ASR is tuned for realtime use, with reported final transcript latencies consistently under 24 milliseconds and word error rates comparable to top commercial systems on NVIDIA benchmarks. The implementation uses a 160ms ASR context to align with parallel turn detection via Pipecat Smart Turn; the pipeline triggers finalization on a 200ms pause and appends synthetic silence to meet the model’s trailing-context needs. The repo includes instructions to run the agent at scale on Modal cloud or locally on DGX Spark and consumer GPUs such as an RTX 5090.
Why it matters
- Sub-24ms final transcription latency can materially improve perceived responsiveness for conversational voice agents.
- Open-model availability and permissive licensing let teams customize inference stacks, fine-tune models, and host inside private VPCs for data controls.
- Efficient LLM and ASR designs make low-cost, low-latency voice agents more accessible beyond proprietary-cloud offerings.
- A streaming ASR plus parallel turn detection pattern demonstrates a practical approach to reducing end-to-end voice-response time.
Key facts
- Three NVIDIA models used: Nemotron Speech ASR, Nemotron 3 Nano (30B), and a preview Magpie TTS checkpoint.
- Nemotron Speech ASR is published on Hugging Face and optimized for extremely low-latency streaming transcription.
- NVIDIA reports Nemotron Speech ASR delivers final transcripts consistently in under 24 milliseconds on their benchmarks.
- Nemotron Speech ASR offers four context sizes: 80ms, 160ms, 560ms, and 1.2s; the example uses the 160ms setting.
- Turn detection runs in parallel using Pipecat Smart Turn and the system triggers on a 200ms pause, plus 120ms synthetic silence to finalize transcripts.
- Nemotron 3 Nano (30B) is positioned as top-performing in its class on multi-turn conversation benchmarks and is quantizable for consumer-class GPUs.
- The models are released under the NVIDIA Permissive Open-Model License, which permits unrestricted commercial use and derivative works.
- Code and examples are available in a public GitHub repository and can be deployed on Modal cloud or locally on DGX Spark / RTX 5090 hardware.
- NVIDIA reports ASR accuracy (word error rate) on benchmarks roughly equivalent to the best commercial ASR models.
What to watch next
- Timing and performance details for the final NVIDIA Magpie TTS release — not confirmed in the source
- Broader adoption of Nemotron Speech ASR in production voice systems beyond NVIDIA benchmarks — not confirmed in the source
- Progress on speech-to-speech LLMs replacing multi-model pipelines in production voice agents — not confirmed in the source
Quick glossary
- ASR (Automatic Speech Recognition): Models that convert spoken audio into text and related speech metadata.
- LLM (Large Language Model): A neural network trained on large text corpora to generate or analyze natural language.
- Text-to-Speech (TTS): Systems that synthesize spoken audio from text input.
- Turn detection: Techniques for determining when a speaker has finished a turn so the agent can respond.
- Word Error Rate (WER): A common metric for ASR accuracy that measures transcription errors relative to a reference.
Reader FAQ
Is the example code available?
Yes — the post links to a GitHub repository with the code for the voice agent.
Can these NVIDIA models be used commercially?
Yes; Nemotron Speech ASR and Nemotron 3 Nano are released under the NVIDIA Permissive Open-Model License, which allows unrestricted commercial use and derivative works.
Does Nemotron Speech ASR match commercial ASR accuracy?
NVIDIA reports that Nemotron Speech ASR has benchmark word error rates roughly equivalent to the best commercial ASR models.
Is the Magpie TTS model production-ready?
Not confirmed in the source.
How to Build Ultra-low-latency Voice Agents With NVIDIA Cache-aware Streaming ASR This post accompanies the launch of NVIDIA Nemotron Speech ASR on Hugging Face. Read the full model announcement here….
Sources
- Building voice agents with Nvidia open models
- NVIDIA Unveils New Open Models, Data and Tools to …
- How to Build a Voice Agent with RAG and Safety Guardrails
- Nvidia launches Nemotron 3 model family as open …
Related posts
- Caterpillar pilots Nvidia-powered AI agents in its construction machines
- Tailwind Labs says it cut about 75% of its engineering staff amid downturn
- Odinn unveils Omnia luggable datacenter with up to 4 Nvidia H200 GPUs