NVIDIA Releases NemotronLabs VoiceChat 11B Speech Model
NVIDIA launches NemotronLabs VoiceChat 11B, an open full-duplex speech-to-speech model featuring ~450 ms turn-taking and live tool calling capabilities.
Stock photo for illustration only, not from the actual event
- NVIDIA introduces the NemotronLabs VoiceChat 11B model for research purposes
- Features rapid turn-taking responsiveness at approximately 450 milliseconds
- Built on a hybrid Mamba/Transformer architecture trained on 550k hours of audio
- Comes with documented limitations including a two-minute audio context ceiling
NVIDIA has expanded its artificial intelligence portfolio with the introduction of NemotronLabs VoiceChat 11B, an open-source full-duplex speech-to-speech conversational model. Both the model weights and the container have been made publicly available under a permissive license. However, the NVIDIA team emphasizes that the current checkpoint is strictly intended for research purposes and is not yet ready for production deployments.
Under the hood, the model utilizes a hybrid Mamba and Transformer architecture, combining three existing NVIDIA components with a newly developed output path. The system generates multiple output streams concurrently, including agent audio, agent text, and a running user transcription.
The training regimen for VoiceChat 11B involved approximately 550,000 hours of audio data spanning both real and synthetic corpora, building upon previous foundational frameworks such as SALM-Duplex and Audio Flamingo 3 to enhance conversational fluidity.
Stock photo for illustration only, not from the actual event
NVIDIA has outlined specific operational constraints for the model. Developers are advised to limit sessions to a maximum of five tools, as the model cannot reliably execute multiple tools simultaneously. Furthermore, users cannot interrupt the agent while a tool execution is actively running, and both system prompts and tool responses must be strictly ASCII-only to ensure compatibility with text-to-speech engines.
The release of NemotronLabs VoiceChat 11B highlights NVIDIA's ongoing drive to bridge the gap between human conversation dynamics and artificial intelligence responsiveness. By utilizing a hybrid Mamba and Transformer backbone, the model effectively tackles long-context audio challenges. Nonetheless, documented failure modes such as audio degradation and run-away self-talk underscore the experimental nature of full-duplex speech systems at this current technological stage.
Performance metrics across standard evaluation benchmarks highlight the following results:
- Full-Duplex-Bench 1.0: Smooth turn-taking TOR of 0.82 at 448 ms, user-interruption TOR of 1.00 at 480 ms, and pause-handling TOR of 0.153 (synthetic) and 0.255 (Candor), where lower scores represent better performance.
- AU Harness BFCL-v3 Spoken Tool Calling: Achieved 58.5% in simple tasks, 62.5% in multiple, 42.5% in parallel, 27.5% in parallel-multiple, 89.6% in irrelevance, and 56.1% on average.
- Full-Duplex-Bench v3: Reached 82.5% in tool selection, 44.2% in argument accuracy, and 33% in pass@1.
In comparative rankings, NVIDIA reports that VoiceChat 11B secures the #2 position among open full-duplex models on VoiceBench, as well as #2 among open models on Full-Duplex-Bench 1.0. Developers can access the project via the Hugging Face model card, GitHub under NeMo Speech, and the NGC container.
Source: MarkTechPost
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment