NVIDIA Releases Nemotron 3 Diarization 8-Speaker Model
NVIDIA launches Nemotron 3 Diarization, a 100M-parameter open-weight model supporting real-time speaker tracking for up to 8 people on NVIDIA GPUs.
Stock photo for illustration only, not from the actual event
- NVIDIA releases Nemotron 3 Diarization, a 100M-parameter open-weight model.
- Supports tracking up to 8 simultaneous speakers, doubling the previous 4-speaker limit.
- Ranks 1st in Voice Arena's Diarization-Bench out of 12 evaluated systems.
- Available for commercial use under OpenMDW License 1.1 on Linux systems.
Automatic speech recognition (ASR) provides the text, but it fails to identify who spoke each word, leaving summarization tools unable to determine who made commitments or objections. Speaker diarization solves this by outputting active time intervals for each speaker, combining with ASR to generate a speaker-attributed transcript essential for meeting tools, call analytics, and voice-agent memory.
NVIDIA's previous streaming checkpoint supported 4 speakers, but Nemotron 3 Diarization doubles this capacity to 8, targeting messy multi-party audio where participants talk simultaneously. The model accepts 16 kHz single-channel audio in various formats, transforming inputs into Mel-spectrogram features processed by a 31-layer Transformer encoder featuring rotary positional embeddings (RoPE).
Stock photo for illustration only, not from the actual event
The architecture directly handles overlapping speech by activating multiple channels when two people talk at once. Following the Sortformer approach, speakers are ordered by arrival time to maintain stable labels across streaming chunks, assisted by the Arrival-Order Speaker Cache (AOSC) and a FIFO queue for recent frame contexts, operating at a recommended minimum buffer setting of 0.32 seconds.
Real-time multi-speaker diarization presents significant engineering hurdles due to overlapping dialogue and rapid conversational shifts. NVIDIA's integration of Transformer layers with RoPE and arrival-order caching successfully maintains consistent speaker tracking across audio chunks without requiring costly re-matching steps in every frame, establishing a robust foundation for production meeting intelligence applications.
In Voice Arena's initial Diarization-Bench evaluations, the model achieved a 14.72% Diarization Error Rate (DER) across 139 English conversations, outperforming 12 competing systems. Throughput also experienced massive gains, reaching 15,113× RTFx at 30.4 seconds using BF16 precision on an NVIDIA RTX PRO 5000 GPU compared to the baseline's 2,619×.
Trained on roughly 10,000 hours of real conversations combined with over 82,000 hours of simulated multi-talker mixtures spanning 21 languages, the model can be easily installed via Python 3.12 using NVIDIA NeMo toolkit packages. Developers can load the pretrained weights directly to process audio streams and integrate with ASR tools like Parakeet TDT for full transcript attribution.
Source: MarkTechPost
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment