At Inflection AI, our goal is to make conversations with Pi feel as natural and seamless as possible. That means bringing real-time speech processing, multi-speaker understanding, and multilingual capabilities together into a unified system that better supports the way people naturally communicate.

Real conversations rarely follow a perfect back-and-forth. People talk to Pi with friends, family, or coworkers around. They interrupt each other, finish each other’s sentences, and sometimes speak at the same time. For Pi to navigate these interactions, it needs to understand not just what is being said, but who is speaking, even when voices overlap.

Our engineering team has been working hard to bring these capabilities together, starting with Inflection AI’s Conversational Interaction Stack and integrating NVIDIA Nemotron Speech open models with our own pipeline to improve future versions of the Pi user experience.

Here’s how we approached building a more capable multi-speaker conversational stack, step by step.

Step 1: Adding diarization with NVIDIA Streaming Sortformer

Our first step was implementing speaker diarization and identifying which speaker is talking at each moment in a conversation.

We started by integrating NVIDIA Streaming Sortformer into Pi’s speech pipeline, giving Pi an initial ability to distinguish between speakers. This established a working multi-speaker foundation that we could build on and refine as we tackled more complex conversational scenarios.

Step 2: Speaker-conditioned ASR with NVIDIA Multitalker-Parakeet

Knowing who is speaking is only part of the challenge. Pi also needs to accurately capture what each person says, particularly when multiple people speak at once.

Our existing automatic speech recognition (ASR) backend was fine-tuned on our data but transcribed audio without distinguishing between individual speakers. In overlapping conversations, this could result in words from different speakers being combined into a single transcript.

To address this, we integrated NVIDIA Multitalker Parakeet (MT-Parakeet) into our existing ASR stack, enabling speaker-conditioned transcription. By using diarization output to guide the transcription process, we can now capture each speaker’s words separately, even in overlapping speech.

This marked an important step toward building a more complete understanding of multi-person conversations. With English-language transcription capabilities in place, our next focus was improving speaker diarization and expanding multilingual support.

Step 3: Improving speaker diarization and overlapping speech with Nemotron 3 Diarization

With speaker-conditioned ASR in place, our next priority was refining diarization accuracy.

Because MT-Parakeet relies on diarization to identify individual speakers, getting those labels right is essential. In real-world conversations, where people interrupt one another or speak simultaneously, even small speaker-labeling errors can affect how Pi interprets and responds to a conversation.

Building on our initial implementation with Sortformer, we integrated NVIDIA Nemotron 3 Diarization, which delivered a clear improvement in speaker diarization in our testing. Following are the results on AMI-SDM in terms of Diarization Error Rate (lower is better).

One of the most valuable improvements was its ability to handle overlapping speech. When two people speak at once, Pi needs to distinguish between their voices and understand which words belong to each person. By combining Nemotron 3 Diarization with MT-Parakeet, we’ve strengthened Pi’s ability to identify who said what, including during overlapping conversations.

This brings us closer to a conversational experience where Pi can follow the natural flow of a group discussion, rather than relying on a sequence of individual turns.

Step 4: Expanding Pi’s capabilities with Nemotron 3.5 Transcribe for multilingual conversations

Pi connects with people around the world, so supporting conversations across languages was another important part of building a more natural and accessible experience.

NVIDIA Nemotron 3.5 Transcribe extends Pi’s speech recognition across languages using a single multilingual model. The open weights also give us a path to adapt the model to Pi’s needs.

The checkpoint we tested performs offline transcriptions. Our early experiments applied it to shorter audio segments, and we’re building the buffering and transcript-handling logic needed to integrate this approach into Pi’s streaming pipeline.

Our initial chunked-inference evaluations on AMI-SDM English benchmarks have shown promising results, as reflected in the word error rates (WERs; lower is better) below:

Since the models MT-Parakeet and Nemotron 3.5 Transcribe are for different cases, the WER comparison is not like-to-like. However, these results show promising direction with Nemotron 3.5 Transcribe in Pi.

An important part of this work is ensuring inference of the Pi ASR model is not very heavy. A single multilingual model could simplify how we deploy and maintain ASR across languages. We’re comparing it with our current pipeline to measure improvements in transcription accuracy, response latency, and serving cost. Integrating multilingual transcription with our speaker-aware pipeline is ongoing work. These improvements represent the next phase of our work: bringing streaming, multi-speaker, and multilingual speech understanding into a unified system that allows Pi to participate in more natural conversations.

What’s Next

Advancing Pi from a primarily single-speaker experience toward one that can better support dynamic, overlapping, multi-person conversations marks an important step in our work building human-centered AI with relational intelligence.

NVIDIA Nemotron Speech open models have provided valuable building blocks along the way. Through our engineering work, we’ve integrated and refined these technologies into an increasingly sophisticated speech pipeline, from establishing speaker diarization with Sortformer to enabling speaker-conditioned transcription with MT-Parakeet and strengthening speaker diarization with Nemotron 3 Diarization.

Now, with our ongoing work on Nemotron 3.5 Transcribe, we’re focused on extending those capabilities across languages while maintaining the speed and responsiveness that natural conversation demands.

The innovation lies not only in the capabilities of individual models, but in how we bring them together in Inflection AI’s Conversational Interaction Stack  to create an AI experience that feels more intuitive, responsive, and human-centered. 

We look forward to releasing this work in public releases of Pi and will be sharing more as this work progresses.