Mistral: Voxtral TTS, Forge, Leanstral, & what's next for Mistral 4 — w/ Pavan Kumar Reddy & Guillaume Lample

Mistral is releasing Voxtral TTS, their first speech generation audio model, supporting nine languages. This follows their previous audio model, Voxtral, a transcription model with updates including more languages and features like context biasing and real-time transcription. The new TTS model is small and fast, performing at the same level as larger models but at a fraction of the cost. While the model will not be open-weighted this time, Pavan notes it features a novel in-house architecture involving an autoregressive flow matching architecture and a neural audio codec.

Pavan discusses the differences between audio understanding and audio generation models, explaining how audio is fed into the model. Audio understanding models use an audio encoder to produce continuous embeddings, similar to image models. The interesting part of audio generation is producing audio output using a neural audio codec that converts audio into latent tokens. Unlike text, there is no "winner model" in audio, making it an exciting space to explore.

The team chose an autoregressive approach for real-time streaming applications like voice agents. They opted to add audio as another head to their regular transformer decoder model for easier end-to-end modeling. Flow matching was chosen over discrete diffusion for its superior performance. Flow matching models the distribution of predictions at a particular time step, accommodating inflections and pronunciations. Disfluencies, such as "ums" and "ahs," contribute to the entropy that flow matching helps model.

Pavan reflects on the progress made in voice assistants, noting that despite advancements, there's still a gap between speaking to an agent and a person. Guillaume emphasizes that voice is the next step in making interactions with agents more natural and efficient. Mistral helps customers deploy models in-house due to privacy concerns and to leverage their proprietary data for fine-tuning. Fine-tuning models on company-specific data leads to significant improvements, as the model becomes trained on the company's knowledge.

Mistral offers tailored solutions for customers, including training new models with specific language mixes and dialects. They also build custom models for specific use cases, such as offline audio models for cars. Customers often seek Mistral's help after prototyping with closed-source models due to high costs. Mistral provides open-weight models for customers to customize themselves. Pavan notes that voice personalization and adaptation are good use cases for fine-tuning, allowing enterprises to create customized voices that reflect their brand and safety considerations.

Guillaume explains that Mistral Small is a model that combines different models, merging capabilities like function coding. He notes that OpenAI may be moving away from the Omni model vision, but Mistral may pursue it. Guillaume states that it makes no sense to use a large model for simple tasks like transcription, and that separate models are more efficient. He also notes that there are many capabilities that people don't talk about, but that are important, such as computer-aided design.

Swyx notes that joining voice with video is an important trend, especially considering spatial audio. Guillaume emphasizes that open source is core to Mistral's DNA, and that they have been open sourcing since the beginning. He notes that open source allows for faster progress and contributes to the ecosystem. Mistral wants to contribute to the open source community and does not want the best models to be only accessible to companies.

Vibhu asks about the frontiers of research in foundation model training. Guillaume states that they are still working a lot on the pre-training side, and that ML4 pre-training will be a big step up. He notes that they are building infrastructure that will anticipate extremely long scenarios on the up-priced data game. Swyx notes that the team is doing incredible work and has laid out an impressive vision for open source and voice.

Guillaume states that they are hiring a lot of people in their science team, and that they are hiring in all their offices. He notes that they are exploring AI for science, and that there are a lot of areas where they think that you could get some extremely promising results. Guillaume states that a good forward deployed engineer needs to be familiar with the tech, know how to do fine-tuning, and know how to start some RL pipeline.