French company Kyutai recently unveiled their latest creation, a conversational AI assistant named Moshi, at an event in Paris. The innovative technology boasts the ability to engage in natural conversations with users, a feat that Kyutai claims sets Moshi apart as the first publicly accessible AI assistant with such capabilities. The company plans to release the technology as open source, allowing developers and researchers to explore, adapt, and expand upon its functionalities.
During the presentation, Kyutai CEO Patrick Perez highlighted the swift development of Moshi by a team of eight members in just six months. One of Moshi’s standout features is its real-time speech and listening abilities, with Kyutai reporting a theoretical latency of merely 160 milliseconds. In practical terms, this latency ranges between 200 and 240 milliseconds, showcasing the AI assistant’s rapid responsiveness.
Moshi’s architecture is built on a novel approach termed an Audio Language Model, which eschews the conventional method of converting speech to text. Instead, the model compresses audio data and treats it as pseudo-words, enabling direct interaction with audio data for speech prediction. This approach grants Moshi a natively multimodal capability akin to GPT-4o.
To train Moshi, Kyutai utilized diverse data sources that encompassed human motion data and YouTube videos. The team started with training a text model dubbed Helium before combining text and audio data for further training. Synthetic dialogues were employed for fine-tuning conversational abilities.
Despite Moshi’s underlying language model sporting a modest 7 billion parameters, the AI assistant displays impressive language proficiency and promptness, hinting at its potential with more potent and expansive modules. To infuse Moshi with a consistent voice, Kyutai collaborated with voice actress Alice, who recorded monologues and dialogues across various styles for training a speech synthesis system.
Kyutai envisions numerous applications for Moshi, particularly in enhancing accessibility for individuals with disabilities. The company has made the Moshi demo available online, with plans to release the technology as open source in the near future, accompanied by a forthcoming research paper.
Founded in 2023, Kyutai secured a substantial investment of 300 million euros from prominent French figures like Xavier Niel and Rodolphe Saadé. With renowned AI researchers like Yann LeCun and Bernhard Schölkopf onboard as scientific advisors, Kyutai’s commitment to open science and publishing all models as open source has proven attractive to researchers seeking to contribute to the field.
