Meta Platforms recently introduced Voicebox, a machine learning model that can generate speech from text. Voicebox can perform various tasks including editing, noise removal, and style transfer, which sets it apart from other text-to-speech models. The model has been trained across six languages and is not confined to a specific task. This capability will power many applications in the future, and Voicebox can be used to bring speech to people who are unable to speak, customize the voices of non-playable game characters and virtual assistants, or help individuals communicate in a natural, authentic way.
The researchers at Meta utilized a special training method, Flow Matching, to train the model. This technique is far more efficient and generalizable than diffusion-based learning methods used in other generative models. Voicebox can learn from varied speech data without those variations having to be carefully labeled, which was made possible by training Voicebox on 50,000 hours of speech and transcripts from audiobooks.
The model uses text-guided speech infilling as its training goal, which means it must predict a segment of speech given its surrounding audio and the complete text transcript. During training, the model is provided with an audio sample and its corresponding text. It is then trained to generate the masked part using the surrounding audio and the transcript as context. The model learns to generate natural-sounding speech from text in a generalizable way.
One of the interesting applications of Voicebox is voice sampling. The model can generate various speech samples from a single text sequence. This capability can be used to generate synthetic data to train other speech processing models. Our results show that speech recognition models trained on Voicebox-generated synthetic speech perform almost as well as models trained on real speech, with 1 percent error rate degradation as opposed to 45 to 70 percent degradation with synthetic speech from previous text-to-speech models, Meta writes.
Despite its many uses, Voicebox has limits. Since it has been trained on audiobook data, it does not transfer well to conversational speech that is casual and contains non-verbal sounds. It also doesn’t provide full control over different attributes of the generated speech, such as voice style, tone, emotion, and acoustic condition. Meta is exploring techniques to overcome these limitations in the future.
Due to ethical concerns about the misuse of AI-generated content, Meta has not released Voicebox. However, they have provided technical details on the architecture and training process in their technical paper. The paper also includes details about a classifier model that can detect speech and audio generated by Voicebox to mitigate the risks of using the model.
