Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have developed a new approach to language modelling, challenging the conventional belief that smaller models have limited capabilities. Their new self-learning model, named SimPLE, uses self-training to allow the model to learn from its own predictions, without the reliance on human-generated annotations. This smaller model has surpassed larger counterparts by up to 500 times in specific language understanding tasks. The model can automatically label a large amount of data but still avoid wrong predictions that might mislead the model, thanks to the method’s combination of uncertainty estimation and voting. The potential of smaller models as equally powerful and environmentally sustainable alternatives has been pointed out by the researchers.
MIT is a private research university in Cambridge, Massachusetts, United States. MIT has five schools, including the School of Engineering, School of Science, Sloan School of Management, School of Humanities, Arts, and Social Sciences, and School of Architecture and Planning. MIT is a member of the Association of American Universities (AAU).
Hongyin Luo is an MIT CSAIL postdoctoral associate and research lead author. Luo is part of the Entailment as Robust Self-Learners project that is scheduled to be presented in July at the Meeting of the Association for Computational Linguistics in Toronto, Canada. Luo, James Glass and Yoon Kim are the authors of a paper that has outlined this research project.
