Microsoft has introduced its latest AI model, Orca, that can learn and mimic the reasoning process of large foundation models (LFMs) like GPT-4. LFMs like GPT-4 demand extensive computing resources and pose challenges of large-scale data handling and task variety. Orca, which is a 13-billion parameter model, learns from a vast database of information, including explanations, intricate instructions, and detailed thought processes of GPT-4. However, unlike other AIs, Orca is smaller and tailored for specific use cases, meaning it does not require dedicated computing resources. It can be optimized and tailored for specific applications without the need for a large-scale data center.
One of the most significant differences between Orca and other AIs is its open-source architecture. While ChatGPT and Google Bard are privately owned, Orca supports an open-source framework, encouraging the public to contribute to its improvement. This means Orca can harness the power of the public and take on the private models built by large tech companies.
Orca is based on Vicuna, another instruction-tuned model, but it surpasses it by 100% on complex zero-shot reasoning benchmarks, like Big-Bench Hard. According to Microsoft’s research paper, Orca not only performs well on these benchmarks but also holds its ground against OpenAI’s ChatGPT in BBH benchmarks, despite its smaller size. Additionally, Orca displays academic prowess in competitive exams like LSAT, GRE, and GMAT, both in zero-shot settings without CoT, although it trails behind GPT-4.
Orca has the capability to learn through step-by-step explanations from both human experts and other LFMs to improve its capabilities and skills. According to Microsoft’s research team, Orca learns while getting rid of the formidable challenges posed by large-scale data handling and task variety, which benefits companies and users alike who want a tailored and optimized AI model.
