Researchers from the University of Edinburgh and Allen Institute for AI are investigating if Large Language Models (LLMs) can self-improve, much like what AlphaGo Zero did by repeatedly engaging in competitive games with clearly laid out rules. The researchers believe that many LLMs may enhance one another in a negotiating game with little to no human interaction. This has far-reaching effects, as powerful agents may be built with few human annotations if the agents can progress independently. However, it also suggests powerful agents with little human supervision, which is problematic.
To explore this hypothesis, they invited two language models, a customer and a seller, to haggle over a purchase. The customer was asked to pay less for the product, while the seller was asked to sell it for a greater price. They then asked a third language model to take the role of the critic and provide comments to a player once a bargain had been reached. The researchers repeated the game and encouraged the player to refine their approach, utilising AI input from the critic LLM.
Their method, known as ICL-AIF (In-Context Learning from AI Feedback), leverages the AI critic’s comments and the prior dialogue history rounds as in-context demonstrations. This turns the player’s real development in the previous rounds and the critic’s ideas for changes into the few-shot cues for the subsequent round of bargaining. They use in-context learning, as fine-tuning large language models with reinforcement learning is prohibitively expensive.
They found that improving buyer role models can be more difficult than vendor role models, as trying to sell something for more money (or purchase something for less) runs the risk of not making a transaction at all. However, the model can engage in less verbose, but more deliberate (and ultimately more successful) bargaining.
The researchers anticipate that their work will be an important step towards enhancing language models’ bargaining in a gaming environment with AI feedback. The code for the experiment is available on GitHub.
Frequently Asked Questions (FAQs) Related to the Above News
What is the University of Edinburgh and Allen Institute for AI investigating in their research?
The researchers are investigating whether Large Language Models (LLMs) can self-improve, much like AlphaGo Zero did by repeatedly engaging in competitive games with clearly laid out rules.
How might LLMs enhance one another?
The researchers believe that many LLMs could enhance one another in a negotiating game with little to no human interaction.
What are the implications of LLMs being able to self-improve?
It could lead to the creation of powerful agents with few human annotations, but it also poses the problem of creating powerful agents with little human supervision.
How did the researchers test their hypothesis?
They invited two language models, a customer and a seller, to haggle over a purchase, and asked a third language model to take the role of the critic and provide comments to a player once a bargain had been reached.
What is the researchers' method known as?
Their method is known as ICL-AIF (In-Context Learning from AI Feedback).
What does ICL-AIF do?
It leverages the AI critic's comments and the prior dialogue history rounds as in-context demonstrations, turning the player's real development in the previous rounds and the critic's ideas for changes into the few-shot cues for the subsequent round of bargaining.
Why do the researchers use in-context learning?
Fine-tuning large language models with reinforcement learning is prohibitively expensive.
What did the researchers find about improving buyer role models?
They found that improving buyer role models can be more difficult than vendor role models, as trying to sell something for more money (or purchase something for less) runs the risk of not making a transaction at all. However, the model can engage in less verbose, but more deliberate (and ultimately more successful) bargaining.
What do the researchers anticipate their work will achieve?
The researchers anticipate that their work will be an important step towards enhancing language models' bargaining in a gaming environment with AI feedback.
Where can one access the code for the experiment?
The code for the experiment is available on GitHub.
Please note that the FAQs provided on this page are based on the news article published. While we strive to provide accurate and up-to-date information, it is always recommended to consult relevant authorities or professionals before making any decisions or taking action based on the FAQs or the news article.
