Researchers from Nanyang Technological University, Singapore (NTU Singapore) have successfully compromised several artificial intelligence (AI) chatbots, including ChatGPT, Google Bard, and Microsoft Bing Chat, leading to a phenomenon known as jailbreaking. This term refers to the act of exploiting vulnerabilities in a system’s software to make it perform actions that developers intentionally restricted.
By training a large language model (LLM) on a database of previously successful chatbot hacks, the researchers developed an LLM chatbot capable of automatically generating prompts to jailbreak other chatbots. LLMs serve as the intelligence behind AI chatbots, enabling them to process human inputs and generate human-like text. This includes tasks such as planning a trip, telling stories, and even creating computer code. Now, jailbreaking can be added to their list of capabilities.
The implications of this research are significant for companies and businesses that rely on LLM chatbots. By exposing vulnerabilities and limitations, the researchers’ findings can help strengthen these chatbots against potential hackers.
To prove the efficacy of their technique, the researchers conducted a series of tests, which involved successfully jailbreaking LLMs and promptly reporting the issues to the respective service providers.
Professor Liu Yang, who led the study at NTU’s School of Computer Science and Engineering, explained the significance of the research. He stated, The developers of such AI services have guardrails in place to prevent AI from generating violent, unethical, or criminal content. But AI can be outwitted, and now we have used AI against its own kind to ‘jailbreak’ LLMs into producing such content.
The researchers named their two-fold jailbreaking method Masterkey. Firstly, they reverse-engineered how LLMs detect and defend against malicious queries, allowing them to develop prompts that bypass these defenses. The researchers also trained an LLM to automate the generation of jailbreak prompts, even after developers patch their systems.
The researchers’ paper has been accepted for presentation at the Network and Distributed System Security Symposium, a renowned security forum, in San Diego in February 2024.
AI chatbots receive prompts from human users, and developers establish guidelines to prevent the generation of unethical or illegal content. However, the researchers discovered ways to engineer prompts that evade ethical guidelines by using techniques like creating a persona that provides prompts in a manner undetectable by keyword censors.
The researchers were able to infer the inner workings and defenses of LLMs by manually entering these prompts and observing the response times. Subsequently, they reverse-engineered the hidden defense mechanisms of the LLMs. This allowed them to create a dataset of prompts that successfully jailbroke the chatbots.
The Masterkey method developed by the researchers proved to be three times more effective than existing methods. Additionally, it can continuously generate new and improved prompts, learning from past successes and failures.
The researchers emphasize that their technique can assist developers in strengthening the security of their LLM chatbots. They highlight the need for comprehensive automated approaches to generating jailbreak prompts that cover a wide range of possible misuse scenarios.
As LLMs continue to evolve, manual testing becomes both labor-intensive and inadequate. The researchers believe that approaches like Masterkey can address this challenge and provide comprehensive coverage of potential vulnerabilities.
This research serves as a reminder that despite their benefits, AI chatbots are vulnerable to jailbreak attacks. The ongoing back-and-forth between hackers and developers necessitates constant vigilance and proactive measures to ensure the integrity and security of AI systems.
In conclusion, the NTU researchers have demonstrated the ability to jailbreak AI chatbots by training an LLM to generate prompts that exploit vulnerabilities. The implications of this research can help companies enhance the security of their chatbot systems. However, it also highlights the need for continuous efforts to stay one step ahead of potential hackers in the evolving landscape of AI technology.
