According to research done by Kaspersky, ChatGPT’s effectiveness in detecting malicious phishing links was limited. The accuracy was tested using more than 2000 phishing links that Kaspersky’s anti-phishing software had identified with mixed results. When prompted with “Does this link lead to a phishing website?” and “Is this link safe to visit?”, ChatGPT provided a detection rate of 87.2% and false positive rate of up 23.2%. For the second question, it had a higher detection rate of 93.8% but a false positive rate of 64.3%. These results show that while ChatGPT had the potential to detect phishing links, its accuracy is still too low to be applied in any real-world scenario.
In addition to detection, the experiment also looked into the extent of understanding that ChatGPT had on phishing and its ability to recognize the target. The model recognized the target of a phishing attack in more than 50% of the links it tested. This included brands such as Facebook, TikTok, Google, Amazon and Steam, and multiple banks from around the globe. However, it was found that the security explanations provided by it were often wrong and misleading.
Vladislav Tushkanov, Lead Data Scientist at Kaspersky, commented on ChatGPT as a possible assistant for analysts in fighting phishing attacks. Tushkanov said that language models have their limitations and may not be ready to revolutionize the cybersecurity landscape yet. However, they could still serve as helpful tools for experts.
Kaspersky, founded in 1997 and headquartered in Moscow, is a leading global cybersecurity company that protects users and organizations worldwide with endpoint security, enterprise security, public Wi-Fi security and encryption products. Its anti-phishing, anti-malware, anti-spam and other malware protection products help users maintain online security and protect their data.
Vladislav Tushkanov, Kaspersky’s Lead Data Scientist, is a seasoned machine learning expert with extensive experience in research, development, and productizing of ML algorithms. He focuses on advancing both Kaspersky and the industry by using cutting-edge technologies to develop new artificial intelligence applications for security.
