A recent study conducted by researchers at Salus Security delved into the capabilities of GPT-4, an advanced artificial intelligence model, in the realm of smart contract auditing. The findings shed light on the AI’s proficiency in generating and parsing code, particularly in detecting true positives within smart contracts.
The researchers utilized a dataset of 35 smart contracts, featuring a total of 732 vulnerabilities within the SolidiFI-benchmark vulnerability library. This allowed them to evaluate GPT-4’s ability to identify security weaknesses across seven common vulnerability types. Impressively, the AI demonstrated over 80% precision in detecting actual vulnerabilities worth investigating.
However, the study also highlighted a significant drawback of GPT-4 in generating false negatives. The recall rate of the AI, which stood at a mere 11%, indicates a notable insufficiency in vulnerability detection. The researchers concluded that while GPT-4 shows promise in certain areas, its overall accuracy in identifying vulnerabilities is limited to only 33%. Therefore, they advocate for the continued use of dedicated auditing tools and human expertise in auditing smart contracts until AI systems like GPT-4 can enhance their capabilities.
In conclusion, while AI such as GPT-4 shows potential in the realm of smart contract auditing, there remain significant limitations that necessitate the involvement of traditional auditing methods and human insight to ensure robust security measures are maintained in the blockchain space.
