In a recent study published in IEEE Transactions on Software Engineering, researchers delved into the capabilities of OpenAI’s ChatGPT when it comes to cracking coding problems post-2021. The findings revealed a decline in ChatGPT’s performance, possibly due to a lack of exposure to these newer problems during its training phase. This drop in proficiency was evident in the varied success rates of ChatGPT, ranging from a high of 89% to a low of 0.66%.
The research team conducted tests on ChatGPT’s coding abilities using 728 problems from the LeetCode platform across five programming languages. While ChatGPT excelled in solving problems predating 2021, its performance faltered with newer challenges. Yutian Tang, a lecturer at the University of Glasgow involved in the study, stressed the importance of understanding both the strengths and limitations of ChatGPT to enhance its coding capabilities.
Interestingly, the study revealed that ChatGPT generated code with lower runtime and memory overheads compared to at least half of human solutions. However, the AI struggled when it came to error correction, often producing incorrect code due to a lack of comprehension of algorithmic problems.
Moreover, security concerns were raised regarding vulnerabilities in AI-generated code, such as missing null tests. The complexity of the generated code was noted to be highest in C, followed by C++ and Python, with similarities to human-written code complexity.
To mitigate these issues, Tang suggested developers provide additional information and context to help ChatGPT better grasp problems and avoid vulnerabilities. By offering relevant knowledge and highlighting potential pitfalls, developers can guide ChatGPT towards producing more secure and accurate code.
The study emphasizes the need for a nuanced approach to leveraging AI in coding tasks, balancing its strengths with human oversight and guidance to ensure optimal results. As technology continues to evolve, understanding the capabilities and limitations of AI like ChatGPT will be crucial for harnessing its potential effectively in the realm of coding and software development.
