A recent study has shown that while AI models like ChatGPT have the potential to boost productivity, they are not yet capable of replacing human programmers. The study, published in IEEE Transactions on Software Engineering, compared the performance of ChatGPT in generating code to that of human programmers, focusing on functionality, complexity, and security.
The research found that ChatGPT’s success rate in producing functional code varied widely depending on the task difficulty, programming language, and other factors. While the AI could sometimes match or even surpass human programmers, it also displayed significant limitations. The study highlighted the importance of understanding both the strengths and weaknesses of AI models like ChatGPT.
The study further tested ChatGPT’s ability to solve coding problems from the LeetCode platform across five programming languages. While the AI performed well on pre-2021 coding problems, its success rate dropped significantly when faced with newer challenges introduced after 2021. This suggests that ChatGPT struggles with unfamiliar, more recent coding problems due to its training data not including these challenges.
Yutian Tang, a lecturer at the University of Glasgow involved in the study, emphasized the need for ongoing development and training for AI models like ChatGPT to keep up with the evolving field of software engineering. While AI can enhance productivity and automate certain coding tasks, it is not yet a substitute for human programmers.
The study’s findings underscore the importance of continuous improvement and training for AI models like ChatGPT to effectively address the ever-changing landscape of software engineering. While AI can excel in solving familiar problems, it still lacks the critical thinking skills of human programmers when faced with new and complex challenges.
