A recent study revealed that students of accounting exams outperform OpenAI’s ChatGPT chatbot. Group of researchers from Brigham Young University (BYU) and 186 other universities conducted this study which was published in the journal Issues in Accounting Education. The results of the study showed that students scored an average of 76.7 percent in accounting exams, while ChatGPT scored only 47.4 percent.
The study further revealed that ChatGPT outperformed the student average in 11.3 percent of the questions in accounting information systems (AIS) and auditing. However, the bot’s performance was weaker in financial, managerial, and tax assessments. It is believed that the bot struggled to understand the mathematical calculations needed for these kind of questions.
Moreover, the study showed that ChatGPT performed much better on true/false questions (68.7% correct) and multiple-choice questions (59.5%), but fair poorly on short-answer questions (with correct answers ranging from 28.7% to 39.1%). The researchers also observed that the chatbot had trouble understanding complicated questions, and sometimes gave detailed answers to the wrong answers.
Furthermore, the chatbot also provided wrong information in some cases such as generating references for non-existent sources. Also, it made serious mathematical errors such as adding instead of subtracting or dividing numbers incorrectly.
To analyze the performance of ChatGPT compared to that of university accounting students, David Wood, a professor of accounting at BYU and the lead author of the study, made a social media pitch which was welcomed by 327 co-authors from 186 universities in 14 countries who offered 25,181 classroom accounting exam questions. Undergraduate BYU students were also involved in this research, providing the chatbot with 2,268 questions from various accounting domains of varying complexity and types, including AIS, auditing, financial accounting, managerial accounting, and tax.
OpenAI is an artificial intelligence research laboratory that is owned by the US company, Tesla. They are a world leader in AI research and have developed ChatGPT and other AI-based computer systems that can interact with humans in natural language. This technology will be ever more useful in improving the accuracy of knowledge testing and assessment from bots.
