Researchers from Brigham Young University (BYU) and 186 other universities recently studied how OpenAI’s ChatGPT product would fare on accounting exams. Their findings, published in the journal Issues in Accounting Education, revealed that despite the artificial intelligence (AI) technology having provided impressive performance, students had done better than ChatGPT, scoring an overall average of 76.7 per cent, compared to the ChatGPT’s score of 47.4 per cent.
The researchers have considered ChatGPT’s performance as “impressive”. It is a remarkable breakthrough which could revolutionize how teaching and learning are done in the academia. The AI product uses machine learning to generate natural language text, and was found to do better on true/false questions (68.7 per cent correct) and multiple-choice questions (59.5 per cent); however it was seen to struggle with short-answer questions (between 28.7 and 39.1 per cent). It was further seen to perform more poorly on tax, financial and managerial assessments, which could possibly be attributed to ChatGPT’s difficulty with the required mathematical processes.
The variables of the examination included accounting information systems (AIS), auditing, financial accounting, managerial accounting, and tax. Questions varied in difficulty and type, from true/false, multiple choice, to short answer questions.
Lead study author David Wood, a BYU professor of accounting, wanted to include as many professors as possible to find out how the AI would fare against university students in an accounting exam; hence he recruited 327 co-authors from 186 educational institutions in 14 countries. The researchers fed 25,181 classroom accounting exam questions and 2,268 textbook test bank questions to ChatGPT.
Interestingly, ChatGPT was found to provide authoritative written explanations for incorrect answers. Moreover, it was seen to sometimes make up facts. It provided real-looking fabricated references where the work and authors did not exist, as well as showing nonsensical mathematical errors.
OpenAI is an artificial intelligence company co-founded by Elon Musk and Sam Altman in 2015. The company is focused on making “safe artificial general intelligence” and its latest product, ChatGPT, is a game-changer. It uses natural language processing (NLP) technology to generate text.
David Wood is a professor of accounting at BYU and lead author of the study. Wood is hoping that the results of the study will add to the ongoing debate on the role of models such as ChatGPT in education. He has begun recruiting more professors for the research, aiming to make it even more comprehensive.
