A recent study published by Issues in Accounting Education has found that students fare better than OpenAI’s ChatGPT chatbot product in accounting exams. Despite this, researchers have commended the “impressive” performance of ChatGPT, noting that it could be a game changer when it comes to teaching and learning.
The study conducted by researchers from Brigham Young University (BYU) and 186 other universities involved 25,181 classroom accounting exam questions. Additionally, 2,268 textbook test bank questions were asked to ChatGPT. Results showed that on average, students scored 76.7% while ChatGPT achieved a score of 47.4%.
The chatbot was found to be particularly skilled in answering accounting information systems (AIS) and auditing questions, but struggled with the mathematical processes needed for tax, financial, and managerial assessments. In addition, it did relatively well in true/false (68.7%) and multiple choice (59.5%) questions, but not in short answer questions (28.7-39.1%). It was also observed that ChatGPT sometimes fabricated references, did mathematical errors and answered the same questions in different ways, demonstrating that higher-order questions were harder for it.
Still, researchers consider this as a breakthrough technology in terms of teaching and learning. This is due to its ability to explain its answers, even if they were wrong, as well as its knack for providing natural language texts.
OpenAI is a San Francisco-based artificial intelligence research laboratory which is co-chaired by Elon Musk. It aims to advance digital intelligence in a way that is beneficial to humanity. It has done considerable work in the field of robotics, natural language processing and artificial general intelligence.
David Wood is the lead study author at BYU and a professor of accounting. He and 327 other authors from 186 different educational institutions collaborated on this project. Wood himself initiated the research with a recruiting pitch on social media and it quickly gained traction.
The findings of the study shed light on the ongoing debate on how artificial intelligence like ChatGPT must be incorporated in education. The performance clearly demonstrated that there could be much to gain from utilizing AI in this area, but with certain precautions and limitations.
