In a recent study conducted by a group of researchers from Brigham Young University (BYU) in the US and other 186 universities from 14 countries, the researchers found that the OpenAI chatbot product, ChatGPT, vied with the accounting exam taken by students, but it didn’t match up. Nevertheless, the researchers still said that ChatGPT’s overall performance in the exam was “impressive” and can be a “game changer” when it comes to teaching and learning.
The students who took the accounting test scored an average of 76.7%, while the ChatGPT, received a score of 47.4%. Though the AI bot fared better in true/false and multiple-choice questions, it had a harder time in tax, financial, and managerial assessments, as it struggled with the mathematical processes the latter type required. It was also found to often provide authoritative written descriptions for incorrect answers, or answer the same question different ways.
ChatGPT, which utilizes machine learning to generate natural language text, was also found to provide explanations for its answers, even though they were wrong, and it had a tendency to make up facts sometimes. For example, when giving a reference, it went ahead and created a real-looking yet completely fabricated source. The AI bot sometimes even made nonsensical mathematical errors.
David Wood of BYU’s accounting department, the lead author of the research, spread a call for participation on social media, recruiting 327 professors from 186 universities to see how their students would fare in comparison to the OpenAI chatbot. The resulting 25,181 student examinations and 2,268 textbook test bank questions covered data related to AIS, auditing, financial accounting, managerial accounting, and tax.
In discussing their results, researcher Wood explained that higher-order questions were harder for the AI to answer, though it still provided value contrary to popular beliefs. He also added to an ongoing debate about how models such as ChatGPT should be assessed in classrooms, with the team hoping that their work gives important insights into the use of AI technology in education.
