A recent academic paper underlines the experimentation with recently released ChatGPT version 4.0, which took the Certified Public Accountant (CPA) Exam after the first failure and passed comfortably. This was a major breakthrough for the Accounting Today experiment which had used ChatGPT 3.5 for the first test. To assess the capabilities of GPT 4.0, the researchers conducted a “zero-shot” scenario, wherein the bot relied solely on its pre-existing knowledge and understanding to generate sufficient results. ChatGPT scored an average of 67.8%, an equivalent of acing the CPA Exam without studying for it.
The academics then tested a “10-shot” scenario where they provided the AI with 10 sample accounting questions and made slight changes to the settings, eliminating randomness and reducing creativity. Although this test resulted in an average score of 74.4%, this still wasn’t enough to pass the CPA Exam. Finally, the “chain of thought” prompting technique was employed to prime ChatGPT. This technique requires decomposing a difficult task into several intermediate steps to get the final response. The AI, relying on its ability to remember things in the conversation and apply them to its replies, was primed with 10 accounting questions. Elaborate chain of thoughts prompting, backed with prior training, gave ChatGPT sufficient boost to pass the CPA Exam with an average score of 84.3%.
The study’s authors, ChatGPT founder and CEO David A. Wood and his team of researchers, further expanded their exploration to include exams of Certified Management Accountants (achieving an average of 86.6%), Certified Internal Auditors (at 85.5%) and Enrolled Agents (at 83.8%). In response to the success of these results, the CEO said to Accounting Today that the breakthrough has shaken the domain of accounting, questioning the “competitive advantage” of the human accountant compared to machines. Additionally, Wood expresses hope that the newfound capabilities of AI will free accountants to innovate and add greater value to their field and to the society.
