Professor Bryan Caplan recently made an astonishing discovery – the AI language model ChatGPT had improved its score from a disastrous D-grade on an economics test of his to an A-grade in just three months. Caplan, who works in George Mason University in the department of economics, remembered he had given his fall midterms to the chatbot in January; a test which tested students’ comprehension of economics rather than their memorization of text book facts.
The old version of ChatGPT didn’t play the part very well, achieving only 31 out of a 100, when the median was 50. Professor Caplan noted that the machine’s responses were more political than economic and it had difficulty comprehending fundamentals such as comparative and absolute advantages. Writing his assessment on his monthly blog post, Caplan quipped that ‘ChatGPT does a fine job of imitating a very weak GMU econ student’.
This wasn’t the only academic that ChatGPT underperformed; when pitted against a Wharton Business School exam, it ‘passed’ but made ‘surprising mistakes’ regarding basic calculations. Caplan’s response to all this was to place a bet on the AI model, with a bet that it would not score an A-grade on six out of seven exams before 2029.
He was proven wrong when ChatGPT-4, the paid-for update of GPT-3.5, made a staggering appearance, scoring 73%, equivalent to an A and surpassing many of his students. This was because it had addressed the fundamental issues of GPT-3: it was now 40% more likely to return accurate replies, could better comprehend complex instructions, and had an awareness of principles championed by well-known economists, such as Paul Krugman.
This was a huge turn around for the professor, who was used to winning all of his bets, with many of them revolving around technical topics such as unemployment rate readings. The professor is now more confident in another AI-related wager he created with Eliezer Yudkowsky. It is a bet that claims AI won’t lead to the end of the world before 2030. He said that he’s “on board to do a bunch more end of the world AI bets” because he believes ‘these people are out of their minds’.
But the professor has more worries than just betting. AI Bots such as ChatGPT have caused headaches for examiners, as it is difficult to determine if answers are plagiarized as there is no visible evidence of wrong-doing. For this reason, Caplan is considering getting rid of graded homework in his economics course, instead relying on regularly changing exams to ward off students turning to AI for answers.
ChatGPT is an AI language model developed by Sam Altman that has made strides in its performance since its launch in January 2021. With its upgraded version ChatGPT-4, its accuracy and efficiency in understanding and answering questions related to economics has seen a great improvement, surprising even professor Bryan Caplan. It is yet to be seen what the implications of this bot are, as it can pose a challenge to traditional education as well as challenge professor Caplan’s betting prowess.
