New Study Finds AI Chatbot ChatGPT’s Japanese Texts Easily Distinguishable from Human Writing
A recent study has revealed that ChatGPT, a generative artificial intelligence (AI) chatbot, may initially appear to be fluent in Japanese, but its Japanese texts are still 100 percent distinguishable from those written by humans. Researchers have drawn upon stylistic features commonly used in criminal investigations to explore this issue further.
The study, which was published in a journal on August 9th, addresses growing concerns about the potential misuse of ChatGPT for tasks such as writing university essays, academic papers, and other reports. While previous research has shown that ChatGPT-generated English-language research papers can be accurately distinguished from human-written ones, no similar study had been conducted for Japanese texts.
To investigate this matter, the study was undertaken by Wataru Zaitsu, an associate professor of criminal psychology at Mejiro University in Tokyo, and Jin Mingzhe, a specially appointed professor of data science at Kyoto University of Advanced Science. The pair embarked on a comparison between 72 existing Japanese-language academic papers on psychology and 144 texts generated using two versions of ChatGPT. Both sets of texts were written under the same titles as the original articles.
During their analysis, the researchers primarily focused on stylistic features, such as the positioning of commas and the sequence of parts-of-speech. The results indicated that ChatGPT-generated texts showed a higher frequency of using a comma following the postpositional particle wa and the prefix hon, translating to the present. Moreover, an AI classifier could accurately differentiate between ChatGPT-generated and human-written Japanese texts based on these distinctive features with an accuracy rate of 100 percent.
The study also investigated whether GPT-4, the latest version of the chatbot, maintained these distinguishable traits. The researchers confirmed that the results remained consistent with the previous findings, even with the more powerful text generation capabilities of GPT-4.
Zaitsu, who previously served as a senior researcher with the criminal investigation laboratory of the Toyama prefectural police, highlighted that the analysis technique used in this study draws on methodologies employed in criminal investigations to authenticate the authorship of notes, letters, and other writings. He noted that although ChatGPT-generated Japanese texts may appear natural to the human eye, it is relatively easy to distinguish them from human-written texts by relying on data. Zaitsu emphasized that this process is simpler than differentiating between texts written by different individuals.
Acknowledging that other text-generative AI chatbots produce Japanese texts that are closer to those written by humans, Zaitsu expressed his intention to continue studying methods for distinguishing between them.
The research paper, which contains detailed information on the study, has been published in the Plos One science journal. The findings shed light on the need to remain cautious when utilizing AI chatbots like ChatGPT for educational or academic purposes, especially in instances where the authenticity of the written content is crucial.
In conclusion, the study has demonstrated that ChatGPT’s Japanese texts can be easily identified as AI-generated due to specific stylistic features. The implications of these findings go beyond simple differentiation, as they raise important considerations about the potential misuse of AI in various fields. As technology continues to advance, it is crucial to strike a balance between leveraging AI capabilities and maintaining the integrity of human-authored content.
