The accuracy of AI models when analyzing SEC filings has been revealed to be only 79%, highlighting the challenges faced by these models and the need for improvement. According to a report by CNBC, the study focused on large language models (LLMs) commonly used in analyzing these filings. Even the best-performing AI model configuration tested achieved only 79% accuracy in answering questions when provided with the complete filing and relevant question.
Researchers identified several issues with the AI models, including their tendency to refuse to answer questions or provide incorrect information not present in the SEC filings. This lack of accuracy and reliability raises concerns, especially in regulated industries, where precision is of utmost importance.
The finance industry highly values the ability to quickly extract important data and analyze financial narratives. If AI models could accurately summarize SEC filings or promptly answer related questions, users in the competitive finance sector could gain a significant advantage.
However, the entry of AI models into the industry has not been without challenges, as reported by Patronus AI. One of the major challenges is the nondeterministic nature of LLMs, meaning they do not consistently produce the same output for the same input. This necessitates rigorous testing to ensure accurate and reliable results. Patronus AI aims to address this challenge by automating LLM testing with software.
Despite the challenges and limitations identified in the study, the co-founders of Patronus AI remain optimistic about the potential of LLMs to assist professionals in the finance industry. They believe that with continued improvement, these models can provide valuable support to analysts and investors. However, human involvement is currently necessary to ensure accuracy and reliability.
The findings of this study underscore the need for ongoing advancements in AI models to meet the demands of regulated industries like finance. While LLMs have already made significant progress in analyzing SEC filings, their accuracy and reliability must continue to improve to provide optimal support to users in the sector.
