Title: Most Top News Sites Block AI Bots, but Right-Wing Media Welcomes Them
In a recent study conducted by Ontario-based AI detection startup Originality AI, it was found that over 88 percent of the top news outlets in the United States are now blocking web crawlers used by artificial intelligence (AI) companies to collect training data. These web crawlers are crucial for AI projects, such as chatbots, as they gather essential information from news sites. Surprisingly, right-wing media differs significantly from their liberal counterparts, as they have chosen not to block these AI web scrapers.
The data collected in mid-January from approximately 40 top news sites revealed that major newspapers like The New York Times, The Washington Post, and The Guardian, as well as various magazines and special-interest sites, all block AI web crawlers. OpenAI’s GPTBot, one of the most prominent crawlers, is widely blocked. However, among the top right-wing news outlets surveyed, including Fox News, the Daily Caller, and Breitbart, none block these AI web scrapers, including Google’s AI data collection bot. Notably, Bari Weiss’ new website The Free Press also does not block AI scraping bots.
Researchers’ speculations regarding this discrepancy revolve around combatting perceived political bias. As Originality AI founder and CEO Jon Gillham suggests, AI models reflect the biases of their training data. By leaving their right-leaning content accessible to AI web crawlers, right-wing media may introduce alternative perspectives into AI algorithms that could counterbalance any perceived liberal bias in existing AI tools.
Originality AI identified these website preferences by analyzing the robots.txt files used by websites to inform web crawlers which pages are off-limits. The analysis focused on the top news sites in the US, and while only one of those sites held a significantly right-wing perspective, Originality AI also examined nine well-known right-leaning outlets. Surprisingly, none of the nine right-wing sites blocked GPTBot.
Conservative leaders in the US, as well as Elon Musk, have expressed concerns over perceived liberal biases in AI tools such as ChatGPT. At a recent hearing on AI, Senator Marsha Blackburn brought attention to an AI-generated poem praising President Biden, claiming that generating a similar ode to Trump was impossible using ChatGPT. Hence, right-leaning outlets might view the decision not to block AI web crawlers as an opportunity to balance biases in AI algorithms.
While some believe that including right-wing content in AI training data could impact model parameters, AI ethics researcher Jeremy Baum from UCLA remains skeptical that it would significantly influence the outputs of finished AI systems like chatbots. This skepticism stems from the vast amount of existing data already collected from mainstream news outlets and the tendency for AI companies to employ liberal-leaning individuals.
As the AI industry continues to grapple with the issue of bias, the differing strategies of news outlets in blocking or not blocking AI web crawlers stand as an intriguing aspect. It remains to be seen how this discrepancy will impact the development and deployment of AI tools in the future.
Overall, this findings demonstrate the complex relationship between news outlets and AI, highlighting the potential influence of political perspectives on the development of AI technology.
