Google has announced the launch of two new web crawlers designed specifically for scraping image and video content for research and development purposes. The search giant revealed details of these specialized crawlers, emphasizing that websites can block them without affecting their rankings.
The new crawlers are variants of Google’s existing GoogleOther crawler, which was originally introduced in April 2023. These latest versions are optimized for crawling binary data, specifically images and videos. Websites can utilize user agent tokens in their robots.txt file to block these new crawlers if desired.
Google has clarified that the data scraped by these crawlers is not intended for AI training purposes. Instead, the purpose of these crawlers is to assist Google product teams in conducting research and development activities. The original GoogleOther crawler, known for its versatility in retrieving publicly accessible content, has been repurposed for these specific image and video crawling tasks.
To identify the new crawlers in server logs and differentiate them from other web crawlers, Google has updated the user agent strings for both the new variants and the regular GoogleOther crawler. By referencing these user agent strings, website owners can easily identify and block the new crawlers if necessary.
For publishers concerned about their images and videos being scraped for research and development, understanding and utilizing the new user agent strings can be valuable. Identifying these genuine Google crawlers enables publishers to manage their content accordingly and decide whether to opt out of having their media assets crawled by these specialized bots.
In a move towards transparency and user control, Google’s introduction of these new web crawlers aims to provide publishers with the ability to make informed decisions regarding their website content. With the option to block these crawlers, website owners can exercise control over their digital assets while contributing to Google’s research and development efforts.
