Anthropic accused of aggressive data scraping
AI company Anthropic was accused of aggressive and excessive data scraping practices by several website publishers. Website publishers, including Freelancer.com and iFixit, accused Anthropic of "egregious" data scraping. The company's ClaudeBot (initially named GPTBot) is primarily used to scrape data to train it's large language models (LLMs). Freelancer.com reported 3.5 million visits from Anthropic's web crawler in just four hours, significantly impacting site performance and revenue. And ClaudeBot reportedly visited technology advice site iFixit.com a million or so times over a 24-hour period, gobbling its content, driving its IT team to distraction, and reportedly costing it over USD 5,000 in bandwidth charges. The scraping activities reportedly violated the terms of use of these and other websites - which explicitly prohibit the reproduction, copying, or distribution of content without prior permission, including using content for training AI models. Anthropic responded by saying that it aims not to be intrusive or disruptive, but gid not provide detailed comments on the broader accusation. It also published details of its web scraping activities on its website. Blocking web scraping activities is challenging, as AI developers can bypass blocks by launching new crawlers with different names. It transpired that organisations blocking Anthropic focused on two AI scraper bots called “ANTHROPIC-AI” and “CLAUDE-WEB” and had been unaware of ClaudeBot. System 🤖 ClaudeBot Documents Anthropic. Does Anthropic crawl data from the web, and how can site owners block the crawler? Operator: Anthropic Developer: Anthropic Country: USA Sector: Multiple Purpose: Scrape data Technology: Bot/intelligent agent Issue: Cheating/plagi arism; Copyright; Transparency
- Date it happened
- 2024-07-01
- Organisation involved
- Anthropic
- Product, system or model
- ClaudeBot
This incident was imported from AIAAIC and is used under CC BY-SA 4.0. Our additions to it — the structured fields, the translation, the checks against other reports — are published under the same licence.
This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.