WF-ZDAWBF1 May 2024
AI crawler causes $5,000 bandwidth costs for Read the Docs
In May 2024, an AI crawler downloaded 73 TB of zipped HTML files from Read the Docs, a documentation hosting platform, costing over $5,000 in bandwidth charges. The crawler repeatedly downloaded the same files due to a bug and lack of rate limiting. Read the Docs blocked the crawler and the company reimbursed the costs. Another crawler from Facebook downloaded 10 TB in June 2024, but Facebook did not respond.
- Company involved
- Read the Docs
- AI system involved
- AI crawler
10 source articles · read the reporting →