AI crawler causes $5,000 bandwidth costs for Read the Docs
In May 2024, an AI crawler downloaded 73 TB of zipped HTML files from Read the Docs, a documentation hosting platform, costing over $5,000 in bandwidth charges. The crawler repeatedly downloaded the same files due to a bug and lack of rate limiting. Read the Docs blocked the crawler and the company reimbursed the costs. Another crawler from Facebook downloaded 10 TB in June 2024, but Facebook did not respond.
- Date it happened
- 2024-05-01
- Organisation involved
- Read the Docs
- Product, system or model
- AI crawler
This incident was imported from about.readthedocs.com. Our additions to it — the structured fields, the translation, the checks against other reports — are published under the same licence.
This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.