Technology

AI scrapers consume major resource share at git.kernel.org

The open-source repository's administrator reports that bots use an estimated 20% of its server capacity by repeatedly bypassing security challenges.

Updated Aug 30, 2026 · 14:35 UTC1 reports

The short version

AI scrapers and training crawlers generate an estimated 98% of the traffic on git.kernel.org, while legitimate user requests make up only about 2%.
The scrapers bypass traditional, efficient cloning methods and instead generate billions of URLs to scrape individual commits and patches via cgit, consuming roughly 20% of the site's total server capacity.
Git.kernel.org plans to turn off features and gate anonymous access to reduce crawlable URLs while keeping repository data available for download.

Across five geo-distributed nodes with 90 total CPU cores, 14 to 16 cores are constantly dedicated to rendering git commits as HTML for AI scrapers.[Hacker News]
Scrapers generate billions of URLs by rendering commits as HTML instead of directly cloning the repositories, which would be much more efficient.[Hacker News]
Basic IP and ASN bans proved insufficient as bots transitioned to using vast proxy networks of residential and mobile IPs via proxy SDK monetization.[Hacker News]
Proof-of-work math challenges (Anubis) initially deterred crawlers, but the bots have adapted to solve higher-difficulty challenges to access the site.[Hacker News]
Currently, about 33% of the roughly 6 million daily commit requests successfully solve the Anubis math challenge to access git.kernel.org.[Hacker News]

It is impossible to determine with absolute certainty which individual requests solving the math challenges are automated bots and which are human users, though most requesting old commits in inactive forks are suspected to be bots.[Hacker News]