Technology
Linux Foundation IT director says AI web scrapers consume nearly all git.kernel.org traffic
Infrastructure administrators plan to restrict web features and limit crawlable URLs as automated scraping bogs down kernel repository servers.
The short version
- The Linux Foundation's IT infrastructure director reported that automated web scrapers account for roughly 98% of traffic to git.kernel.org, overwhelming server resources.
- Scrapers are utilizing residential and mobile IP proxies to bypass traditional blocks, dedicating 14 CPU cores across five nodes solely to rendering commits as HTML.
- To protect server availability, the organization will turn off select web features, limit crawlable URLs, and gate costly anonymous requests.
- The Linux Foundation confirmed that all kernel repository data will remain available for download, though users may face additional access hurdles.
Key facts
- The Linux Foundation's director of IT infrastructure stated that legitimate requests comprise roughly 2% of the traffic on git.kernel.org, with the remaining 98% coming from scrapers.[Slashdot]
- Git.kernel.org processes about 6 million daily requests to view random commits.[Slashdot]
- Across five geo-distributed nodes, 14 CPU cores are continuously dedicated to rendering git commits as HTML for scraping bots.[Slashdot]
- Automated scrapers have shifted from using data-center IP addresses to routing requests through millions of residential or mobile IPs via proxy SDKs.[Slashdot]
- The Linux Foundation plans to disable certain web features and gate resource-intensive anonymous actions to mitigate server load, while maintaining data download access.[Slashdot]
What remains uncertain
- The exact identity of all organizations or entities operating the aggressive scraping bots remains unspecified.[Slashdot]