Technology
Tests show standard LRU caching resists alternatives in LLM prefix workloads
Hacker News reports that experimental eviction policies failed to outperform basic LRU methods in prefix-cache simulations.
The short version
- A simulation testing alternative eviction algorithms against production Least Recently Used baselines found that none improved cache hit performance.[Hacker News]
- Tests showed recomputation was driven primarily by rapid tool loops seconds apart rather than sessions exceeding expiration timers.[Hacker News]
- In ablation testing, each tested algorithmic variation deteriorated cache performance compared to basic LRU.[Hacker News]
- A systematic gap of 4 to 6 percentage points when reproducing published baseline data remains unexplained.[Hacker News]
Key facts
- An investigator tested three distinct policy designs on a prefix-cache simulator and failed to surpass the baseline LRU performance.[Hacker News]
- Every evaluated replacement component degraded performance, according to ablation tests.[Hacker News]
- Under memory capacity limits, rapid tool-calling loops occurring seconds apart drove most recomputation, whereas time-to-live expirations never triggered.[Hacker News]
- Requests arriving within 10 seconds made up 33.1 percent of recomputations, while requests after a five-minute gap accounted for 17.5 percent.[Hacker News]
- Flat block LRU and radix-leaf-restricted LRU differed by only 0.02 percentage points on the tested workload.[Hacker News]
- A software bug initially allowed non-recency policies to evict currently constructing prefixes, making Belady's algorithm underperform LRU until identified.[Hacker News]
What remains uncertain
- The tester identified an unexplained systematic offset of 4 to 6 percentage points when reproducing published Mooncake hit-rate-vs-capacity data.[Hacker News]
Sources
Outlet counts describe coverage, not independent confirmation. Reports may share a wire service or original source.