HTTP Caching Proxy

A mini CDN: an HTTP caching reverse proxy in C on a Raspberry Pi 5. Requests hit an in-memory cache — a hash table (djb2, separate chaining) for O(1) lookup and an LRU list for O(1) eviction — and misses are forwarded to an nginx origin, cached, and relayed. The point was never just to build one, though. It was to build it four different ways, benchmark them rigorously against each other, and understand why they differ.

First the cache itself: a hit (in-memory lookup + memcpy) served ~29× the throughput of a miss (a full origin round-trip). Then eviction policy — I implemented LFU alongside LRU and compared hit rates across workloads. LFU wins under stable popularity, but under drifting popularity it collapses to ~9.5% versus LRU's ~75%: old frequency counts become permanent baggage that can't be evicted while newly-hot items never build up. That's why production caches use adaptive policies like ARC and LRU-K rather than committing to either extreme.

Then the core of the project: four concurrency architectures sharing that cache — single-threaded, thread-per-connection, a fixed thread pool with a job queue, and a single-threaded epoll event loop (the way nginx works, driving both client and origin sockets through one state machine without ever blocking). Each was a response to what the previous one's benchmarks revealed.

Halfway through, an experienced reviewer flagged that my benchmarks were flawed — so I rebuilt the methodology and re-measured. The load generator had been running on the same 4-core Pi as the proxy (contaminating every result); the origin was Python's http.server (so miss numbers measured Python's ceiling, not my proxy); and connections closed after every request (so a TCP handshake dominated each measurement). I moved load generation off-box over verified Ethernet, swapped in nginx, and added HTTP keep-alive. The tell that the original numbers were wrong: they violated Little's Law by 3–14×. The corrected numbers reconcile — at 50 connections and 62,304 req/s, Little's Law predicts 802µs latency and I measured 793µs.

The corrected results reframed everything: there is no single 'fastest' architecture. Thread-per-connection had the highest peak throughput; the thread pool scaled the flattest and most predictably; and epoll had by far the best tail latency (1.57ms p99 at 50 connections versus tenths of a second for the others) and zero socket errors — which is exactly why production servers use an event loop. 'Fastest' turned out to be the wrong question: peak throughput, predictable scaling, and tail latency each point at a different design, and you only see it by measuring the distribution under load, not the headline number. I documented the correctness gaps honestly too — no TTL, no request coalescing, epoll origin keep-alive left as future work — because a caching proxy has requirements beyond raw speed.