Three shedding policies, the same 100 rps goodput at twice capacity. Unbounded FIFO falls to 7.8 rps.
Written by
Distributed systems, cloud-native architecture, and the JVM. mostly shipping, occasionally reading.
Production System Labs - Series 2, Post 4. Runnable Java experiments on backpressure and load control. Deterministic outputs, checked-in CSVs, reproducible on any machine.
Posts 2 and 3 rejected the newest arrival when a gate was full. This post asks which work should survive when the server cannot serve everything.
SheddingScenario.java runs four policies against a synthetic single server with fixed 10 ms service, 100 rps capacity, and a 200 ms client deadline. fifo uses an unbounded oldest-first queue with no active shedding. tail-drop rejects at the door when total in-system occupancy reaches 20. expire keeps FIFO order but discards work at dequeue when it can no longer finish before the deadline. lifo serves the newest queued request first.
The fresh run reproduced the 200 rps rows in bp-post4-shed-sweep.csv:
policy offered_rps goodput_rps shed_pct served_late_pct p99_served_ms shed_wait_p50_ms wasted_pct
fifo 200.0 7.8 50.0 46.1 2481.0 1250.0 92.2
tail-drop 200.0 100.0 50.0 0.0 200.0 0.0 0.0
expire 200.0 100.0 50.0 0.0 200.0 195.0 0.0
lifo 200.0 100.0 50.0 0.0 10.0 2495.0 0.0
The three explicit shedding policies deliver 100.0 rps goodput and shed 50.0%. FIFO falls to 7.8 rps goodput and spends 92.2% of consumed service on requests completed after their deadlines.
FIFO also reports 50.0% shed, but it never rejects or expires a request during the run. ShedSimulator.java stops starting work when the five-second window closes and counts everything left in the queue as shed. That common accounting keeps goodput, served-late, and shed shares tied to the same arrival set.
The goodput_rps column is on-time completions divided by the five-second arrival window. It is normalized deterministic model output, not measured closed-loop throughput.
The policies cannot be ranked by goodput or shed share in this row. Served latency and shed wait separate them.
We once ran a system where the long-suffering requests were the ones that got served; the queue guaranteed it. The fresher a request was, the more certain it was to be shed when the cap kicked in. Nobody had chosen that. The data structure had.
Tail-drop and expire produce the same aggregate goodput and p99 served latency in this run. Their shed paths differ.
Tail-drop records 0 ms median shed wait because it rejects at admission. Its bound is capacity x deadline = 100 x 0.2 = 20, but the implementation counts total in-system occupancy, queued plus in service. An admitted request therefore has at most 19 service slots ahead and can finish within 200 ms in this fixed-service model.
Expire records 195 ms median shed wait at 200 rps. Its dequeue check includes the next 10 ms service slot:
double arrivalMs = policy.pickNewest() ? queue.pollLast() : queue.pollFirst();
double waitMs = serverFreeMs - arrivalMs;
if (policy.dequeueExpiry() && waitMs + serviceTimeMs > clientDeadlineMs) {
outcome.shedWaits.recordValue(clamp(waitMs));
outcome.shed++;
continue;
}
The deadline supplies the expiry threshold, so this policy has no separate queue bound. The cost is late information: an expired request waits for most of its budget before the model discards it.
LIFO changes the pick order. At overloaded sweep points from 125.0 through 300.2 rps, its p99 served latency stays between 10 and 17 ms. The newest work gets the next service slot while older work sinks in the stack. At 200 rps, LIFO's median shed wait is 2495 ms. That value comes from old requests still queued when the five-second model window closes. The experiment does not model connection holding or a client-side timeout.
ShedPolicy.java expresses the policies as pick order, door bound, and dequeue expiry. In the simulator, pollLast versus pollFirst decides whether new or old work gets the next slot.
The second experiment exposes recovery after a burst. Its demand curve repeats 80 rps valleys for 1000 ms and 600 rps spikes for 500 ms. The CSV records p99 sojourn for requests completed in each aligned 100 ms window.
The first spike begins at 1000 ms and ends at 1500 ms:
window_start_ms fifo_p99_ms tail_drop_p99_ms expire_p99_ms lifo_p99_ms
900 10.0 10.0 10.0 10.0
1000 77.0 77.0 77.0 12.0
1200 243.0 200.0 200.0 12.0
1500 493.0 200.0 200.0 73.0
1800 743.0 175.0 175.0 385.0
2100 993.0 100.0 100.0 697.0
2400 1243.0 25.0 25.0 1008.0
2500 1327.0 77.0 77.0 13.0
FIFO p99 climbs from 77 ms at the start of the spike to 1243 ms nine hundred milliseconds after it ends. The queue must serve stale work before reaching fresh arrivals. The second spike arrives at 2500 ms with FIFO already at 1327 ms; the final window still reports 2461 ms. That is the burst hangover: the incident outlives the load that caused it.
Tail-drop and expire coincide in this time series. Their p99 reaches the 200 ms deadline, then falls to 25 ms by the last window before the next spike. Bounded or expired work prevents one spike from becoming permanent backlog.
LIFO reports 12 ms during the first spike because it serves fresh arrivals. The valley exposes the other side: p99 rises through 73, 385, 697, and 1008 ms as sparse new arrivals leave service slots for old work buried during the spike. The next spike makes p99 fresh again at 13 ms. Fresh inside the burst, archaeology in the lull.
Pure LIFO needs an expiry rule if old work must never consume service after its deadline. Ben Maurer's Fail at Scale describes Facebook combining adaptive LIFO with CoDel: newest-first service under overload, with controlled dropping based on queue delay. The connection is a production reference, not a behavior modeled by this lab.
An unbounded FIFO still makes a choice. It serves the oldest request, retains every newer one, and keeps spending service after deadlines. At 200 rps, that choice produces 7.8 rps goodput and 92.2% wasted service in this model.
Tail-drop gives fast rejection and needs a measured occupancy bound. Expire uses the deadline as its discard threshold but reports failure late. LIFO preserves fresh service during overload but strands old work until load falls or the run ends. A production policy can combine mechanisms; this experiment separates them so each cost remains visible.
Goodput answers whether capacity produced useful completions. P99 served latency answers what successful callers experienced. Shed wait answers how long rejected work consumed the caller's budget before the decision arrived. All three belong in a shedding review. At 200 rps, the controlled policies deliver the same 100.0 rps goodput while p99 served latency is 200, 200, and 10 ms. Which one you want is a product decision, not an infrastructure one.
The next post adds request value. A health check and a checkout should not necessarily have the same survival probability.
The fresh command was:
./gradlew :backpressure-playground:runLoadShedding \
-Pargs="--deterministic --duration 5s --output-dir ./results/article-grounding/bp-series2-post4"
The run wrote two CSVs, two PNG charts, manifest.json, and report.html. Both generated CSVs matched their checked-in golden files byte for byte:
bp-post4-shed-sweep.csv bc2ba213029757e0f7a86fd9b940ad35d7d787184b69d563c22b8ca0029e9609
bp-post4-hangover.csv 2a2941d8f543cef2b563fab8ec83af554d4ab8378bc459c01ca89b2c9654d775
The manifest records golden_match: true for both files. ShedSimulatorTest.java passed seven tests covering the deadline bound, overload goodput, served freshness, expiry, shed wait, burst hangover, and deterministic output. GoldenOutputTest.java generated both CSVs and compared them with the golden data line by line.
References for the shedding and queue-control concepts, separate from the generated lab numbers: