Code repositoryproduction-systems-labs
J
Written by
Jagdish Salgotra
Distributed systems, cloud-native architecture, and the JVM. mostly shipping, occasionally reading.
Was this article helpful? or email →anonymous · no account needed
A deterministic Java model where a fixed-capacity server loses goodput after offered load exceeds capacity. At 110 rps against 100 rps capacity, goodput falls to 42 rps and retries increase normalized attempt load without rescuing it.
Written by
Distributed systems, cloud-native architecture, and the JVM. mostly shipping, occasionally reading.
Production System Labs - Series 2, Post 1. Runnable Java experiments on backpressure and load control. Deterministic outputs, checked-in CSVs, reproducible on any machine.
A service that can finish 100 requests per second receives 110. The comfortable intuition says it serves 100 and the overflow suffers: a ten percent problem, degraded but mostly fine. The lab says otherwise.
The numbers below are deterministic synthetic model output, not production measurements. CollapseScenario.java sweeps offered load from a quarter of capacity to three times capacity against a single server with a fixed 10 ms service time, an effectively unbounded queue, a 200 ms client deadline, and no admission control. Capacity in this model is 100 rps. Goodput counts only requests completed within the deadline.
The fresh 5s deterministic run produced the same CSV checked into golden/bp-post1:
mode offered_rps goodput_rps wasted_pct p50_ms p99_ms avg_queue_depth
no-retry 100 100.0 0.0 10.0 10.0 0.00
no-retry 110 42.0 61.8 259.0 505.0 25.41
no-retry 125 19.2 84.6 634.0 1246.0 62.80
no-retry 150 11.6 92.3 1257.0 2483.0 125.17
no-retry 200 7.8 96.1 2505.0 4955.0 250.00
no-retry 300 5.8 98.1 5003.0 9903.0 500.00
At 100 rps the system is perfect. At 110 rps, ten percent past capacity, goodput is not 100; it is 42. At double capacity it is 7.8. The service time never moved from 10 ms, but at that point only 3.9% of the offered requests complete before the deadline. The curve past capacity is not a plateau at the limit. It is a cliff.
We once spent an incident bridge staring at a dashboard where every server was at full CPU, the request rate looked survivable, and the success rate kept falling anyway; nobody could explain where the capacity was going.
The capacity goes to requests whose clients have already given up. With no admission control, every arrival joins the queue, and under sustained overload the queue only grows. Wait time climbs past the 200 ms deadline, the client times out and walks away, and then the server gets to that request and serves it anyway, burning a full 10 ms slot to produce an answer nobody is waiting for.
The service-then-discard mechanism lives in CollapseSimulator.java:
double serviceStartMs = Math.max(arrival.timeMs(), serverBusyUntilMs);
double finishMs = serviceStartMs + serviceTimeMs;
serverBusyUntilMs = finishMs;
long sojournMs = Math.max(serviceTimeMs, Math.round(finishMs - arrival.timeMs()));
sojournHistogram.recordValue(Math.min(sojournMs, MAX_LATENCY_MS));
if (sojournMs <= clientDeadlineMs) {
goodput++;
} else {
wastedServed++;
double giveUpMs = arrival.timeMs() + clientDeadlineMs + retryBackoffMs;
scheduleRetry(events, seq, retriesEnabled, arrival, giveUpMs, durationMs);
}
The server advances its busy clock on every admitted request, in arrival order, including the doomed ones. That is the whole collapse. At 110 rps offered, the backlog grows by roughly ten requests a second. The run records 61.8% waste and an average queue depth of 25.41. Here, wasted_pct means the percentage of processed attempts that delivered nothing because their sojourn exceeded the deadline; it is not CPU utilization or a measured fraction of wall-clock capacity.
The same relationship becomes brutal at 300 rps: 98.1% of processed attempts deliver nothing, the median sojourn reaches 5003 ms for a 10 ms service operation, and goodput falls to 5.8 rps.
No GC pause, lock contention, slow downstream, or component failure exists in this model. An effectively unbounded queue plus a client deadline is enough. The exact values belong to these constants and this 5s window. The queue-and-deadline mechanism is the result that transfers.
The retry mode gives each timed-out request up to three retries with a fixed 50 ms backoff. The server, deadline, service time, offered-load sweep, and run window remain unchanged.
No jitter appears here because the model isolates retry amplification under a deterministic fixed schedule; jittered backoff is a mitigation outside this experiment.
mode offered_rps effective_rps goodput_rps wasted_pct p99_ms
retry 100 100.0 100.0 0.0 10.0
retry 110 281.2 42.0 85.1 8959.0
retry 150 520.4 11.6 97.8 20815.0
retry 200 716.6 7.8 98.9 30527.0
retry 300 1092.6 5.8 99.5 49151.0
The goodput_rps values match the no-retry mode at every offered-load level. The retries rescue nothing in this run. At 110 rps offered, p99 sojourn moves from 505 ms without retries to 8959 ms with them.
The effective_rps column needs a precise reading. It is the total number of first attempts and retries processed by the simulation, divided by the 5s original-arrival window; it is not measured wall-clock throughput. A value of 281.2 therefore means the model processed 1406 attempts generated by five seconds of 110 rps original demand. At 300 rps, the normalized attempt rate reaches 1092.6, or 3.64x the offered rate, while goodput remains 5.8 rps.
That is the retry storm represented by this experiment: deadline misses create more attempts, and those attempts join the same growing queue. Individual retries look reasonable in isolation. Their aggregate effect here is extra work, a larger queue, and no additional goodput.
The collapse in this model needs no injected failure. A fixed-capacity server, an unmanaged queue, a client deadline, and sustained offered load above service capacity are sufficient.
Provisioning headroom changes where the model crosses capacity. It does not change the queue-and-deadline mechanism after arrivals exceed service rate. Protection has to bound admitted work or bound the wait, not merely move the threshold.
Raw service time is also a reassuring number in the wrong place. The modeled operation remains fixed at 10 ms while no-retry goodput falls from 100.0 rps at capacity to 42.0 rps at 110 rps offered. Goodput exposes what the service-time constant cannot: whether completed work still reaches the client before the deadline.
Retries belong in every overload review. The focused test in CollapseSimulatorTest.java pins the behavior that matters here: retries increase effective load at 150 rps without changing goodput, while remaining inert below capacity because no timeout fires.
This post deliberately does not fix the failure. No token bucket or bounded admission policy appears in the scenario. Admission control, the subject of Post 2, changes the shape by rejecting work before it can become stale in the queue.
The command used for the fresh run was:
./gradlew :backpressure-playground:runLoadCollapse \
-Pargs="--deterministic --duration 5s --output-dir ./results/article-grounding/bp-series2-post1"
The run wrote bp-post1-collapse-sweep.csv, bp-post1-goodput.png, bp-post1-amplification.png, manifest.json, and report.html. The generated CSV and checked-in golden CSV have the same SHA-256:
965e6578eba2dbb33316af1fa12dc225f99b5b1d05ca6dee2572270b2ab84eeb
The run manifest reports golden_match: true. GoldenOutputTest.java independently runs the same deterministic Article 1 path and compares the generated CSV with the checked-in golden data line by line.
References for the overload and retry concepts, separate from the generated lab numbers: