A deterministic sweep: goodput peaks at an admission limit of capacity times deadline, then collapses eight slots later.
Written by
Distributed systems, cloud-native architecture, and the JVM. mostly shipping, occasionally reading.
Production System Labs - Series 2, Post 2. Runnable Java experiments on backpressure and load control. Deterministic outputs, checked-in CSVs, reproducible on any machine.
Post 1 ended with a server collapsed to 7.8 rps of goodput at twice its 100 rps capacity, with most service slots spent on requests whose clients had already given up. The cure is unsettling the first time you deploy it: refuse work, immediately, at the door, while the server still has capacity to spare. A concurrency limit admits a new request only if fewer than limit requests are already in the system, and fail-fast rejects the rest.
The numbers below are deterministic synthetic model output, not production measurements. AdmissionScenario.java sweeps the limit itself against the same fixed 10 ms service time and 200 ms client deadline as Post 1. The demand curve alternates a 200 ms valley at 50 rps with a 100 ms spike at 300 rps. Across this 5s run, the generated arrival count produces an offered rate of 131.4 rps.
The fresh run reproduced bp-post2-limit-sweep.csv:
admission_limit goodput_rps reject_pct served_late_pct p99_ms utilization_pct
1 60.2 54.2 0.0 10.0 60.2
8 88.4 32.7 0.0 80.0 88.4
12 98.2 25.3 0.0 120.0 98.2
16 99.0 24.7 0.0 160.0 99.0
20 99.8 24.0 0.0 200.0 99.8
28 29.2 22.8 54.9 280.0 100.0
40 12.0 21.0 69.9 400.0 100.0
none 12.0 0.0 90.9 1750.0 100.0
Goodput climbs from 60.2 rps at limit 1 to 99.8 rps at limit 20, then falls to 29.2 rps at 28 and 12.0 rps at 40. Eight additional in-flight slots past the best point lose 70.6 rps of goodput even though reported utilization reaches 100%.
We once inherited a service whose thread pool had been resized upward in every incident retro for two years, because more threads always felt like more capacity; nobody could say what number it should have been, only that the last number had not been enough.
The sweep's shape is the design lesson. The two failure directions are not symmetric.
A tight limit pays on the gentle side. Limit 1 rejects 54.2% of arrivals and reports 60.2% utilization because the gate turns away spike traffic that the following valley could have absorbed. The p99 remains 10 ms, and no arrival is served late.
A loose limit pays on the cliff side. Limit 28 admits requests with up to 27 others ahead of them. With fixed 10 ms service, that permits a 280 ms sojourn, not a 280 ms queue wait. The result exceeds the 200 ms deadline: 54.9% of arrivals are served late while utilization reports 100%. Utilization cannot distinguish useful work from late work; goodput can.
The boundary comes from the same arithmetic as Little's Law. Capacity is 100 rps, and the deadline is 0.2 s, so the model's in-flight bound is 100 x 0.2 = 20. AdmissionSimulator.java computes that value directly:
public int littlesLawLimit() {
return (int) Math.round(serverCapacityRps() * (clientDeadlineMs / 1000.0));
}
The limit is a promise about total time in the system. At limit 20, the measured p99 is exactly 200 ms, the deadline itself, and served_late_pct remains 0.0.
The sweet spot depends on the demand shape. DemandCurve.java uses repeating valleys and spikes so extra admission capacity can carry work from a spike into the following valley. Under flat permanent overload, that slack has no valley to exploit, and a tighter limit rejects sooner. The experiment isolates that trade-off; it does not claim every production arrival pattern has the same optimum.
The mechanism is small enough to read in full. The gate counts how many admitted requests remain ahead of an arrival, then rejects or lets the FIFO server take it:
for (double arrivalMs : arrivals) {
long occupancyAhead = occupancyAhead(serverBusyUntilMs, arrivalMs);
if (occupancyAhead >= admissionLimit) {
rejected++;
continue;
}
double serviceStartMs = Math.max(arrivalMs, serverBusyUntilMs);
double finishMs = serviceStartMs + serviceTimeMs;
serverBusyUntilMs = finishMs;
long sojournMs = Math.max(serviceTimeMs, Math.round(finishMs - arrivalMs));
sojournHistogram.recordValue(Math.min(sojournMs, MAX_LATENCY_MS));
if (sojournMs <= clientDeadlineMs) {
goodput++;
} else {
servedLate++;
}
}
The early continue is the difference from Post 1's unmanaged queue. With the limit fixed at 20, the second experiment replays the offered-load sweep with no control and with admission control. The fresh run reproduced bp-post2-plateau.csv:
mode offered_rps goodput_rps reject_pct p99_ms
no-control 125.0 19.2 0.0 1246.0
no-control 200.0 7.8 0.0 4955.0
no-control 300.2 5.8 0.0 9911.0
admission-limited 125.0 103.8 17.0 200.0
admission-limited 150.0 103.8 30.8 200.0
admission-limited 200.0 103.8 48.1 200.0
admission-limited 300.2 103.8 65.4 200.0
The cliff is a plateau again. At the highest generated load, 300.2 rps, the limited mode reports 103.8 rps goodput and 200 ms p99 while rejecting 65.4% of arrivals. The no-control mode reports 5.8 rps goodput and 9911 ms p99.
The 103.8 rps value is not measured wall-clock throughput above a 100 rps server. The simulator divides on-time completions by the 5s arrival window and finishes admitted work after the last arrival, so end-of-window draining can lift the normalized result above nominal capacity. The 300.2 rps offered value comes from 1501 deterministic arrivals divided by the same 5s window.
The no-control rows use the same service-then-discard mechanism as Post 1 and match its shared exact offered-load points. The final point is not identical: this generator emits 300.2 rps, while Post 1 uses exactly 300 rps.
AdmissionSimulatorTest.java pins the important behavior: limit 20 beats tight and loose limits, tight admission underutilizes the server, no control produces served-late work, and admission control restores more than five times the no-control goodput at the nominal 300 rps level.
Admission control makes the system decide what gets in before the queue decides how long everyone waits.
The first input is measured service capacity. The second is the client deadline. Their product gives this model a starting in-flight bound. Real services still need validation against service-time variance, downstream limits, traffic shape, and the exact boundary used for both measurements.
The asymmetry in this run favors rounding down when evidence is incomplete. Limit 16 gives up 0.8 rps relative to limit 20; limit 28 gives up 70.6 rps. That is a result from this curve, not a universal percentage, but it shows why a larger limit is not automatically safer.
Rejections are a first-class outcome. At 300.2 rps offered, the limited model rejects 65.4% while holding p99 at the deadline and normalized goodput near capacity. Rejection here is the protection working. Sustainable responses still require more capacity or less demand.
This post bounded how much gets in. The next one is about delivery: two gates that admit the same average rate but hand it to the server in different shapes, the token bucket and the leaky bucket.
The command used for the fresh run was:
./gradlew :backpressure-playground:runAdmissionControl \
-Pargs="--deterministic --duration 5s --output-dir ./results/article-grounding/bp-series2-post2"
The run wrote both CSVs, both PNG charts, manifest.json, and report.html. The generated CSVs matched their checked-in golden files byte for byte. Their SHA-256 values were:
bp-post2-limit-sweep.csv 35cc8107e84c0e60b0a7e2cd2d6476ef720e6892f7874583d81cd10864e05493
bp-post2-plateau.csv b0fd5a5518c669cc96a11c01f1026d3797f57e21a507d9927263557d20d3a238
The manifest reports golden_match: true for both artifacts. GoldenOutputTest.java independently runs the deterministic Article 2 path and compares both generated CSVs with the checked-in golden data line by line.
References for the admission-control and queueing concepts, separate from the generated lab numbers: