Priority admission keeps 99.7% critical success at 300 rps; blind shedding delivers 32.3%. Background pays with 12.9%.
Written by
Distributed systems, cloud-native architecture, and the JVM. mostly shipping, occasionally reading.
Production System Labs - Series 2, Post 5. Runnable Java experiments on backpressure and load control. Deterministic outputs, checked-in CSVs, reproducible on any machine.
Post 2 bounded how much work enters. Post 3 chose how a rate gate delivers it. Post 4 chose which queued work to abandon. One decision remains: whose work survives.
SloControlScenario.java assembles a synthetic bounded system with fixed 10 ms service, 100 rps capacity, a 200 ms client deadline, an in-system bound of 20, and dequeue expiry. A fixed-seed classifier marks 25% of arrivals critical and the rest background.
Two door policies see the same arrivals. blind rejects any arrival when the system is full. priority lets a critical arrival evict the newest queued background request; rejection follows only when no queued background request remains.
The fresh run reproduced bp-post5-protection.csv:
policy offered_rps critical_rps crit_ok% bg_ok% crit_p99_ms slo_met
blind 150.2 37.1 70.8 68.7 200.0 no
blind 200.2 49.4 51.1 52.3 200.0 no
blind 300.0 74.2 32.3 35.4 200.0 no
blind 400.2 99.8 25.3 26.3 200.0 no
blind 500.2 124.0 20.0 21.1 200.0 no
priority 150.2 37.1 100.0 59.1 200.0 yes
priority 200.2 49.4 100.0 36.3 200.0 yes
priority 300.0 74.2 99.7 12.9 200.0 yes
priority 400.2 99.8 93.3 3.5 200.0 no
priority 500.2 124.0 80.8 1.1 200.0 no
The rates use a 4.8 s scoring window, not the full five-second run. Arrivals after duration - deadline lack a complete 200 ms outcome window and are excluded. Values such as 200.2 rps and total goodput near 104 rps are normalized scoring-window rates, not measured closed-loop throughput.
The 200 ms p99 column is the first warning. Survivor latency does not distinguish the policies under overload. Critical success does.
The bound caps admitted sojourn at the deadline, and dequeue expiry discards work that can no longer finish in time. Every served request therefore completes within 200 ms in this model. The protection sweep reports critical p99 of 200 ms for both policies at every overloaded point.
Critical success tells a different story. At 400.2 rps, blind completes 25.3% of critical arrivals within deadline; priority completes 93.3%. A latency panel built only from completed requests sees the same 200 ms p99 and misses the rejected, evicted, or expired work.
The lab's actual SLO is a success-rate objective: at least 99% of scored critical arrivals must complete within 200 ms. That objective includes failures by construction.
We once watched a war room declare a degraded service healthy because every latency panel was green; the checkouts that never happened were not on any panel.
Blind shedding moves both classes together. At 200.2 rps, critical success is 51.1% and background success is 52.3%. At 300.0 rps, the values are 32.3% and 35.4%. A class-unaware door makes critical work fail at roughly the same rate as background work.
Priority changes who occupies the bounded queue. SloControlSimulator.java contains the admission rule:
private void admit(ArrayDeque<Integer> queue, int index, boolean serverBusy,
ClassPolicy policy, int queueBound, Outcome outcome) {
if (queue.size() + (serverBusy ? 1 : 0) < queueBound) {
queue.addLast(index);
return;
}
if (policy == ClassPolicy.PRIORITY && outcome.critical[index]) {
Iterator<Integer> newestFirst = queue.descendingIterator();
while (newestFirst.hasNext()) {
if (!outcome.critical[newestFirst.next()]) {
newestFirst.remove();
queue.addLast(index);
return;
}
}
}
}
No adaptive controller appears here. Static criticality changes the eviction target.
The nominal feasibility boundary is capacity / critical share = 100 / 0.25 = 400 rps. Below that boundary, critical traffic can fit in the server in aggregate. At 300.0 rps, the run offers 74.2 rps critical and priority delivers 99.7% critical success, enough for the 99% SLO.
The boundary is not a promise that 400 rps will pass. At 400.2 rps, realized critical load is 99.8 rps, leaving almost no room for timing variation or background work already in service; critical success falls to 93.3%. At 500.2 rps, critical load alone is 124.0 rps, above physical capacity. The measured 80.8% success is close to the arithmetic upper ratio 100 / 124 = 80.6%.
Background pays for protection. Its success rate falls from 59.1% at 150.2 rps to 12.9% at 300.0, 3.5% at 400.2, and 1.1% at 500.2. That cost is the policy, not a side effect. Background means work the system is allowed to lose first.
The second experiment repeats 80 rps valleys for 1000 ms and 360 rps spikes for 500 ms. Nominal critical spike load is 90 rps, below the server's 100 rps capacity, so protection is feasible. The CSV scores success by 100 ms arrival window through the 4.8 s cutoff.
The first spike shows the split:
window_start_ms blind_critical_pct priority_critical_pct blind_background_pct priority_background_pct
900 100.0 100.0 100.0 100.0
1000 84.6 100.0 78.3 69.6
1100 0.0 100.0 32.3 16.1
1200 33.3 100.0 25.9 0.0
1300 18.2 100.0 32.0 0.0
1400 50.0 100.0 26.5 23.5
1500 100.0 100.0 100.0 100.0
Blind critical success falls to 0.0% in the 1100 ms window. Priority critical success remains 100.0% in every scored window of the fresh CSV. Background absorbs the burst: priority background success reaches 0.0% at 1200 and 1300 ms, then returns to 100.0% when the valley begins at 1500 ms.
The burst artifact contains success rates only. It does not report per-window latency, so no latency claim is inferred from this chart.
The bounded architecture turns overload into explicit decisions. The occupancy bound limits waiting. Expiry prevents doomed service. Policing or shaping chooses where bursts land. Shedding chooses which work leaves. Criticality decides whose success rate the system protects.
Capacity planning needs the feasibility boundary. Below capacity / critical share, priority can protect critical work by transferring failures to background. Near the boundary, deterministic timing already matters. Above it, critical demand or critical capacity must change because background sacrifice cannot create service slots.
The organizational requirement is sharper than the code. A critical class exists only when another class is genuinely allowed to fail. The 1.1% background success rate at 500.2 rps is what that agreement looks like under this model. Without that agreement, every request is critical and priority has nothing to evict.
The fresh command was:
./gradlew :backpressure-playground:runSloLoadControl \
-Pargs="--deterministic --duration 5s --output-dir ./results/article-grounding/bp-series2-post5"
The run wrote two CSVs, two PNG charts, manifest.json, and report.html. Both generated CSVs matched their checked-in golden files byte for byte:
bp-post5-protection.csv 203c11498ab8bb7c7ed9fdba300e977b5d8864341382af6ec9db495cac108ecc
bp-post5-slo-burst.csv 15ae9d057e9bc224882a461eef3a80cdccaa668a13204b0e0f16f92b5cdcd3be
The manifest records golden_match: true for both files. SloControlSimulatorTest.java passed nine tests covering bound and ceiling arithmetic, deadline-flat served latency, blind class degradation, priority protection below the ceiling, failure at the ceiling, background cost, burst protection, deterministic output, and invalid short windows. GoldenOutputTest.java generated both CSVs and compared them with the golden data line by line.
References for criticality and overload control, separate from the generated lab numbers: