A deterministic Java SLO simulation against a 99%-under-200ms objective. A timeout caps p99 at 200ms yet scores the same 57.36% SLI as doing nothing, because a fast failure is still a failure; a bulkhead holds 99.60% with 60% of the error budget left. Burn rate fires from a single window.
Written by
Distributed systems, cloud-native architecture, and the JVM. mostly shipping, occasionally reading.
Production System Labs - Series 1, Post 6 Runnable Java experiments on the failure patterns that show up under real load. Deterministic outputs, checked-in CSVs, reproducible on any machine.
Everything in this series so far has produced a number. Post 1 showed the tail hiding behind the average. Post 2 showed latency exploding past saturation. Post 3 spent capacity to cut the tail, Post 4 made sure the tail was measured honestly, and Post 5 chose where overload pain lands. None of that is a decision yet. It is just data.
A service-level objective is what turns the data into a decision you can defend at three in the morning. The lab states one objective, 99% of requests must complete under 200 ms, and SloScenario.java runs four architectures against the same overload to see which ones keep the promise. The outcomes are in golden/post6/post6-slo-summary.csv:
pattern total_requests good_requests bad_requests achieved_sli_pct worst_burn_rate final_budget_remaining_pct alert_triggered
baseline 5000 2868 2132 57.36 100.00 0.00 true
bulkhead 5000 4980 20 99.60 1.00 60.00 false
timeout-budget 5000 2868 2132 57.36 100.00 0.00 true
concurrency-limit 5000 4960 40 99.20 2.00 20.00 true
The request rate is deterministic in this experiment: concurrency * 100, so the run command at the end of this post produces 1000 rps for 5s, or 5000 requests per pattern.
I still remember insisting a service was basically fine during an incident because CPU and error rate looked normal, right up until someone asked what fraction of requests had actually finished under our latency target, and not one of us had that number.
The most instructive row is timeout-budget, because it looks like a fix and is not one.
The intuition is reasonable: requests are blowing past 200 ms, so cap them at 200 ms with a timeout. And it works, in the narrow sense that its per-second p99 in golden/post6/post6-slo-windows.csv never exceeds 200.0, while the unprotected baseline climbs to 444 ms and 644 ms during the overload:
pattern elapsed_s good_requests bad_requests p99_ms burn_rate
baseline 3 0 1000 444.0 100.00
baseline 4 0 1000 644.0 100.00
timeout-budget 3 0 1000 200.0 100.00
timeout-budget 4 0 1000 200.0 100.00
Look at the SLI, though. The timeout-budget pattern achieves the same 57.36% as the baseline, with the same 2132 bad requests. The timeout bounded the latency number and changed nothing about the promise. A request that would have taken 400 ms now returns at 200 ms, but it returns as a failure, and the SLI counts a deadline miss as bad whether it comes back slow or comes back fast-but-broken. The user who needed that answer did not get it either way.
That is the trap with latency mechanics in isolation. Bounding p99 is not the same as meeting an SLO. The SLO is defined on good requests, and in this lab a request is good only when it both succeeds and lands under the threshold. The check lives in SloScenario.java:
if (outcome.success() && outcome.latencyMs() <= target.latencyThresholdMs()) {
good++;
} else {
bad++;
}
A timeout that fires moves a request from the slow-success column to the failure column. It protects the latency graph and abandons the objective. The bulkhead and concurrency-limit patterns, which actually cap the overload the requests experience rather than just truncating the wait, keep the SLI at 99.60% and 99.20%.
You do not need thirty days of data to know you are in trouble. You need the burn rate, and the burn rate is computable from a single window.
The objective of 99% under 200 ms defines an error budget of 1%. SloTarget.java supplies that target, and burn rate is just how fast the current request stream is spending that budget relative to the rate that would exactly exhaust it over the SLO window:
public static double burnRate(long badEvents, long totalEvents, SloTarget target) {
if (totalEvents == 0L) {
return 0.0;
}
double badRatio = badEvents / (double) totalEvents;
return badRatio / target.errorBudgetRatio();
}
BurnRateCalculator.java is intentionally that small. A burn rate of 1.0 means you are spending budget exactly as fast as the SLO allows. The baseline's second second shows a 5.2% bad ratio against a 1% budget, which is a burn rate of 5.2x, and that is enough to act on immediately:
pattern elapsed_s bad_event_pct burn_rate budget_remaining_pct alerting
baseline 1 0.00 0.00 100.00 false
baseline 2 5.20 5.20 0.00 true
baseline 3 100.00 100.00 0.00 true
By the end of the second second the baseline has already consumed its entire window budget and the burn rate hits 100x once every request is bad. The alert does not wait for a month of accounting; the instantaneous burn rate fires the moment the stream is unsustainable. The protective patterns keep their burn rate low enough to survive: the bulkhead's worst burn rate over the whole run is 1.0, and it ends with 60% of its error budget still in hand.
The patterns differ in how they protect the budget, and the trade-offs map straight back onto Post 5. The bulkhead isolates the overloaded slice so failure in one place does not consume the budget everywhere, and it spends almost nothing. The concurrency limit caps in-flight work, which keeps the SLI at 99.20% but does shed a thin slice of requests and trips the alert briefly at a 2.0 burn rate before recovering, ending with 20% budget remaining. Both keep the promise. The difference is how much margin they leave for the next surprise.
An SLO is the contract that makes every earlier measurement actionable; without it you have graphs, with it you have a budget you can defend.
The SLI has to describe what the user experiences, not what is easy to graph. In this lab, good means succeeded and under 200 ms. CPU, heap, and error rate can all look normal while the SLI sits at 57.36%, which is exactly the gap that makes incidents feel inexplicable.
Bounding latency is not meeting the objective. The timeout-budget pattern held p99 at 200 ms and still failed 2132 requests, the same as doing nothing, because a fast failure is still a failure against the budget. Burn rate belongs in the same conversation, computed per window, because a 5.2x burn rate in the second second was enough to know the baseline was lost. A month of data is not required to see an unsustainable stream.
Architecture should protect the budget, not decorate the graph. The bulkhead and concurrency limit kept the SLI above 99%; the difference between them was the 60% versus 20% budget they left for the next event.
The whole series collapses into this one contract. Tail measurement from Post 1, honest load generation from Post 4, queueing pressure from Post 2, variance reduction from Post 3, and overload placement from Post 5 all become one number you are willing to be paged on.
The point of the budget is not to hit 100%. It is to know, at any moment, how much room you have left before the promise breaks, and to have already decided what you will give up to keep it.
The command that regenerates the numbers used here is:
./gradlew :latency-lab:runSloPolicy \
-Pargs="--deterministic --duration 5s --concurrency 10 --output-dir ./results/slo-policy --snapshot-interval 1s --slo-target p99<200ms"
The generated artifacts used in this post are:
golden/post6/post6-slo-summary.csv
golden/post6/post6-slo-windows.csv
golden/post6/post6-burn-rate.png
golden/post6/post6-error-budget.png
GoldenOutputTest.java runs this deterministic Article 6 path and compares post6-slo-summary.csv and post6-slo-windows.csv against the checked-in golden files. A fresh run for this post wrote results/article-grounding/post6/manifest.json; both Article 6 CSV artifacts had golden_match: true, with SHA-256 values matching their golden SHA-256 values.
References used for the underlying SLO ideas, not for the generated lab numbers: