Two gates, near-identical goodput at 47.8 and 48.0 rps. The downstream peak differs: 290 rps against 100 rps.
Written by
Distributed systems, cloud-native architecture, and the JVM. mostly shipping, occasionally reading.
Production System Labs - Series 2, Post 3. Runnable Java experiments on backpressure and load control. Deterministic outputs, checked-in CSVs, reproducible on any machine.
Post 2 chose how much work to admit. This post asks where a burst should wait.
The deterministic model in ShapingScenario.java puts a token bucket and a leaky bucket in front of the same single server used in Posts 1 and 2. Service time is fixed at 10 ms, server capacity is 100 rps, and the client deadline is 200 ms. Both gates have a sustained rate of 100 rps. The demand curve repeats a 20 rps valley for 1000 ms and a 600 rps spike for 200 ms, producing 113.2 rps over the five-second arrival window.
The fresh run reproduced bp-post3-burst-sweep.csv:
limiter burst goodput_rps reject_pct served_late_pct gate_delay_p99 server_wait_p99 downstream_peak
token-bucket 1 31.2 72.4 0.0 0.0 0.0 100.0
token-bucket 20 47.8 57.8 0.0 0.0 190.0 290.0
token-bucket 80 25.4 15.4 62.2 0.0 790.0 610.0
leaky-bucket 1 31.2 72.4 0.0 0.0 0.0 100.0
leaky-bucket 20 48.0 57.6 0.0 190.0 0.0 100.0
leaky-bucket 80 25.4 15.2 62.4 790.0 0.0 100.0
The full sweep covers burst dimensions 1, 2, 4, 8, 12, 16, 20, 28, 40, 80. Goodput stays near-identical between the gates at every measured point. At dimension 20, the difference is 0.2 rps; at dimensions 1, 8, 28, 40, and 80, the reported values are equal. Goodput alone does not expose the design difference.
The goodput_rps column is on-time completions divided by the five-second arrival window. It is normalized model output, not measured closed-loop throughput. The model has no network, scheduler contention, backpressure propagation, or production traffic variance.
The useful columns say where the wait lands and what rate reaches the server.
Dimension 20 shows the split. The token bucket records 190 ms p99 wait at the server and none at the gate. The leaky bucket records 190 ms at the gate and none at the server. The token bucket reaches a 290 rps downstream peak; the leaky bucket stays at 100 rps.
The token bucket banks credit during quiet periods. An arrival with a token passes immediately, so the gate adds no delay and the burst reaches the server. TokenBucketGate.java contains the whole decision:
@Override
public OptionalDouble offer(double arrivalMs) {
refill(arrivalMs);
if (tokens < 1.0) {
return OptionalDouble.empty();
}
tokens -= 1.0;
return OptionalDouble.of(arrivalMs);
}
private void refill(double nowMs) {
tokens = Math.min(burstCapacity, tokens + (nowMs - lastRefillMs) * refillPerMs);
lastRefillMs = nowMs;
}
The leaky bucket schedules admitted arrivals one leak interval apart. The queue absorbs the burst before it reaches the server. LeakyBucketGate.java returns that scheduled release time:
@Override
public OptionalDouble offer(double arrivalMs) {
if (pendingAhead(arrivalMs) >= queueCapacity) {
return OptionalDouble.empty();
}
double releaseMs = Math.max(arrivalMs, nextReleaseMs);
nextReleaseMs = releaseMs + leakIntervalMs;
return OptionalDouble.of(releaseMs);
}
The release time is the distinction. One gate returns the arrival time. The other returns a place in a paced schedule.
The second CSV counts offered and released requests in aligned 100 ms windows. The first spike is enough to see the result:
window_start_ms offered_rps token_bucket_rps leaky_bucket_rps
900 20.0 20.0 20.0
1000 600.0 290.0 100.0
1100 600.0 100.0 100.0
1200 20.0 20.0 100.0
1300 20.0 20.0 100.0
1400 20.0 20.0 50.0
1500 20.0 20.0 20.0
The offered spike begins at 1000 ms. The token bucket sends 290 rps in that window, then returns to the gate rate. The leaky bucket holds its output at 100 rps through the spike and for two full windows after offered load has fallen to 20 rps; the 1400 ms window carries the final 50 rps of the drain.
Policing transmits the burst; shaping smears it into the future.
The model's deadline budget is capacity x deadline = 100 x 0.2 = 20. A bucket size of 20 spends that budget at the server. A leaky queue depth of 20 spends it at the gate.
At dimension 20, both p99 waits are 190 ms, just inside the 200 ms client deadline, and neither gate serves work late. At dimension 28, p99 wait rises to 270 ms; served-late reaches 17.8% for token and 18.4% for leaky. At dimension 80, p99 reaches 790 ms, served-late exceeds 62%, and both gates fall to 25.4 rps goodput.
Dimension 1 produces the same strict gate in this run: 31.2 rps goodput, 72.4% rejection, no late service, and a 100 rps downstream peak. Dimension 20 raises goodput to 47.8 and 48.0 rps by carrying more of each spike without crossing the deadline. The cost is location. Token-bucket wait appears in the server queue; leaky-bucket wait appears before release.
The best result here remains far below Post 2's 99.8 rps. A fixed rate gate does not use current server occupancy as its admission signal. The token bucket can bank only its configured credit, while the leaky bucket can hold only its configured queue. The lower goodput is the price this model pays for bounding the rate delivered downstream.
These buckets are not competing implementations of one idea. They answer one question: who absorbs the burst?
A token bucket fits a downstream that can absorb short bursts: a service you operate with enough headroom, a tier that scales with load, or an API contract that permits bursts while bounding the average. Immediate release preserves the input burst, and any resulting queue forms after the gate. Callers still experience that server wait as end-to-end latency. At dimension 20 in this run, the downstream peak is 290 rps.
A leaky bucket fits a downstream that must never see a burst: a rate-contracted third-party API, a fragile legacy system, or a disk that degrades past a known IOPS ceiling. Its flat output holds the burst at the gate, where callers pay the added delay. The queue continues draining after the offered spike has ended. At dimension 20, downstream output never exceeds 100 rps.
The deadline gives this model a starting dimension, not a production constant. Real limits need measured service-time variance, downstream capacity, traffic shape, and one end-to-end deadline boundary. Goodput cannot choose between these gates when their values are this close; wait location and downstream peak can.
The next post changes the question from where work waits to which work should be abandoned.
The fresh command was:
./gradlew :backpressure-playground:runTokenVsLeaky \
-Pargs="--deterministic --duration 5s --output-dir ./results/article-grounding/bp-series2-post3"
The run wrote two CSVs, two PNG charts, manifest.json, and report.html. Both generated CSVs matched their checked-in golden files byte for byte:
bp-post3-burst-sweep.csv d29c75e9eaa020ba6192686883d30a6366ab7640534847a20d55e339072987c6
bp-post3-shaping.csv 80841498be4d0e0a48076599602ed624b18d23e45dcbd6e880ebd580b7471543
The manifest records golden_match: true for both files. ShapingSimulatorTest.java passed seven tests covering the deadline budget, goodput proximity, wait location, downstream burst, the dimension sweet spot, oversized dimensions, and deterministic output. GoldenOutputTest.java generated both CSVs and compared them with the golden data line by line.
References for the rate-limiting concepts, separate from the generated lab numbers: