Series navigation
Code repositoryproject-loom
J
Written by
Jagdish Salgotra
Distributed systems, cloud-native architecture, and the JVM. mostly shipping, occasionally reading.
Was this article helpful? or email →anonymous · no account needed
Quorum aggregation, per-tenant bulkheads, and a deadline shape that survives a slow upstream. Cancellation comes first.
Written by
Distributed systems, cloud-native architecture, and the JVM. mostly shipping, occasionally reading.
Fan-out code gets confusing when the policy is only implied.
If three sibling tasks start at the same time, does the parent need all three results, the first successful result, the results that arrived before a deadline, or a degraded response after the primary path fails? Those are different contracts. They should not all look like "fork a few tasks and join."
This article is learning material. Structured concurrency is still preview, and the examples are here to make policy shape visible rather than to claim production behavior. The main branch now builds with OpenJDK 25.0.2 and uses the Java 25 preview structured-concurrency API, with the Java 21 version separately managed in the feature/java-21 branch. The measurements below were generated from the current Java 25 code. The Java 21 preview syntax shown here remains valid for learning purposes, and Part 9 covers the migration details.
For the Article 7 pass, the local toolchain was OpenJDK 25.0.2 and Maven 3.9.12, and the build command was:
mvn clean compile -DskipTests
The detailed reproduction steps for this article are in testing-and-benchmarking.md.
The build succeeded and compiled 35 source files.
Then I ran the standalone demos and the focused HTTP checks. These are the results that matter for this article:
The focused load checks were run one endpoint at a time:
The variance is the useful part of the table. /first-success and /async/race are fixed-delay examples, so their latency is tight. /cache/data and /data/with-fallback contain random misses and failures, so their average is a blend of different policy paths.
For /cache/data, a 12-request sequence returned L1 in 11-13ms most of the time and L2 in 55-56ms twice:
cache-01 L1-userdata Duration: 12ms status=200
cache-02 L1-userdata Duration: 13ms status=200
cache-03 L2-userdata Duration: 56ms status=200
cache-04 L1-userdata Duration: 13ms status=200
cache-12 L2-userdata Duration: 55ms status=200
For /data/with-fallback, the same kind of sequence showed both branches:
fallback-01 Secondary-userdata Duration: 252ms status=200
fallback-02 Primary-userdata Duration: 106ms status=200
fallback-09 Primary-userdata Duration: 101ms status=200
fallback-10 Secondary-userdata Duration: 260ms status=200
fallback-12 Secondary-userdata Duration: 261ms status=200
That is the real contract. Primary success is about 100ms. Fallback is about 250-260ms because the code waits for the primary to fail, then calls the secondary.
First success is useful when sibling tasks are allowed to answer the same logical question. Cache tiers, replicated reads, and equivalent read endpoints fit that shape. Non-idempotent writes do not.
In Java 21 preview syntax, the small teaching example looks like this:
static void runWithShutdownOnSuccess() throws Exception {
try (var scope = new StructuredTaskScope.ShutdownOnSuccess<String>()) {
scope.fork(() -> slowService("Service-A", 1000));
scope.fork(() -> slowService("Service-B", 500));
scope.fork(() -> slowService("Service-C", 200));
scope.join();
String result = scope.result();
logger.info("First successful result: {}", result);
}
}
The migrated Java 25 version in StructuredExampleWithSuccess.java uses StructuredTaskScope.open(...) with a joiner that stops when a subtask reaches SUCCESS. The output from the fresh run was:
Service-C completed on thread:
First successful result: Service-C result
That output is small, but it proves the policy. Service-C has the 200ms delay, Service-B has 500ms, and Service-A has 1000ms. The parent does not wait for all three.
The service version on port 8080 uses the same idea with cache names:
scope.fork(() -> slowService("Cache-1", 500));
scope.fork(() -> slowService("Cache-2", 200));
scope.fork(() -> slowService("Database", 800));
The fresh HTTP result was:
First successful result: Cache-2-OK-200ms (Duration: 207ms)
The clean service version in BusinessService.getCachedData() is more interesting because the cache tiers can miss:
return scopedHandler.runFirstSuccess(
() -> getFromL1Cache(key),
() -> getFromL2Cache(key),
() -> getFromDatabase(key)
);
The delays are 10ms for L1, 50ms for L2, and 200ms for the database. L1 randomly fails 30 percent of the time. L2 randomly fails 50 percent of the time. The database always succeeds.
That explains the benchmark shape. The average latency was 64.62ms, but the standard deviation was 65.33ms and the max was 289.04ms. This endpoint is not one operation with one latency. It is a policy with multiple visible paths.
First success should not hide source-level health. If L1 starts failing constantly, the caller may still get data from L2 or the database, but that does not mean the system is healthy. First-success code needs per-source counters because success at the parent boundary can mask damage underneath.
Partial results answer a different question. The caller is not asking for the first valid answer. The caller wants as much useful data as can arrive inside a budget.
The standalone demo in AdvancedStructuredPatterns.TimeoutWithPartialResults uses four tasks:
List<Callable<String>> tasks = List.of(
() -> { Thread.sleep(100); return "Quick result"; },
() -> { Thread.sleep(500); return "Medium result"; },
() -> { Thread.sleep(1000); return "Slow result"; },
() -> { Thread.sleep(2000); return "Very slow result"; }
);
The timeout is 600ms. The fresh run returned exactly what the delays predict:
Completed: 2/4 tasks
Results: [Quick result, Medium result]
Timed out: [2, 3]
That output is the contract. It does not pretend the slow and very slow sections succeeded. It tells the caller which indexes missed the budget.
The same class also has a progressive-results example. Four tasks with delays of 100ms, 200ms, 150ms, and 250ms completed in this order:
Task 0 completed: Result 1
Task 2 completed: Result 3
Task 1 completed: Result 2
Task 3 completed: Result 4
Progressive Results Summary:
- Completion rate: 100.0% (4/4 tasks)
- Total execution time: 272 ms
- Results: [Result 1, Result 2, Result 3, Result 4]
- Errors: 0
Two details matter. The progress callbacks arrive in completion order, but the final result list is returned in task order. A caller that streams updates needs to handle out-of-order arrival. A caller that renders the final response can still preserve stable section order.
This is why partial-results APIs should report both terminal progress and usable output. "A task stopped running" and "a task produced a value the caller can use" are not the same thing.
A hedged read is not a generic race. It is a delayed duplicate read used to cap tail latency when one logical source is occasionally slow.
The checked-in hedged read is in ConcurrentServiceLayer.hedgedReadWithDelay():
long hedgeAfterMillis = 30;
scope.fork(() -> simulateServiceCall("primary", 300));
scope.fork(() -> {
Thread.sleep(hedgeAfterMillis);
return simulateServiceCall("hedge", 150);
});
The primary path takes 300ms. The hedge waits 30ms and then takes 150ms. In this deterministic demo, the hedge should win at roughly 180ms plus handler overhead. The fresh sequence did exactly that:
hedge-01 Race Winner: hedge-ok Duration: 188ms status=200
hedge-02 Race Winner: hedge-ok Duration: 186ms status=200
hedge-03 Race Winner: hedge-ok Duration: 191ms status=200
hedge-04 Race Winner: hedge-ok Duration: 191ms status=200
hedge-05 Race Winner: hedge-ok Duration: 187ms status=200
The wrk run averaged 191.96ms with only 2.96ms of standard deviation.
This example teaches the mechanics, not a universal hedging policy. Because the primary is always 300ms in the demo, the hedge always fires. In a real service, that would be a warning sign. A hedge should fire only for the tail, not for most requests. The number to watch is not only parent latency. It is hedge-fire rate and downstream duplicate load.
The delay is the policy. Without the delay, the hedge is just a permanent second request.
Fallback looks similar to first success, but the contract is different. First success races equivalent sources. Fallback waits for the primary path to fail, then returns a different, degraded answer.
The helper in ScopedRequestHandler.runWithFallback() keeps those phases separate:
public <T> T runWithFallback(Callable<T> primary, Callable<T> fallback) throws Exception {
try (var scope = StructuredTaskScope.open(StructuredTaskScope.Joiner.awaitAllSuccessfulOrThrow())) {
var primaryFuture = scope.fork(primary);
scope.join();
return primaryFuture.get();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw e;
} catch (Exception primaryFailure) {
logger.warn("Primary task failed, using fallback: {}", primaryFailure.getMessage());
return fallback.call();
}
}
BusinessService.getDataWithFallback() wires that helper to two simulated database calls. The primary sleeps for 100ms and fails randomly 30 percent of the time. The secondary sleeps for 150ms and always returns a value.
That explains the measured sequence. Primary success showed up around 100-106ms. Fallback responses showed up around 252-261ms because the fallback runs after the failed primary attempt:
fallback-01 Secondary-userdata Duration: 252ms status=200
fallback-02 Primary-userdata Duration: 106ms status=200
fallback-10 Secondary-userdata Duration: 260ms status=200
fallback-12 Secondary-userdata Duration: 261ms status=200
The load test averaged 149.25ms with 70.39ms standard deviation. That average is not a single behavior. It is a blend of primary successes and fallback responses.
This is the main testing lesson for fallback code: never test only the HTTP status. Both paths returned 200. The meaningful observation is whether the response was primary or secondary and whether the fallback ran after the primary failed.
The four patterns differ by contract, not by syntax. When sibling sources are equivalent and idempotent, first success gives the parent one answer while source-level failure stays visible through separate monitoring. When incomplete data is acceptable, bounded partial results make the missing sections part of the response contract. A hedged read belongs to one logical read with rare tail spikes and a downstream system that can tolerate a small amount of delayed duplicate work. Fallback is different again: the degraded answer is part of the product contract and should run only after the primary path fails.
Those distinctions matter more than the API call names. The same StructuredTaskScope can express several policies. The review question is whether the code makes the policy local enough to inspect.
Fan-out tests should follow the policy path, not just assert that the handler returned. First-success tests need a known winner and separate visibility into slower or failed siblings. Partial-results tests need a deadline that some tasks meet and some tasks miss, otherwise they only prove the all-success case. Hedged-read tests need to report whether the hedge actually fired, because a low parent latency number can hide doubled downstream work. Fallback tests need one primary-success case and one fallback case, with response text or metrics that make the branch visible.
The checked-in examples are deliberately small, so the expected timing is easy to reason about. When a 200ms cache branch wins in about 207ms, or a 30ms delayed hedge plus 150ms of work returns in about 190ms, the benchmark is not proving that structured concurrency is faster in general. It is proving that the code follows the policy it claims to teach.
Structured concurrency gives the parent a clear ownership boundary. It does not choose the fan-out contract for you.
That choice has to happen before the fork. If any successful answer is enough, say that in code. If partial data is acceptable, return the missing sections honestly. If a hedge is allowed, delay it and count it. If fallback is part of the contract, keep it separate from a race.
Part 8 moves from pattern mechanics to operational checks. Once a scope owns the work, the next question is what you monitor so those ownership boundaries stay visible under load.