Series navigation
Code repositoryproject-loom
J
Written by
Jagdish Salgotra
Distributed systems, cloud-native architecture, and the JVM. mostly shipping, occasionally reading.
Was this article helpful? or email →anonymous · no account needed
Downstream capacity caps fan-out, not thread count. One semaphore per dependency stops a hot path draining the pool.
Written by
Distributed systems, cloud-native architecture, and the JVM. mostly shipping, occasionally reading.
All code in this series targets Java 21 preview APIs. Part 9 covers migration to Java 25.
Forking work is easy. That is part of the appeal of virtual threads and structured concurrency.
But every fork still points somewhere.
A request that forks five database calls is not only one request anymore. It is also five database pool borrowers. A request that fans out to four HTTP dependencies is four outbound connection users. A workflow that starts CPU-heavy transforms is not just concurrent; it is competing for a fixed number of cores.
Structured concurrency gives the parent a clear lifetime for related work. It does not decide how much downstream capacity that work is allowed to consume.
That is the central tension in this article: the scope owns task lifetime, but a resource policy owns admission. If those two ideas get mixed together, the code can look clean while still overwhelming a database, a downstream service, or the CPU.
This article is learning material, not production guidance. The repository uses simulated sleeps instead of real databases and HTTP pools. The measurements below are useful because they show the shape and the limits of the checked-in examples, not because they predict production throughput.
The easiest mistake is to count only parent requests.
If one incoming request starts five child calls, then 40 concurrent parent requests can create up to 200 child calls in flight. Virtual threads make that affordable for the JVM. They do not make the downstream system 5x larger.
In Java 21 preview syntax, the basic fan-out shape still looks familiar:
try (var scope = new StructuredTaskScope.ShutdownOnFailure()) {
Subtask<String> one = scope.fork(() -> callServiceOne());
Subtask<String> two = scope.fork(() -> callServiceTwo());
Subtask<String> three = scope.fork(() -> callServiceThree());
Subtask<String> four = scope.fork(() -> callServiceFour());
Subtask<String> five = scope.fork(() -> callServiceFive());
scope.join();
scope.throwIfFailed();
return List.of(one.get(), two.get(), three.get(), four.get(), five.get());
}
That code is easy to read. It also hides a multiplication factor.
The parent scope owns the lifetime of the five calls. It can cancel siblings when one fails. It can make sure no child outlives the parent. But it does not know whether the database pool has 10 permits, whether the HTTP client has 50 connections, or whether the CPU phase should run at one task per core.
Those limits need to be explicit somewhere else in the design.
The Article 5 HTTP evidence comes from AdvancedStructuredConcurrencyMicroservice.java, which exposes the advanced examples on port 8082. The relevant service code is in ConcurrentServiceLayer.java.
This article is learning material. The main branch now builds with OpenJDK 25.0.2 and uses the Java 25 preview structured-concurrency API, with the Java 21 version separately managed in the feature/java-21 branch. The measurements below were generated from the current Java 25 code. The Java 21 preview syntax shown here remains valid for learning purposes, and Part 9 covers the migration details.
The scatter-gather endpoint forks five simulated service calls:
One request returned:
Scatter-Gather Results: [service-1-ok, service-2-ok, service-3-ok, service-4-ok, service-5-ok]
Duration: 189ms
That is the normal fan-out behavior: the response follows the slowest branch, not the sum of all branches.
The focused load check used 40 connections for 10 seconds:
wrk -t4 -c40 -d10s http://localhost:8082/pattern/scatter-gather
Latency avg: 188.68ms
Requests/sec: 210.12
Requests: 2,120
Because this endpoint forks five simulated calls per request, that run represents roughly 10,600 simulated branch calls over 10 seconds. That number is derived from the code shape, not from an external dependency counter. The repository is sleeping, not calling a real database. Still, it is the right mental model: fan-out turns parent traffic into downstream traffic.
The bulkhead endpoint has a different shape:
One request returned:
Bulkhead Pattern: Critical[critical-auth-ok, critical-payment-ok] Non-Critical[analytics-ok, logging-ok]
Duration: 205ms
The load check was:
wrk -t4 -c40 -d10s http://localhost:8082/pattern/bulkhead
Latency avg: 206.56ms
Requests/sec: 190.36
Requests: 1,920
The result again tracks the slowest branch, which is the 200ms analytics call. That is an important detail. This checked-in bulkhead example separates critical and non-critical work into different scopes, but it still waits for both groups before returning. It demonstrates a policy boundary. It does not demonstrate degraded response after optional work misses a budget.
That distinction matters. A separate scope makes the boundary visible, but the product policy still has to decide what happens when the non-critical side is slow or failing.
After the two focused runs and the single requests, the service metrics reported:
Total Requests: 4122
Timeout Count: 0
Average Response Time: 194.17ms
CPU Usage: 15.49%
Memory Usage: 163.04MB / 776.00MB
Those aggregate metrics line up with the endpoint timings: scatter-gather sat near 189ms, bulkhead near 206ms, and the combined average landed between them.
The standalone Article 5 code lives in AdvancedStructuredPatterns.java.
The resource-aware example groups tasks by resource type:
public List<String> executeResourceAware(List<ResourceTask> tasks) throws Exception {
var cpuTasks = tasks.stream().filter(t -> t.getType() == ResourceType.CPU).toList();
var memoryTasks = tasks.stream().filter(t -> t.getType() == ResourceType.MEMORY).toList();
var ioTasks = tasks.stream().filter(t -> t.getType() == ResourceType.IO).toList();
try (var scope = new StructuredTaskScope.ShutdownOnFailure()) {
var cpuResult = scope.fork(() -> executeResourceGroup(cpuTasks));
var memoryResult = scope.fork(() -> executeResourceGroup(memoryTasks));
var ioResult = scope.fork(() -> executeResourceGroup(ioTasks));
scope.join();
scope.throwIfFailed();
List<String> allResults = new ArrayList<>();
allResults.addAll(cpuResult.get());
allResults.addAll(memoryResult.get());
allResults.addAll(ioResult.get());
return allResults;
}
}
The local run used:
CPU-1: 200ms
CPU-2: 300ms
MEM-1: 100ms
MEM-2: 200ms
IO-1: 100ms
The output was:
Resource-aware results: [CPU-1 completed, CPU-2 completed, MEM-1 completed, MEM-2 completed, IO-1 completed]
That output is real, but it also exposes the limitation of the example. Grouping by CPU, MEMORY, and IO makes the orchestration clearer, but it does not by itself enforce a capacity number. There is no semaphore in this checked-in scheduler. There is no real database pool. There is no measured queue wait.
That is not a reason to hide the example. It is a reason to teach the boundary honestly.
Resource grouping answers: which category does this work belong to?
Capacity limiting answers: how many tasks in that category may run at once?
A fuller version of this pattern needs the second answer. For a database, that number should usually come from the connection pool or the driver settings. For outbound HTTP, it should line up with the client connection limits and the downstream contract. For CPU-heavy work, it should be bounded around available cores and measured saturation, not just request count.
Structured concurrency composes with those limits. It does not replace them.
A bulkhead is useful because it prevents every class of work from sharing one undifferentiated pool of failure and capacity.
The standalone bulkhead example uses a parent scope with two child groups:
try (var scope = new StructuredTaskScope.ShutdownOnFailure()) {
var criticalBulkhead = scope.fork(() -> executeBulkhead(criticalTasks, "Critical"));
var normalBulkhead = scope.fork(() -> executeBulkhead(normalTasks, "Normal"));
scope.join();
scope.throwIfFailed();
return new BulkheadResult(criticalBulkhead.get(), normalBulkhead.get());
}
In the local run, the critical group had two tasks at 100ms and 150ms. The normal group had three tasks at 200ms, 250ms, and 300ms.
The output was:
Critical bulkhead completed with 2 results
Normal bulkhead completed with 3 results
Bulkhead results: Critical: [Critical-1, Critical-2], Normal: [Normal-1, Normal-2, Normal-3]
Because the parent waits for both groups, the standalone example completes when the 300ms normal task completes. That is fine for the example, but it is not the only valid policy.
If normal work is optional, the parent may return after critical work and let the normal group fail closed or degrade. If normal work is required, then it should remain part of the response contract. If normal work uses a scarce dependency, it may need a separate permit budget even inside its own scope.
The useful rule is that "critical" and "normal" should not just be names in a log line. They should correspond to different failure behavior, different capacity limits, or both.
The adaptive example processes 20 tasks in batches. Each task sleeps for a random 100-299ms.
The code starts at a batch size of five. After each batch, it adjusts the next batch size based on duration:
if (duration < 100) {
batchSize = Math.min(batchSize * 2, 10);
} else if (duration > 500) {
batchSize = Math.max(batchSize / 2, 2);
}
In the local run, all four batches stayed in the middle range:
Batch completed in 261ms, next batch size: 5
Batch completed in 298ms, next batch size: 5
Batch completed in 288ms, next batch size: 5
Batch completed in 298ms, next batch size: 5
Adaptive results: 20 tasks completed
That output is useful because it shows the adjustment rule did not move. The observed batch durations were not below 100ms and not above 500ms, so the batch size stayed at five.
This is another place where a learning example should not overclaim. The adaptive controller here uses only batch duration. Real adaptive concurrency usually needs a better signal: queue wait, timeout rate, downstream error rate, saturation, or a measured latency percentile. Duration alone can be misleading because the batch may be slow for reasons unrelated to capacity.
The teaching point is still valuable: if concurrency is adjusted, the adjustment must be driven by a visible signal. Otherwise "adaptive" is just a hardcoded number with a nicer name.
Before the Article 5 standalone run could reach the resource-aware patterns, the Java 25 migrated code failed in the earlier conditional-cancellation and progressive sections.
The failure was the same family of bug from Part 4:
java.lang.IllegalStateException: join not called
java.lang.IllegalStateException: Owner did not join after forking
The code was reading subtask results while polling task state, before the owner had joined the scope. I fixed that by publishing results from inside the child task, observing terminal state without calling Subtask.get(), and calling scope.join() before returning from the scope.
That matters in a resource-aware article because capacity policy does not weaken ownership. Even if you are grouping work by CPU, memory, I/O, critical, or normal, the scope owner still has to join what it forked.
The successful rerun reached all seven standalone patterns and ended with:
All advanced patterns completed!
Resource-aware code needs tests that make pressure visible, not just tests that confirm happy-path success. Branch multiplication is the first thing to check: if one request forks five child calls, the load test should report implied child calls alongside parent throughput, not just parent requests per second. The bulkhead boundary needs its own evidence too. What actually happens when the normal side is slow while the critical side succeeds? If the response waits, say that. If it degrades, verify the degraded path and confirm normal work does not outlive the parent scope.
The adaptive controller needs inputs that cross both thresholds, not just a run that stays comfortably in the middle range as this one did. Queue wait and rejection counts during bursts matter more than average latency on a quiet run. Semaphore wait time, pool wait time, timeout count, rejected request count, and p95/p99 latency show where capacity is actually being spent. Virtual threads can make the parent code look calm while the downstream queue is doing the suffering.
Part 6 moves from individual policies to composition. Timeout, fallback, retry, bulkhead, and resource admission are easier to reason about when each policy has a clear owner and a clear boundary.
The companion repository at github.com/salgotraja/project-loom has runnable examples for every post in this series. The README covers build setup and the scripts used to reproduce the checks.