The archive · 10 posts · Aug 2025 - now

The notebook, in full. Sorted newest first.

Every post, newest first. Filter by topic, or scroll the chronological view.

posts10
topics2
last update2026-05-31
reading time~2 h total
rss
  • Series · 1 partIn flight

    Tail Latency & System Behavior

    Runnable Java experiments on the failure patterns that show up under real load: tail latency, queueing, hedged requests, coordinated omission, backpressure, and SLO engineering. Deterministic outputs, checked-in CSVs, reproducible on any machine.

    • 01
      Part
      Why Average Latency Lies

      A deterministic Java simulation where baseline p99=34ms becomes fan-out p99=597ms without changing the downstream latency sampler. Average latency is structurally incapable of catching that tail pain.

      31 May 2026
      9 min
  • Series · 9 partsIn flight

    Structured Concurrency

    Nine posts written through a fan-out service rollout. Cancellation, timeouts, and what to do when a subtask outlives the request.

    • 09
      Part
      Migrating our fan-out service from Java 21 to Java 25

      Most of the migration was mechanical. ShutdownOnFailure became a Joiner, throwIfFailed disappeared, and StructuredTaskScope.open replaced the constructor. Two things were not mechanical, and those are the ones worth reading.

      17 May 2026
      12 min
    • 08
      Part
      Four operational checks we run on every StructuredTaskScope

      Before a fan-out service can be trusted under load, four things need to be true: outcomes counted per scope, deadlines propagated, bulkheads in place, and pinning watched in JFR. What each one looks like in code.

      10 May 2026
      10 min
    • 07
      Part
      Three structured-concurrency patterns we run in a fan-out service

      Structured concurrency patterns are worth the complexity only when the cancellation policy is decided before the first fork, not after the first timeout. The three we run: aggregation on quorum, bulkheading per tenant, and a deadline shape that protects against one slow upstream.

      04 May 2026
      8 min
    • + 6 earlier parts
      • 06
        Part
        Composing resilience policies as separable layers

        Resilience composition in structured concurrency comes down to one rule: keep each policy a separable layer that fires visibly in logs, or rebuild the complexity you were trying to remove.

        26 Apr 2026
        10 min
      • 05
        Part
        Why downstream capacity is the real ceiling on fan-out

        The real ceiling on a fan-out request is downstream capacity, not the thread count. One semaphore per dependency type stops a hot path from draining the pool everyone else shares. How we set them and what changes when traffic shape shifts.

        19 Apr 2026
        10 min
      • 04
        Part
        Two workflow shapes that show up after fork-and-wait

        Past basic fork-and-wait, two workflow shapes dominate. Streaming progress as subtasks finish works for user-facing flows. Nested scopes that mirror the service tree work for fan-out into fan-out. What each one costs.

        12 Apr 2026
        10 min
      • 03
        Part
        Cancelling siblings before they burn capacity

        Not every failure is a timeout. A failed risk check turns the rest of the fan-out into wasted work, so the fix is to throw inside the subtask, cancelling its siblings before they burn capacity. The pattern, the implementation, and where it backfires.

        06 Apr 2026
        10 min
      • 02
        Part
        What a missed deadline should do, and what it should not

        Timeouts in distributed code are routine, not exceptional. The real question is whether a missed deadline should fail the whole request or return what made it back in time. The three timeout shapes we run and when each one fits.

        30 Mar 2026
        10 min
      • 01
        Part
        What structured scopes actually catch

        Most concurrency bugs in service code come from missing lifecycle, not missing parallelism. A scope that owns its forks is what makes task lifetime visible in the code that started them. The bugs that go away when scopes are right.

        22 Mar 2026
        15 min