Note
This series uses Java 21 as the baseline. Virtual threads are stable in Java 21 (JEP 444). Structured concurrency snippets in this part (StructuredTaskScope, JEP 453) use preview APIs and require --enable-preview.
TL;DR
- Production monitoring for virtual threads needs metrics beyond standard thread-pool dashboards
- Pinning detection is a core signal for performance and latency diagnostics
- JFR integration helps analyze virtual thread behavior and carrier utilization
- Custom monitoring utilities can expose task-level execution and error patterns
- Debugging approach differs from platform-thread-only systems
- Performance tuning typically focuses on pinning, lock choices, and scheduler behavior
The Problem: Observability Gaps in Production
Virtual thread incidents are harder when old dashboards look normal while latency climbs.
The tooling that worked for platform threads often misses what matters here: pinning, carrier utilization, and scope behavior under stress.
The Traditional Monitoring Blind Spot
Here's what most production monitoring looks like when virtual threads enter the picture:
private static String generateJvmInfo() {
StringBuilder info = new StringBuilder();
info.append("JVM Information:\n");
info.append("================\n");
info.append("Java Version: ").append(System.getProperty("java.version")).append("\n");
info.append("JVM Name: ").append(runtimeBean.getVmName()).append("\n");
info.append("JVM Version: ").append(runtimeBean.getVmVersion()).append("\n");
info.append("Uptime: ").append(runtimeBean.getUptime() / 1000).append(" seconds\n");
info.append("Available Processors: ").append(Runtime.getRuntime().availableProcessors()).append("\n");
MemoryUsage heapUsage = memoryBean.getHeapMemoryUsage();
info.append("Heap Usage: ").append(heapUsage.getUsed() / 1024 / 1024).append("MB / ")
.append(heapUsage.getMax() / 1024 / 1024).append("MB\n");
return info.toString();
}
Common monitoring blind spots:
- Thread count interpretation: small carrier-thread counts can hide large virtual-thread fan-out
- CPU interpretation: high CPU can reflect pinning or lock contention, not only business load
- Memory interpretation: virtual threads shift allocation patterns compared to platform threads
- Pinning visibility gap: without explicit signals, pinning is easy to miss
- Stack-trace differences: failure analysis looks different from platform-thread workflows
- Carrier utilization gap: overloaded carriers are easy to overlook without dedicated metrics
Observability Approach for Virtual Threads
Virtual thread observability needs explicit visibility into the relationship between virtual threads and carriers, with pinning as a first-class signal.
The VirtualThreadMonitor
public class VirtualThreadMonitor {
private final AtomicLong taskCount = new AtomicLong(0);
private final AtomicLong successCount = new AtomicLong(0);
private final AtomicLong errorCount = new AtomicLong(0);
private final AtomicLong totalExecutionTime = new AtomicLong(0);
private final Map<String, TaskMetrics> taskMetrics = new ConcurrentHashMap<>();
private final Map<String, Long> errorCounts = new ConcurrentHashMap<>();
private final ThreadMXBean threadBean = ManagementFactory.getThreadMXBean();
private final Runtime runtime = Runtime.getRuntime();
private ScheduledExecutorService scheduler;
private volatile boolean monitoring = false;
public void startMonitoring() {
monitoring = true;
scheduler = Executors.newScheduledThreadPool(1);
scheduler.scheduleAtFixedRate(this::collectMetrics, 0, 1, TimeUnit.SECONDS);
}
public <T> T trackTask(String taskName, Callable<T> task) throws Exception {
long startTime = System.nanoTime();
long startMemory = runtime.totalMemory() - runtime.freeMemory();
taskCount.incrementAndGet();
try {
T result = task.call();
long executionTime = System.nanoTime() - startTime;
long memoryUsed = (runtime.totalMemory() - runtime.freeMemory()) - startMemory;
successCount.incrementAndGet();
totalExecutionTime.addAndGet(executionTime);
taskMetrics.compute(taskName, (key, metrics) -> {
if (metrics == null) metrics = new TaskMetrics();
metrics.addExecution(executionTime, memoryUsed, true);
return metrics;
});
return result;
} catch (Exception e) {
long executionTime = System.nanoTime() - startTime;
errorCount.incrementAndGet();
totalExecutionTime.addAndGet(executionTime);
errorCounts.merge(e.getClass().getSimpleName(), 1L, Long::sum);
throw e;
}
}
private void collectMetrics() {
if (!monitoring) return;
long usedMemory = runtime.totalMemory() - runtime.freeMemory();
int threadCount = threadBean.getThreadCount();
System.out.printf("Threads: %d, Memory: %.2fMB, Tasks: %d (Success: %d, Errors: %d)%n",
threadCount, usedMemory / 1024.0 / 1024.0,
taskCount.get(), successCount.get(), errorCount.get());
}
}
What this monitor provides:
- Virtual thread lifecycle tracking: See creation, completion, and execution patterns
- Pinning detection: Automatically flags potential carrier thread pinning
- Per-task metrics: Understand which operations are problematic
- Memory correlation: Track memory usage patterns for virtual threads
- Real-time alerts: Immediate notification of performance anomalies
Deep Dive: JFR Integration and Advanced Monitoring
JFR for Virtual Threads
private static void startJFRProfiling() {
logger.info("JFR Profiling enabled. Use JVM args: -XX:+FlightRecorder -XX:StartFlightRecording=duration=60s,filename=app.jfr");
}
static void main(String[] args) throws Exception {
startJFRProfiling();
HttpServer server = HttpServer.create(new InetSocketAddress(PORT), 0);
server.setExecutor(Executors.newVirtualThreadPerTaskExecutor());
server.createContext("/metrics", exchange -> {
String metrics = generatePrometheusMetrics();
sendResponse(exchange, metrics);
});
server.start();
}
Virtual Thread Pinning Detection
public class VirtualThreadPinningDetector {
private static final Logger logger = LoggerFactory.getLogger(VirtualThreadPinningDetector.class);
private static final Object SYNC_LOCK = new Object();
private static final ReentrantLock REENTRANT_LOCK = new ReentrantLock();
static void main(String[] args) throws InterruptedException {
logger.info(" Virtual Thread Pinning Detection Demo");
logger.info("Run with: -Djdk.tracePinnedThreads=full");
logger.info(" Test 1: Synchronized blocks (CAUSES PINNING)");
testSynchronizedPinning();
logger.info(" Test 2: ReentrantLock (NO PINNING)");
testReentrantLockNoPinning();
logger.info(" Test 3: Heavy synchronized load (CAUSES PINNING)");
testHeavySynchronizedLoad();
logger.info(" Test 4: Optimized with ReentrantLock (NO PINNING)");
testOptimizedWithReentrantLock();
}
private static void testSynchronizedPinning() throws InterruptedException {
CountDownLatch latch = new CountDownLatch(10);
for (int i = 0; i < 10; i++) {
final int taskId = i;
Thread.startVirtualThread(() -> {
try {
synchronizedWork(taskId);
} finally {
latch.countDown();
}
});
}
latch.await();
logger.info("Synchronized test completed");
}
private static void testReentrantLockNoPinning() throws InterruptedException {
CountDownLatch latch = new CountDownLatch(10);
for (int i = 0; i < 10; i++) {
final int taskId = i;
Thread.startVirtualThread(() -> {
try {
reentrantLockWork(taskId);
} finally {
latch.countDown();
}
});
}
latch.await();
logger.info("ReentrantLock test completed");
}
}
Note
Run with -Djdk.tracePinnedThreads=full to see stack traces in console/logs.
Pinning detection in production:
private static void testPinningMetrics() throws InterruptedException {
PinningMetrics metrics = new PinningMetrics();
CountDownLatch latch = new CountDownLatch(50);
for (int i = 0; i < 25; i++) {
final int taskId = i;
Thread.startVirtualThread(() -> {
try {
metrics.trackOperation("synchronized", () -> synchronizedWork(taskId));
} finally {
latch.countDown();
}
});
}
for (int i = 0; i < 25; i++) {
final int taskId = i;
Thread.startVirtualThread(() -> {
try {
metrics.trackOperation("reentrant", () -> reentrantLockWork(taskId));
} finally {
latch.countDown();
}
});
}
latch.await();
metrics.printReport();
}
Pinning detection helps surface lock hot paths where virtual threads remain mounted on carriers longer than expected.
- Monitor JFR
jdk.VirtualThreadPinned events for duration/threshold alerts.
Real-World Production Examples
Complete Monitoring Microservice
This is a complete monitoring service example for virtual-thread workloads.
public class JvmMonitoringService {
public static final int PORT = 8083;
static void main(String[] args) throws Exception {
startJFRProfiling();
HttpServer server = HttpServer.create(new InetSocketAddress(PORT), 0);
server.setExecutor(Executors.newVirtualThreadPerTaskExecutor());
server.createContext("/metrics", exchange -> {
String metrics = generatePrometheusMetrics();
sendResponse(exchange, metrics);
});
server.createContext("/jvm-info", exchange -> {
String info = generateJvmInfo();
sendResponse(exchange, info);
});
server.start();
}
}
Advanced Debugging Techniques
These reporting techniques help during production debugging.
public void printDetailedReport() {
Duration totalTime = Duration.between(monitoringStartTime.get(), Instant.now());
System.out.printf("Monitoring Duration: %s%n", formatDuration(totalTime));
System.out.printf("Total Tasks: %d%n", taskCount.get());
System.out.printf("Successful Tasks: %d (%.1f%%)%n",
successCount.get(), (successCount.get() * 100.0) / taskCount.get());
System.out.printf("Failed Tasks: %d (%.1f%%)%n",
errorCount.get(), (errorCount.get() * 100.0) / taskCount.get());
if (taskCount.get() > 0) {
System.out.printf("Average Execution Time: %.2fms%n",
(totalExecutionTime.get() / 1_000_000.0) / taskCount.get());
}
System.out.printf("Current Thread Count: %d%n", threadBean.getThreadCount());
System.out.printf("Peak Thread Count: %d%n", threadBean.getPeakThreadCount());
System.out.printf("Total Started Threads: %d%n", threadBean.getTotalStartedThreadCount());
}
These reports are useful for narrowing down pinning and latency regressions.
JVM Tuning for Virtual Threads
These JVM parameters are commonly used when tuning virtual-thread services.
-XX:+UseG1GC
-XX:+UnlockExperimentalVMOptions
-XX:+UseTransparentHugePages
-Djdk.virtualThreadScheduler.parallelism=16
-Djdk.virtualThreadScheduler.maxPoolSize=256
-Djdk.tracePinnedThreads=full
-XX:+FlightRecorder
-XX:StartFlightRecording=duration=300s,filename=production-vt.jfr
Pinning Optimization Patterns
These code patterns reduce pinning risk in hot paths.
public class PinningOptimizationPatterns {
private static final Object syncLock = new Object();
private static final ReentrantLock reentrantLock = new ReentrantLock();
private static final Map<String, String> synchronizedCache = new ConcurrentHashMap<>();
private static final Map<String, String> optimizedCache = new ConcurrentHashMap<>();
private static void synchronizedIncrement() {
synchronized (syncLock) {
try {
Thread.sleep(1);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
}
private static void reentrantLockIncrement() {
reentrantLock.lock();
try {
Thread.sleep(1);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
} finally {
reentrantLock.unlock();
}
}
private static void synchronizedCacheOperation(int taskId) {
synchronized (syncLock) {
try {
synchronizedCache.put("key-" + taskId, "value-" + taskId);
Thread.sleep(2);
synchronizedCache.get("key-" + taskId);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
}
private static void concurrentCacheOperation(int taskId) {
try {
optimizedCache.put("key-" + taskId, "value-" + taskId);
Thread.sleep(2);
optimizedCache.get("key-" + taskId);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
}
Production Monitoring Checklist
This checklist captures a baseline monitoring setup.
public class ProductionMonitoringChecklist {
public void setupProductionMetrics() throws Exception {
startJFRProfiling();
HttpServer server = HttpServer.create(new InetSocketAddress(PORT), 0);
server.setExecutor(Executors.newVirtualThreadPerTaskExecutor());
server.createContext("/metrics", exchange -> {
String metrics = generatePrometheusMetrics();
sendResponse(exchange, metrics);
});
server.createContext("/jvm-info", exchange -> {
String info = generateJvmInfo();
sendResponse(exchange, info);
});
server.start();
}
}
The practical goal is to detect pinning and carrier saturation before latency degrades.
Checklist items:
- Enable pinning visibility early (
-Djdk.tracePinnedThreads=full) in test and staging runs
- Expose runtime metrics endpoints (
/metrics, /jvm-info) and collect them centrally
- Capture JFR profiles during load tests and inspect
jdk.VirtualThreadPinned events
- Alert on latency tails, carrier utilization trends, and error-rate changes
Migration Strategy: From Baseline Metrics to Full Observability
Phase 0: Enable Pinning Trace from Day 1
Start with -Djdk.tracePinnedThreads=full in non-production load environments so early stack traces are available during incident simulation.
Phase 1: Establish Baseline Monitoring
Start by collecting baseline virtual-thread and carrier metrics.
public class BaselineVirtualThreadMonitoring {
public void establishBaseline() {
VirtualThreadMonitor monitor = new VirtualThreadMonitor();
monitor.startMonitoring();
try (var scope = new StructuredTaskScope.ShutdownOnFailure()) {
for (int i = 0; i < 20; i++) {
final int taskId = i;
scope.fork(() -> monitor.trackTask("structured-task-" + taskId, () -> {
Thread.sleep(50 + ThreadLocalRandom.current().nextInt(100));
return "Structured task " + taskId + " completed";
}));
}
scope.join();
scope.throwIfFailed();
} catch (Exception e) {
throw new RuntimeException(e);
}
}
}
Phase 2: Identify and Fix Pinning Issues
Then identify and reduce pinning in hot paths.
public class PinningEliminationPhase {
public void eliminatePinning() {
compareBasicLocking();
compareCacheImplementations();
compareReaderWriterLocks();
}
private void compareBasicLocking() throws InterruptedException {
final int TASKS = 1000;
for (int i = 0; i < TASKS; i++) {
Thread.startVirtualThread(() -> synchronizedIncrement());
}
for (int i = 0; i < TASKS; i++) {
Thread.startVirtualThread(() -> reentrantLockIncrement());
}
}
}
Phase 3: Advanced Observability
Finally, expand into full production observability.
public class ProductionObservabilitySetup {
public void setupProductionMonitoring() {
startJFRProfiling();
HttpServer server = HttpServer.create(new InetSocketAddress(PORT), 0);
server.setExecutor(Executors.newVirtualThreadPerTaskExecutor());
server.createContext("/metrics", exchange -> {
String metrics = generatePrometheusMetrics();
sendResponse(exchange, metrics);
});
server.createContext("/jvm-info", exchange -> {
String info = generateJvmInfo();
sendResponse(exchange, info);
});
}
}
Validate Gains in Staging
- Run comparable load tests before and after observability changes
- Compare p50/p95/p99 latency and timeout rates under induced contention
- Check pinned-thread traces and JFR
jdk.VirtualThreadPinned events in the same test window
- Verify alert noise vs signal quality before production rollout
Key Takeaways
Essential Monitoring Requirements
Monitor these in production:
- Pinning detection: a common cause of virtual-thread performance degradation
- Carrier thread utilization: Should stay below 80% under normal load
- Virtual thread lifecycle: Creation, completion, and failure rates
- Memory patterns: Virtual threads use heap differently than platform threads
- Performance metrics: Execution times and throughput by task type
Critical JVM Configuration
Common production JVM setup:
-Djdk.tracePinnedThreads=full
-XX:+FlightRecorder
-Djdk.virtualThreadScheduler.parallelism=<CPU_CORES>
-XX:+UseG1GC
Common Pitfalls to Avoid
Don't make these production mistakes:
- Synchronized blocks in hot paths: can significantly degrade performance through pinning
- Traditional thread monitoring: Gives misleading metrics for virtual threads
- Ignoring carrier utilization: High utilization indicates pinning
- No pinning detection: You'll have performance problems and not know why
- Inadequate JFR setup: Missing the deep insights you need for tuning
Migration Patterns
A practical sequence teams often follow:
- Start with monitoring: You can't improve what you can't see
- Find and fix pinning first: This gives the biggest performance gains
- Gradual rollout: One service at a time with strong monitoring
- Team training: Virtual threads require different debugging mindsets
- Continuous monitoring: Performance characteristics change with load
Teams that adopt observability early usually resolve pinning and latency issues faster.
What's Next?
In Part 8, we'll explore The Future of Java Concurrency, including Scoped Values, Foreign Function API integration, and ecosystem roadmap implications.
We'll cover:
- Scoped Values and their impact on virtual thread programming
- Integration patterns between stream-style flows and virtual threads
- Migration planning across evolving Loom features
- Community adoption patterns and migration strategies
- The future roadmap for Java concurrency features
Resources
Part 7 complete. This one is about operating virtual threads safely once real traffic is on them.
Series Navigation:
If you deploy virtual-thread monitoring, compare old and new dashboards during induced failure tests.