Instrumenting Executors - issues in Spring Cloud Sleuth and JDK16

[Marcin Grzejszczak]

Hi!

I'm the lead of Spring Cloud Sleuth [1], a project dedicated to working with distributed tracing. We're propagating the tracing context e.g. through threads. That means that when a user spawns a new thread we need to pass the context from the old thread to the new one. Example - if the user uses an Executor or an ExecutorService (e.g. via calling the execute(Runnable r)​ method) then we need to wrap the Runnable​ in its trace representation. That means that we retrieve the context from Thread A , pass it in the constructor of the TraceRunnable​ and then restore it once the run​ method is called in Thread B.

The problem in Sleuth that we have with JDK16 is that we can't use reflection to ensure that we're wrapping all methods of any Executors [2]. In other words we want to create a proxy around an existing Executor and wrap all methods. Currently, we're using reflection cause Executor implementations such as `ScheduledThreadPoolExecutor` have quite a few protected methods that we can't access. What we would like to achieve is a delegation mechanism that we can wrap all objects that the given Executor is using within their API (such as Runnables, Callables, other Executors) in their trace representation and then delegate calls for all methods to the wrapped object. That would also mean the delegation to currently protected methods.

If there's another way to achieve this other than opening the java.util.concurrent API then I would very much like to use it. Currently with JDK16 it's not possible to instrument that code so context propagation might be buggy when dealing with executors.

[1] - https://spring.io/projects/spring-cloud-sleuth
Spring Cloud Sleuth<https://spring.io/projects/spring-cloud-sleuth>
Adds trace and span ids to the Slf4J MDC, so you can extract all the logs from a given trace or span in a log aggregator. Instruments common ingress and egress points from Spring applications (servlet filter, rest template, scheduled actions, message channels, feign client).
spring.io



[2] - https://github.com/spring-cloud/spring-cloud-sleuth/issues/1897
[https://avatars.githubusercontent.com/u/7815877?s=400&v=4]<https://github.com/spring-cloud/spring-cloud-sleuth/issues/1897>
InaccessibleObjectException when calling getScheduledThreadPoolExecutor · Issue #1897 · spring-cloud/spring-cloud-sleuth<https://github.com/spring-cloud/spring-cloud-sleuth/issues/1897>
Describe the bug When calling the method LazyTraceThreadPoolTaskScheduler.getScheduledThreadPoolExecutor(), a new LazyTraceScheduledThreadPoolExecutor is created. The constructor of the pool execut...
github.com





Marcin Grzejszczak
Staff Engineer, Spring Cloud