Create spans for calls to external binaries

[theobisproject]

Is your feature request related to a problem? Please describe.
My application uses for some special cases external binaries. When they are called in a transaction this results in a big empty space in the transaction.

Describe the solution you'd like
Track the execution duration of the external binary and if possible also the execution call.

I would be working on this. Just wanted to get feedback on what libraries/apis are often used to execute external binaries. I was currently thinking of supporting the following

• Java Process Builder API
• Java Runtime API
• Apache Commons Exec

[eyalkoren]

Thanks for the initiative!!
Seems like good picks.
The more you will be able to share common code and logic, the easier it would be to add other frameworks if required later on.

Before you start, I suggest you define what you want to get:

• duration
• span name - should be informative enough, but with restricted cardinality, ie - no command args in name
• command/s? (we will need a place for that in the schema). Must have size limitation
• command args?
• output ? - beware, non-trivial
• error ? - beware, non-trivial
• general result (ie success/failure or exit-code) ? - probably easier and quite informative

Also, beware of span leaks, meaning- know when to end/deactivate spans even if the process is killed before ended etc.

[theobisproject]

Hi @eyalkoren, thanks for your input!
I thought of getting the following information (assuming java -version as command):

• duration
• span name -> Execute java
• command args -> -version
• exit-code

I don't think tracking the process output is something the plugin should do.

I already started a protoype implementation and stumbled about an instrumentation problem with the JDK APIs. Is there some special handling needed for those classes? I already noticed that in the unit tests the instrumentation for java.* classes is disabled. But enabling it didn't change anything.

[eyalkoren]

I don't really know, I will need to test this.
The Java APIs are probably loaded by the bootstrap class loader, but so are the agent classes, so you shouldn't get any LinkageErrors.
However, we currently only instrument classes when they are first loaded. It is possible that these classes are loaded early enough, so we miss them. The agent can relatively easily be used for retransforming already loaded classes, but first we need to know whether this is the case.

But enabling it didn't change anything.

Did you notice this part?

[felixbarny]

In the unit tests, the agent is loaded from the system cl and thus can’t instrument bootstrap classes. You’d have to create an integration test for that.

[theobisproject]

@eyalkoren Hadn't noticed those ignores but will definitely have a look at that. I added a custom elasticapm.properties file in my unit test resources folder.
@felixbarny I than would assume that the already instrumented Java SQL classes are not loaded by the bootstrap classloader since there is a working unit test.

I also will try to get some insight when and if the classes I wan't to instrument are loaded with the -XX:+TraceClassLoading JVM Flag.

[theobisproject]

I traced the classloading of the unit test and can confirm that both JDK APIs are loaded by the bootstrap classloader.
Just if someone is curious here is the output of the unit test I have on JDK 11 with the -Xlog:class+load=debug flag for the two APIs:

[1.814s][info ][class,load] java.lang.ProcessBuilder source: jrt:/java.base
[1.814s][debug][class,load]  klass: 0x0000000800321988 super: 0x0000000800001000 loader: [loader data: 0x0000022f34f966e0 of 'bootstrap'] bytes: 10456 checksum: 12c6ebf3

[0.031s][info ][class,load] java.lang.Runtime source: jrt:/java.base
[0.031s][debug][class,load]  klass: 0x000000080001a150 super: 0x0000000800001000 loader: [loader data: 0x0000017aea8ed470 of 'bootstrap'] bytes: 6156 checksum: 58dd341a

I will then use integration tests to test those instrumentations.

[theobisproject]

Was there in any API currently implemented the need to end the span in a different method than the span was started? I would need this for the implementation of the Java APIs and the async API of commons-exec.

API	Start Method	End Method
Java Process Builder	ProcessBuilder#start	Process#waitFor
Java Runtime	Runtime#exec	Process#waitFor
Commons Exec	Executor#execute	ExecuteResultHandler#waitFor

[felixbarny]

If all happens within a single thread, you can just activate the span with co.elastic.apm.agent.impl.transaction.AbstractSpan#activate and get the currently active span with co.elastic.apm.agent.impl.ElasticApmTracer#getActive. However, care has to be taken to not leak spans i.e. start and activate without deactivate and end.

I guess Process#waitFor can be called from another thread than ProcessBuilder#start though. One idea would be to store the Process returned by ProcessBuilder#start in a WeakConcurrentMap<Process, Span> so that you can look up the span in Process#waitFor given the this reference for Process.

[theobisproject]

So basically I would need something like already is done in HttpUrlConnectionInstrumentation and JdbcHelperImpl?

I also would assume that someone passes the Process object into a different thread and waits there for it to end.

[felixbarny]

So basically I would need something like already is done in HttpUrlConnectionInstrumentation and JdbcHelperImpl?

exactly