Distributed Tracing

Solr includes a general tracing framework based on OpenTracing that can be used to trace the lifecycle of a request for performance monitoring.

Tracing data can be configured to send to arbitrary backends like Jaeger, Zipkin, Datadog, etc. At the moment, only Jaeger is supported out of the box.

A sampled distributed tracing query request on Jaeger looks like this:

query request tracing
Figure 1. Tracing of a Solr query

Module

This is provided via the jaegertracer-configurator Solr Module that needs to be enabled before use.

Configuring Tracer

TracerConfigurator is a class used to provide an instance of io.opentracing.Tracer based on configuration in solr.xml. The JaegerTracerConfigurator provides a JaegerTracer instance to a Solr node.

A setup of a TracerConfigurator in solr.xml looks like this:

<solr>
  <tracerConfig name="tracerConfig" class="org.apache.solr.jaeger.JaegerTracerConfigurator" />
</solr>

If <tracerConfig> is absent, TracerConfigurator will try to pick up the Tracer instance that was registered in io.opentracing.util.GlobalTracer. By doing this, some backends like DataDog are supported out of the box since datadog-java-agent uses Javaagent to register a Tracer in io.opentracing.util.GlobalTracer.

Jaeger Tracer Configurator

The modules/jagertracer-configurator provides a default implementation for setting up Jaeger Tracer.

Add the solr-jaegertracer JAR file and the other JARs provided with this module to your Solr installation, ideally to each node. GSON is a dependency that is only used by Jaeger’s "remote" sampler, which is the default. Solr doesn’t distribute it, so you’ll need to add GSON yourself or configure a different sampler.

Then Jaeger tracer is configured in solr.xml like this:

<tracerConfig name="tracerConfig" class="org.apache.solr.jaeger.JaegerTracerConfigurator" />

There are no configuration elements in the XML; instead, this 3rd party system is configured using System Properties or Environment Variables. The full list are listed at Jaeger-README.

For example, to use the probabilistic sampler, you could set this environment variable:

export JAEGER_SAMPLER_TYPE=probabilistic

or System property:

bin/solr start -DJAEGER_SAMPLER_TYPE=probabilistic