Distributed Solr Tracing
Solr includes a general tracing framework based on OpenTracing that can be used to trace lifecycle of a request for performance monitoring. Tracing data can be configured and send to arbitrary backend like Jaeger, Zipkin, Datadog, etc. For now only Jaeger is supported out of the box.
A sampled distributed tracing query request on Jaeger looks like this:
Setup Tracer
TracerConfigurator
is a class used for provide an instance of io.opentracing.Tracer
based on configuration in solr.xml
.
Like JaegerTracerConfigurator
provide JaegerTracer instance to Solr node.
A setup of a TracerConfigurator
looks like this
<solr>
<tracerConfig name="tracerConfig" class="org.apache.solr.jaeger.JaegerTracerConfigurator">
<str name="agentHost">localhost</str>
<int name="agentPort">5775</int>
<bool name="logSpans">true</bool>
<int name="flushInterval">1000</int>
<int name="maxQueueSize">10000</int>
</tracerConfig>
</solr>
If <tracerConfig>
part is absent, TracerConfigurator
will try to pick up the Tracer
instance was registered in
in io.opentracing.util.GlobalTracer
. By doing this some backend like DataDog is supported out of the box since
datadog-java-agent use Javaagent to register a Tracer
in
io.opentracing.util.GlobalTracer
.
Configuring Sample Rate
By default only 0.1% of requests are sampled, this ensure that tracing activities does not affect performance of a node.
The rate can be changed on the fly (without restarting Solr nodes) by setting new sample rate in cluster property. For example, below call set sample rate to 100%
/admin/collections?action=CLUSTERPROP&name=samplePercentage&val=100
Jaeger Tracer Configurator
Module contrib/jagertracer-configurator
provides a default implementation for setting up Jaeger Tracer.
Note that all library of jaegertracer-configurator must be included in the classpath of all nodes then Jaeger tracer can be setup in solr.xml
like this:
<tracerConfig name="tracerConfig" class="org.apache.solr.jaeger.JaegerTracerConfigurator">
<str name="agentHost">localhost</str>
<int name="agentPort">5775</int>
<bool name="logSpans">true</bool>
<int name="flushInterval">1000</int>
<int name="maxQueueSize">10000</int>
</tracerConfig>
List of parameters for JaegerTracerConfigurator include:
Parameter | Type | Required | Default | Description |
agentHost | string | Yes | The host of Jaeger backend | |
agentPort | int | Yes | The port of Jaeger port | |
logsSpans | bool | No | true | Whether the tracer should also log the spans |
flushInterval | int | No | 5000 | The tracer’s flush interval (ms) |
maxQueueSize | int | No | 10000 | The tracer’s maximum queue size |
Other parameters which are not listed above can be configured using System Properties or Environment Variables. The full list are listed at Jaeger-README.