Solr includes a general tracing framework based on OpenTracing that can be used to trace the lifecycle of a request for performance monitoring.
Tracing data can be configured to send to various backends, either directly or via an agent. Solr can send traces directly in OTLP/gRPC and Jaeger/Thrift formats. You can support many other destinations and cloud providers by installing an agent on each node and sending traces to localhost.
A sampled distributed tracing query request displayed in Jaeger UI looks like this:
Solr includes an always-on trace id generator. This will inject trace id headers for every Solr request, propagating the client supplied value or generating a new id as needed and replaces the existing
The header name it uses for propagation is
X-Trace-Id, this can be changed by updating the system property
This plugin will not be enabled if another Tracing mechanism is setup via Solr config.
This plugin can be disabled by setting the
solr.alwaysOnTraceId system property to
TracerConfigurator is a class used to provide an instance of
io.opentracing.Tracer based on configuration in
These are pluggable, and each plugin is shipped as a Solr Module:
opentelemetry module supports OTLP over gRPC or HTTP.
jaegertracer-configurator module supports the Jaeger format over the Thrift transport.
A setup of a
solr.xml looks like this:
<tracerConfig name="tracerConfig" class="org.apache.solr.opentelemetry.OtelTracerConfigurator" />
<tracerConfig> is absent,
TracerConfigurator will try to pick up the
Tracer instance that was registered in
By doing this, some backends like DataDog are supported out of the box since datadog-java-agent uses Javaagent to register a
This module brings support for the industry standard OpenTelemetry (or short "OTEL") tracing, and exposes a tracer configurator that can be enabled in the
<tracerConfig> tag of
solr.xml as follows:
<tracerConfig name="tracerConfig" class="org.apache.solr.opentelemetry.OtelTracerConfigurator"/>
As an alternative to changing the
solr.xml file, the
OTEL tracer will be enabled if the system property
otel.service.name or environment variable
OTEL_SERVICE_NAME is present. The
opentelemetry module still needs to be enabled for the tracer to work.
Enable the module with either system property
-Dsolr.modules=opentelemetry or environment variable
The default configuration will ship traces to a collector running on localhost, using OTLP over gRPC, and will propagate trace IDs using W3C TraceContext. Here are the environment settings that are enabled by default:
Say you want to send traces to a remote OTEL Collector, with 10% sampling. That could be done with this configuration:
An equivalent configuration using system properties would be:
SOLR_OPTS=-Dotel.exporter.otlp.endpoint=my-remote-collector:4317 -Dotel.traces.sampler=parentbased_traceidratio -Dotel.traces.sampler.arg=0.1
To add custom tags to the trace, use
The OTEL tracer also supports other exporters like Jaeger and Zipkin. However, Solr does not ship with those 3rd party libraries, thus you need to add them to Solr yourself if you have this as a requirement. Say you add the Zipkin exporter jar file(s) to
$SOLR_TIP/lib/, then you would enable OpenTelemetry module to send traces on the Zipkin format using the Protobuf encoding with a configuration like this:
This module is deprecated as of Solr 9.2, and will be removed in version 10. Users should migrate to OpenTelemetry.
jagertracer-configurator provides a default implementation for setting up Jaeger Tracer. Enable the module with either system property
-Dsolr.modules=jaegertracer-configurator or environment variable
Learn more about the available samplers at https://www.jaegertracing.io/docs/sampling/#client-sampling-configuration.
Then Jaeger tracer is configured in
solr.xml like this:
<tracerConfig name="tracerConfig" class="org.apache.solr.jaeger.JaegerTracerConfigurator" />
There are no configuration elements in the XML; instead, this 3rd party system is configured using System Properties or Environment Variables. The full list are listed at Jaeger-README.
For example, to use the probabilistic sampler, you could set this environment variable:
or System property:
bin/solr start -DJAEGER_SAMPLER_TYPE=probabilistic