Solr includes a general tracing framework based on OpenTracing that can be used to trace the lifecycle of a request for performance monitoring.
Tracing data can be configured to send to various backends, either directly or via an agent. Solr can send traces directly in OTLP/gRPC and Jaeger/Thrift formats. You can support many other destinations and cloud providers by installing an agent on each node and sending traces to localhost.
A sampled distributed tracing query request displayed in Jaeger UI looks like this:
TracerConfigurator is a class used to provide an instance of
io.opentracing.Tracer based on configuration in
These are pluggable, and each plugin is shipped as a Solr Module:
opentelemetry module supports OTLP over gRPC or HTTP.
jaegertracer-configurator module supports the Jaeger format over the Thrift transport.
A setup of a
solr.xml looks like this:
<solr> <tracerConfig name="tracerConfig" class="org.apache.solr.opentelemetry.OtelTracerConfigurator" /> </solr>
<tracerConfig> is absent,
TracerConfigurator will try to pick up the
Tracer instance that was registered in
By doing this, some backends like DataDog are supported out of the box since datadog-java-agent uses Javaagent to register a
This module brings support for the industry standard OpenTelemetry (or short "OTEL") tracing, and exposes a tracer configurator that can be enabled in the
<tracerConfig> tag of
solr.xml as follows:
<tracerConfig name="tracerConfig" class="org.apache.solr.opentelemetry.OtelTracerConfigurator"/>
Enable the module with either system property
-Dsolr.modules=opentelemetry or environment variable
The default configuration will ship traces to a collector running on localhost, using OTLP over gRPC, and will propagate trace IDs using W3C TraceContext. Here are the environment settings that are enabled by default:
OTEL_SDK_DISABLED=false OTEL_SERVICE_NAME=solr OTEL_TRACES_EXPORTER=otlp OTEL_EXPORTER_OTLP_PROTOCOL=grpc OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317 OTEL_TRACES_SAMPLER=parentbased_always_on OTEL_PROPAGATORS=tracecontext,baggage
Say you want to send traces to a remote OTEL Collector, with 10% sampling. That could be done with this configuration:
OTEL_EXPORTER_OTLP_ENDPOINT=my-remote-collector:4317 OTEL_TRACES_SAMPLER=parentbased_traceidratio OTEL_TRACES_SAMPLER_ARG=0.1
An equivalent configuration using system properties would be:
SOLR_OPTS=-Dotel.exporter.otlp.endpoint=my-remote-collector:4317 -Dotel.traces.sampler=parentbased_traceidratio -Dotel.traces.sampler.arg=0.1
To add custom tags to the trace, use
The OTEL tracer also supports other exporters like Jaeger and Zipkin. However, Solr does not ship with those 3rd party libraries, thus you need to add them to Solr yourself if you have this as a requirement. Say you add the Zipkin exporter jar file(s) to
$SOLR_TIP/lib/, then you would enable OpenTelemetry module to send traces on the Zipkin format using the Protobuf encoding with a configuration like this:
This module is deprecated as of Solr 9.2, and will be removed in version 10. Users should migrate to OpenTelemetry.
jagertracer-configurator provides a default implementation for setting up Jaeger Tracer. Enable the module with either system property
-Dsolr.modules=jaegertracer-configurator or environment variable
Learn more about the available samplers at https://www.jaegertracing.io/docs/sampling/#client-sampling-configuration.
Then Jaeger tracer is configured in
solr.xml like this:
<tracerConfig name="tracerConfig" class="org.apache.solr.jaeger.JaegerTracerConfigurator" />
There are no configuration elements in the XML; instead, this 3rd party system is configured using System Properties or Environment Variables. The full list are listed at Jaeger-README.
For example, to use the probabilistic sampler, you could set this environment variable:
or System property:
bin/solr start -DJAEGER_SAMPLER_TYPE=probabilistic