Distributed Tracing
Solr includes a general tracing framework based on OpenTracing that can be used to trace the lifecycle of a request for performance monitoring.
Tracing data can be configured to send to various backends, either directly or via an agent. Solr can send traces directly in OTLP/gRPC and Jaeger/Thrift formats. You can support many other destinations and cloud providers by installing an agent on each node and sending traces to localhost.
A sampled distributed tracing query request displayed in Jaeger UI looks like this:
Always-on Trace Id generation
Solr includes an always-on trace id generator. This will inject trace id headers for every Solr request, propagating the client supplied value or generating a new id as needed and replaces the existing rid
mechanism.
The header name it uses for propagation is X-Trace-Id
, this can be changed by updating the system property solr.traceIdHeader
.
This plugin will not be enabled if another Tracing mechanism is setup via Solr config.
This plugin can be disabled by setting the solr.alwaysOnTraceId
system property to false
.
Modules and Configuration
The TracerConfigurator
is a class used to provide an instance of io.opentracing.Tracer
based on configuration in solr.xml
.
These are pluggable, and each plugin is shipped as a Solr Module:
opentelemetry: The opentelemetry
module supports OTLP over gRPC or HTTP.
jaegertracer-configurator: The jaegertracer-configurator
module supports the Jaeger format over the Thrift transport.
A setup of a TracerConfigurator
in solr.xml
looks like this:
<solr>
<tracerConfig name="tracerConfig" class="org.apache.solr.opentelemetry.OtelTracerConfigurator" />
</solr>
If <tracerConfig>
is absent, TracerConfigurator
will try to pick up the Tracer
instance that was registered in io.opentracing.util.GlobalTracer
.
By doing this, some backends like DataDog are supported out of the box since datadog-java-agent uses Javaagent to register a Tracer
in io.opentracing.util.GlobalTracer
.
Open Telemetry Module
This module brings support for the industry standard OpenTelemetry (or short "OTEL") tracing, and exposes a tracer configurator that can be enabled in the <tracerConfig>
tag of solr.xml
as follows:
<tracerConfig name="tracerConfig" class="org.apache.solr.opentelemetry.OtelTracerConfigurator"/>
As an alternative to changing the solr.xml
file, the OTEL
tracer will be enabled if the system property otel.service.name
or environment variable OTEL_SERVICE_NAME
is present. The opentelemetry
module still needs to be enabled for the tracer to work.
Enable the module with either system property -Dsolr.modules=opentelemetry
or environment variable SOLR_MODULES=opentelemetry
.
Configuration
The tracer can be configured through environment variables or Java system properties, see OTEL SDK Environment Variables and Java SDK Autoconfigure.
The default configuration will ship traces to a collector running on localhost, using OTLP over gRPC, and will propagate trace IDs using W3C TraceContext. Here are the environment settings that are enabled by default:
OTEL_SDK_DISABLED=false
OTEL_SERVICE_NAME=solr
OTEL_TRACES_EXPORTER=otlp
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317
OTEL_TRACES_SAMPLER=parentbased_always_on
OTEL_PROPAGATORS=tracecontext,baggage
Say you want to send traces to a remote OTEL Collector, with 10% sampling. That could be done with this configuration:
OTEL_EXPORTER_OTLP_ENDPOINT=my-remote-collector:4317
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1
An equivalent configuration using system properties would be:
SOLR_OPTS=-Dotel.exporter.otlp.endpoint=my-remote-collector:4317 -Dotel.traces.sampler=parentbased_traceidratio -Dotel.traces.sampler.arg=0.1
To add custom tags to the trace, use OTEL_RESOURCE_ATTRIBUTES
:
OTEL_RESOURCE_ATTRIBUTES="application=OnlineBanking,exampleKey=exampleValue"
Consult the OTEL documentation for details on all the configuration options. This version of Solr uses OpenTelemetry SDK v1.35.0.
Using other exporters
The OTEL tracer also supports other exporters like Jaeger and Zipkin. However, Solr does not ship with those 3rd party libraries, thus you need to add them to Solr yourself if you have this as a requirement. Say you add the Zipkin exporter jar file(s) to $SOLR_TIP/lib/
, then you would enable OpenTelemetry module to send traces on the Zipkin format using the Protobuf encoding with a configuration like this:
OTEL_TRACES_EXPORTER=zipkin
OTEL_EXPORTER_ZIPKIN_ENDPOINT=http://localhost:9411/api/v2/spans
Jaeger Tracer Module
This module is deprecated as of Solr 9.2, and will be removed in version 10. Users should migrate to OpenTelemetry. |
The module jagertracer-configurator
provides a default implementation for setting up Jaeger Tracer. Enable the module with either system property -Dsolr.modules=jaegertracer-configurator
or environment variable SOLR_MODULES=jaegertracer-configurator
.
Learn more about the available samplers at https://www.jaegertracing.io/docs/sampling/#client-sampling-configuration.
Then Jaeger tracer is configured in solr.xml
like this:
<tracerConfig name="tracerConfig" class="org.apache.solr.jaeger.JaegerTracerConfigurator" />
There are no configuration elements in the XML; instead, this 3rd party system is configured using System Properties or Environment Variables. The full list are listed at Jaeger-README.
For example, to use the probabilistic sampler, you could set this environment variable:
export JAEGER_SAMPLER_TYPE=probabilistic
or System property:
bin/solr start -DJAEGER_SAMPLER_TYPE=probabilistic