This functionality is now deprecated and will be removed in Solr 9.0.
Solr collects long-term history of certain key metrics both in SolrCloud and in standalone mode.
This information can be used for very simple monitoring and troubleshooting, but also some
SolrCloud components (e.g., autoscaling) can use this data for making informed decisions based on
long-term trends of selected metrics.
The .system collection must exist if metrics history should be persisted. If this collection
is absent then metrics history will still be collected and kept in memory but it will be lost
on node restart.
Before discussing how to configure metrics storage, a bit of explanation about how it works may be helpful.
The metrics history data is maintained as multi-resolution time series, with a fixed total number of data points
per metric history (a fixed size window). Multi-resolution refers to the fact that data from the most detailed
time series is periodically resampled to create coarser-grained time series, which in turn
are periodically resampled again to build even coarser-grained series.
In the default configuration selected metrics are sampled every 60 seconds, and the following
time series are built:
240 samples, every 60 sec (4 hours)
288 samples, every 600 sec (48 hours)
336 samples, every 1h (2 weeks)
180 samples, every 4h (2 months)
365 samples, every 1 day (1 year)
This means that the total number of samples in all data series is constant, and consequently
the size of this data structure is also constant (because the size of the moving window is fixed, and
older samples are replaced by newer ones). This arrangement is referred to as a
round-robin database, and Solr uses implementation of this concept provided by the RRD4j library.
Databases created with RRD4j are compact - for the time series specified above the total size
of data is around 11kB for each of the primary time series, including its resampled data. Each database may contain
several primary time series ("datasources" in RRD4j parlance) and their re-sampled versions (called
This data is updated in memory and then periodically stored in the .system
collection in the form of Solr documents with a binary data_bin field, each document
containing data of one full database. This method of storage is much more compact and generates less
update operations than storing each data point in a separate Solr document. The Metrics History API allows retrieving
detailed data from each database, including retrieval of all individual datapoints.
Databases are identified primarily by their corresponding metric registry name, so for databases that
keep track of aggregated metrics this will be e.g., solr.jvm, solr.node, solr.collection.gettingstarted.
For databases with non-aggregated metrics the name consists of the registry name, optionally with a node name
to identify databases with the same name coming from different nodes. For example, per-node databases are
named like this: solr.jvm.localhost:8983_solr, solr.node.localhost:7574_solr, but per-replica names are
already unique across the cluster so they are named like this: solr.core.gettingstarted.shard1.replica_n1.
Currently the following selected metrics are tracked:
Non-aggregated solr.core and aggregated solr.collection metrics:
numShards (aggregated, active shards)
numReplicas (aggregated, active replicas)
numNodes (aggregated, number of live nodes)
Separate databases are created for each of these groups, and each database keeps data for
all metrics listed in that group.
Currently this list is not configurable. Also, if you change this list in the code then
all existing databases must be first removed from the .system collection because RRD4j doesn’t allow
adding new datasources once the database is created.
This component is responsible for managing in-memory databases and periodically saving them
to the .system collection. If the .system collection is not available the updates to the
databases will be kept in memory, until the time when .system collection becomes available.
If the .system collection is permanently unavailable then data will not be saved and it will
be lost when the Solr node is shut down.
This component provides a REST API for accessing the metrics history. It is also responsible for
collecting and periodically updating the in-memory databases.
This handler also performs aggregation of metrics on per-collection level, and on a cluster level.
By default only these aggregated metrics are tracked - historic data from each node and each replica
in each collection is not collected separately. Aggregated databases are managed on the Overseer leader
node but they are still accessible from other nodes even if they are not persisted - the handler redirects
the call from originating node to the current Overseer leader.
The handler assumes that a simple aggregation (sum of partial metric values from each resource) is
sufficient. This happens to make sense for the default built-in sets of metrics. Future extensions will
provide other aggregation strategies (such as, average, max, min, etc.).
This handler is automatically disabled when metrics collection is disabled using the
<metrics enabled="false"> element in solr.xml.
Metrics History Configuration
There are two ways to configure this subsystem:
/clusterprops.json - this is the primary mechanism. It uses the cluster properties JSON
file in ZooKeeper. Configuration is stored in the /metrics/history element in a JSON map.
solr.xml - this is the secondary mechanism, which is not recommended but provided for consistency
with the existing metrics configuration section in this file. Configuration is stored in the
/solr/metrics/history element of this file.
Currently the following configuration options are supported:
boolean, default is true.
If this is false then metrics history is not collected.
When this is true then metrics are periodically collected, aggregated and saved.
boolean, default is false. When this is true non-aggregated history will be
collected for each replica in each collection. When this is false then only aggregated history
is collected for each collection.
boolean, default is false. When this is true then non-aggregated history will be
collected separately for each node (for node and JVM metrics), with database names consisting of
base registry name with appended node name, e.g., solr.jvm.localhost:8983_solr. When this is false
then only aggregated history will be collected in a single solr.jvm and solr.node cluster-wide
integer, in seconds, default is 60. Metrics values will be collected and respective
databases updated every collectPeriod seconds.
Value of collectPeriod must be at least 1, and if it’s changed then all previously existing databases
with their historic data must be manually removed (new databases will be created automatically).
integer, in seconds, default is 60. Data from modified databases will be saved to Solr
every syncPeriod seconds. When accessing the databases via REST API in index mode the visibility of
most recent data depends on this period, because requests accessing the data from other nodes see only
the version of the data that is stored in the .system collection.
Example /clusterprops.json file with metrics history configuration that turns on the collection of
per-node metrics history for node and JVM metrics. Typically this file will also contain other
properties unrelated to Metrics History API.
Main entry point for accessing metrics history is /admin/metrics/history (or /api/cluster/metrics/history
for v2 API).
The following sections describe actions available in this API. All calls have at least one
required parameter action.
All responses contain a section named state, which reports the current internal state of the API:
boolean, corresponds to the enableReplicas configuration setting.
boolean, corresponds to the enableNodes configuration setting.
one of the following values:
inactive - when metrics collection is disabled (but access to existing metrics history is still available).
memory - when metrics history is kept only in memory because .system collection doesn’t exist. In this mode
clients can access metrics history available on the node that received the request and on the Overseer leader.
index - when metrics history is periodically stored in the .system collection. Data available in memory on
the node that accepted the request is retrieved from memory, any other data is retrieved from the
.system collection (so it’s at least syncPeriod old).
Also, the response header section (responseHeader) contains zkConnected boolean property that indicates
whether the current node is a part of SolrCloud cluster.
The query parameter action=list produces a list of available databases. It supports the following parameters:
optional integer, default is 500. Maximum number of results to return.
In this SolrCloud example the API is in memory mode, and the request was made to a node that is
not Overseer leader. The API transparently forwarded the request to Overseer leader.
Note the presence of the node element in each section, which shows where the information is coming
from - when API is in memory mode this indicates which results are local and which ones are retrieved
from the Overseer leader node. When the API is in index mode this element always shows the node name that
received the request (because the data is retrieved from the .system collection anyway).
Each section also contains a lastModified element, which contains the last modification time when the
database was update. All timestamps returned from this API correspond to Unix epoch time in seconds.
The query parameter action=status provides detailed status of the selected database.
The query parameter action=get retrieves all data collected in the specified database.
The following parameters are supported:
string, required: database name.
string, optional, default is list. Format of the data. Currently the
following formats are supported:
list - each datapoint is returned as separate JSON element. For efficiency, for each
datasource in a database for each time series the timestamps are provided separately from
values (because points from all datasources in a given time series share the same timestamps).
string - all datapoint values and timestamps are returned as strings, with values separated by new line character.
graph - data is returned as PNG images, Base64-encoded, containing graphs of each time series values over time.
In each case the response is structured in a similar way: archive identifiers are keys in a JSON map,
all data is placed in a data element, with timestamps / datapoints / graphs as values in lists or maps.