SolrCloud Autoscaling Triggers | Apache Solr Reference Guide 7.3

Triggers are used in autoscaling to watch for cluster events such as nodes joining, leaving, search rate or any other metric breaching a threshold.

In the future other cluster, node, and replica events that are important from the point of view of cluster performance will also have available triggers.

Trigger implementations verify the state of resources that they monitor. When they detect a change that merits attention they generate events, which are then queued and processed by configured TriggerAction implementations. This usually involves computing and executing a plan to manage the new cluster resources (e.g., move replicas). Solr provides predefined implementations of triggers for specific event types.

Triggers execute on the node that runs Overseer. They are scheduled to run periodically, at a default interval of 1 second between each execution (not every execution produces events).

Event Types

Currently the following event types (and corresponding trigger implementations) are defined:

nodeAdded: generated when a new node joins the cluster
nodeLost: generated when a node leaves the cluster
metric: generated when the configured metric crosses a configured lower or upper threshold value
searchRate: generated when the 1-minute average search rate exceeds configured upper threshold
scheduled: generated according to a scheduled time period such as every 24 hours etc

Events are not necessarily generated immediately after the corresponding state change occurred - the maximum rate of events is controlled by the waitFor configuration parameter (see below).

The following properties are common to all event types:

id: (string) A unique time-based event id.
eventType: (string) The type of event.
source: (string) The name of the trigger that produced this event.
eventTime: (long) Unix time when the condition that caused this event occurred. For example, for a nodeAdded event this will be the time when the node was added and not when the event was actually generated, which may significantly differ due to the rate limits set by waitFor.
properties: (map, optional) Any additional properties. Currently includes nodeName property that indicates the node that was lost or added.

Auto Add Replicas Trigger

When a collection has the parameter autoAddReplicas set to true then a trigger configuration named .auto_add_replicas is automatically created to watch for nodes going away. This trigger produces nodeLost events, which are then processed by configured actions (usually resulting in computing and executing a plan to add replicas on the live nodes to maintain the expected replication factor).

Refer to the section Autoscaling Automatically Adding Replicas to learn more about how the .autoAddReplicas trigger works.

This trigger supports one parameter, which is defined in the <solrcloud> section of solr.xml:

autoReplicaFailoverWaitAfterExpiration: The minimum time in milliseconds to wait for initiating replacement of a replica after first noticing it not being live. This is important to prevent false positives while stopping or starting the cluster. The default is 120000 (2 minutes). The value provided for this parameter is used as the value for the waitFor parameter in the .auto_add_replicas trigger.

See The <solrcloud> Element for more details about how to work with solr.xml.

Metric Trigger

The metric trigger can be used to monitor any metric exposed by the Metrics API. It supports lower and upper threshold configurations as well as optional filters to limit operation to specific collection, shards, and nodes.

This trigger supports the following configuration:

metric: (string, required) The metric property name to be watched in the format metrics:group:prefix, e.g., metric:solr.node:CONTAINER.fs.coreRoot.usableSpace.
below: (double, optional) The lower threshold for the metric value. The trigger produces a metric breached event if the metric’s value falls below this value.
above: (double, optional) The upper threshold for the metric value. The trigger produces a metric breached event if the metric’s value crosses above this value.
collection: (string, optional) The collection used to limit the nodes on which the given metric is watched. When the metric is breached, trigger actions will limit operations to this collection only.
shard: (string, optional) The shard used to limit the nodes on which the given metric is watched. When the metric is breached, trigger actions will limit operations to this shard only.
node: (string, optional) The node on which the given metric is watched. Trigger actions will operate on this node only.
preferredOperation: (string, optional, defaults to MOVEREPLICA) The operation to be performed in response to an event generated by this trigger. By default, replicas will be moved from the hot node to others. The only other supported value is ADDREPLICA which adds more replicas if the metric is breached.

Example: a metric trigger that fires when total usable space on a node having replicas of "mycollection" falls below 100GB

{
  "set-trigger": {
    "name": "metric_trigger",
    "event": "metric",
    "waitFor": "5s",
    "metric": "metric:solr.node:CONTAINER.fs.coreRoot.usableSpace",
    "below": 107374182400,
    "collection": "mycollection"
  }
}

Search Rate Trigger

The search rate trigger can be used for monitoring 1-minute average search rates in a selected collection, and request that either replicas be moved to different nodes or new replicas be added to reduce the per-replica search rate for a collection or shard with search rate hot spots. (Future versions of Solr will also be able to automatically remove some replicas when search rate falls below the configured lower threshold).

This trigger support the following configuration:

collection: (string, optional) collection name to monitor, or any collection if empty.
shard: (string, optional) shard name within the collection (requires collection to be set), or any shard if empty.
node: (string, optional) node name to monitor, or any if empty.
handler: (string, optional) handler name whose request rate represents the search rate (default is /select). This name is used for creating the full metric key, in this case solr.core.<coreName>:QUERY./select.requestTimes:1minRate.
rate: (double, required) the upper bound for the request rate metric value.

If a rate is exceeded for a node (but not for individual replicas placed on this node) then the action requested by this event is to move one replica (with the highest rate) to another node. If a rate is exceeded for a collection or shard then the action requested is to add some replicas - currently at least 1 and at most 3, depending on how much the rate is exceeded, proportional to the threshold rate and the current request rate.

Example: a search rate trigger that monitors collection "test" and adds new replicas if 1-minute average request rate of "/select" handler exceeds 100 requests/sec:

{
 "set-trigger": {
  "name" : "search_rate_trigger",
  "event" : "searchRate",
  "collection" : "test",
  "handler" : "/select",
  "rate" : 100.0,
  "waitFor" : "1m",
  "enabled" : true,
  "actions" : [
   {
    "name" : "compute_plan",
    "class": "solr.ComputePlanAction"
   },
   {
    "name" : "execute_plan",
    "class": "solr.ExecutePlanAction"
   }
  ]
 }
}

Scheduled Trigger

The Scheduled trigger generates events according to a fixed rate schedule.

The trigger supports the following configuration:

startTime

(string, required) The start date/time of the schedule. This should either be a DateMath string e.g., 'NOW', or be an ISO-8601 date time string (the same standard used during search and indexing in Solr, which defaults to UTC), or be specified without the trailing 'Z' accompanied with the timeZone parameter. For example, each of the following values are acceptable:

2018-01-31T15:30:00Z: ISO-8601 date time string. The trailing Z signals that the time is in UTC
NOW+5MINUTES: Solr’s date math string
2018-01-31T15:30:00: No trailing 'Z' signals that the timeZone parameter must be specified to avoid ambiguity

every

(string, required) A positive Solr date math string which is added to the startTime or the last run time to arrive at the next scheduled time.

graceTime

(string, optional) A positive Solr date math string. This is the additional grace time over the scheduled time within which the trigger is allowed to generate an event.

timeZone

(string, optional) A time zone string which is used for calculating the scheduled times.

preferredOp

(string, optional, defaults to MOVEREPLICA) The preferred operation to perform in response to an event generated by this trigger. The only supported values are MOVEREPLICA or ADDREPLICA.

This trigger applies the every date math expression on the startTime or the last event time to derive the next scheduled time and if current time is greater than next scheduled time but within graceTime then an event is generated.

Apart from the common event properties described in the Event Types section, the trigger adds an additional actualEventTime event property which has the actual event time as opposed to the scheduled time.

For example, if the scheduled time was 2018-01-31T15:30:00Z and grace time was +15MINUTES then an event may be fired at 2018-01-31T15:45:00Z. Such an event will have eventTime as 2018-01-31T15:30:00Z, the scheduled time, but the actualEventTime property will have a value of 2018-01-31T15:45:00Z, the actual time.

Trigger Configuration

Trigger configurations are managed using the Autoscaling Write API and the commands set-trigger, remove-trigger, suspend-trigger, and resume-trigger.

Trigger configuration consists of the following properties:

name: (string, required) A unique trigger configuration name.
event: (string, required) One of the predefined event types (nodeAdded or nodeLost).
actions: (list of action configs, optional) An ordered list of actions to execute when event is fired.
waitFor: (string, optional) The time to wait between generating new events, as an integer number immediately followed by unit symbol, one of s (seconds), m (minutes), or h (hours). Default is 0s.
enabled: (boolean, optional) When true the trigger is enabled. Default is true.

Additional implementation-specific properties may be provided.

Action configuration consists of the following properties:

name: (string, required) A unique name of the action configuration.
class: (string, required) The action implementation class.

Additional implementation-specific properties may be provided

If the actions configuration is omitted, then by default, the ComputePlanAction and the ExecutePlanAction are automatically added to the trigger configuration.

Example: adding or updating a trigger for nodeAdded events

{
 "set-trigger": {
  "name" : "node_added_trigger",
  "event" : "nodeAdded",
  "waitFor" : "1s",
  "enabled" : true,
  "actions" : [
   {
    "name" : "compute_plan",
    "class": "solr.ComputePlanAction"
   },
   {
    "name" : "custom_action",
    "class": "com.example.CustomAction"
   },
   {
    "name" : "execute_plan",
    "class": "solr.ExecutePlanAction"
   }
  ]
 }
}

This trigger configuration will compute and execute a plan to allocate the resources available on the new node. A custom action is also used to possibly modify the plan.

Autoscaling Policy and Preferences SolrCloud Autoscaling Trigger Actions