SolrCloud Autoscaling Triggers

Trigger Properties

Trigger configuration consists of the following properties:

Additional implementation-specific properties may be provided, as described in the sections for individual triggers below.

Action Properties

Action configuration consists of the following properties:

Additional implementation-specific properties may be provided, as described in the sections for individual triggers below.

If the actions configuration is omitted, then by default, the ComputePlanAction and the ExecutePlanAction are automatically added to the trigger configuration.

Example Trigger Configuration

This simple example shows the configuration for adding (or updating) a trigger for nodeAdded events.

{
 "set-trigger": {
  "name" : "node_added_trigger",
  "event" : "nodeAdded",
  "waitFor" : "1s",
  "enabled" : true,
  "actions" : [
   {
    "name" : "compute_plan",
    "class": "solr.ComputePlanAction"
   },
   {
    "name" : "custom_action",
    "class": "com.example.CustomAction"
   },
   {
    "name" : "execute_plan",
    "class": "solr.ExecutePlanAction"
   }
  ]
 }
}

This trigger configuration will compute and execute a plan to allocate the resources available on the new node. A custom action could also be used to possibly modify the plan.

Node Added Trigger

The NodeAddedTrigger generates nodeAdded events when a node joins the cluster. It can be used to either move replicas from other nodes to the new node or to add new replicas.

In addition to the parameters described at Trigger Configuration, this trigger supports one more parameter:

{
  "set-trigger": {
    "name": "node_added_trigger",
    "event": "nodeAdded",
    "waitFor": "5s"
  }
}

{
  "set-trigger": {
    "name": "node_added_trigger",
    "event": "nodeAdded",
    "waitFor": "5s",
    "preferredOperation": "ADDREPLICA",
    "replicaType": "PULL"
  }
}

Node Lost Trigger

The NodeLostTrigger generates nodeLost events when a node leaves the cluster. It can be used to either move replicas that were hosted by the lost node to other nodes or to delete them from the cluster.

In addition to the parameters described at Trigger Configuration, this trigger supports the one more parameter:

{
  "set-trigger": {
    "name": "node_lost_trigger",
    "event": "nodeLost",
    "waitFor": "120s"
  }
}

{
  "set-trigger": {
    "name": "node_lost_trigger",
    "event": "nodeLost",
    "waitFor": "120s",
    "preferredOperation": "DELETENODE"
  }
}

Auto Add Replicas Trigger

When a collection has the parameter autoAddReplicas set to true then a trigger configuration named .auto_add_replicas is automatically created to watch for nodes going away. This trigger produces nodeLost events, which are then processed by configured actions (usually resulting in computing and executing a plan to add replicas on the live nodes to maintain the expected replication factor).

Refer to the section Autoscaling Automatically Adding Replicas to learn more about how the .autoAddReplicas trigger works.

In addition to the parameters described at Trigger Configuration, this trigger supports one parameter, which is defined in the <solrcloud> section of solr.xml:

Metric Trigger

The metric trigger can be used to monitor any metric exposed by the Metrics API. It supports lower and upper threshold configurations as well as optional filters to limit operation to specific collection, shards, and nodes.

In addition to the parameters described at Trigger Configuration, this trigger supports the following parameters:

{
  "set-trigger": {
    "name": "metric_trigger",
    "event": "metric",
    "waitFor": "5s",
    "metrics": "metric:solr.node:CONTAINER.fs.coreRoot.usableSpace",
    "below": 107374182400,
    "collection": "mycollection"
  }
}

Index Size Trigger

This trigger can be used for monitoring the size of collection shards, measured either by the number of documents in a shard or the physical size of the shard’s index in bytes.

When either of the upper thresholds is exceeded for a particular shard the trigger will generate an event with a (configurable) requested operation to perform on the offending shards - by default this is a SPLITSHARD operation.

Similarly, when either of the lower thresholds is exceeded the trigger will generate an event with a (configurable) requested operation to perform on two of the smallest shards. By default this is a MERGESHARDS operation, and is currently ignored because that operation is not yet implemented (see SOLR-9407).

When splitMethod=link is used the resulting sub-shards will initially have nearly the same size as the parent shard due to the hard-linking of parent index files, and will differ just in the lists of deleted documents. In order to correctly recognize the effectively reduced index size an estimate is calculated using a simple formula: indexCommitSize * numDocs / maxDoc. This value is then compared with aboveBytes and belowBytes limits.

Additionally, monitoring can be restricted to a list of collections; by default all collections are monitored.

In addition to the parameters described at Trigger Configuration, this trigger supports the following configuration parameters (all thresholds are exclusive):

Events generated by this trigger contain additional details about the shards that exceeded thresholds and the types of violations (upper / lower bounds, bytes / docs metrics).

{
 "set-trigger": {
  "name" : "index_size_trigger",
  "event" : "indexSize",
  "collections" : "test1,test2",
  "aboveBytes" : 1000000000,
  "aboveDocs" : 1000000000,
  "belowBytes" : 200000,
  "belowDocs" : 200000,
  "belowOp" : "NONE",
  "waitFor" : "1m",
  "enabled" : true,
  "actions" : [
   {
    "name" : "compute_plan",
    "class": "solr.ComputePlanAction"
   },
   {
    "name" : "execute_plan",
    "class": "solr.ExecutePlanAction"
   }
  ]
 }
}

Search Rate Trigger

The search rate trigger can be used for monitoring search rates in a selected collection (1-min average rate by default), and request that either replicas be moved from "hot nodes" to different nodes, or new replicas be added to "hot shards" to reduce the per-replica search rate for a collection or shard with hot spots.

Similarly, if the search rate falls below a threshold then the trigger may request that some replicas are deleted from "cold" shards. It can also optionally issue node-level action requests when a cumulative node-level rate falls below a threshold.

Per-shard rates are calculated as arithmetic average of rates of all searchable replicas in a given shard. This method was chosen to avoid generating false events when a simple client keeps sending requests to a single specific replica (because adding or removing other replicas can’t solve this situation, only proper load balancing can - either by using CloudSolrClient or another load-balancing client).

This trigger calculates node-level cumulative rates using per-replica rates reported by replicas that are part of monitored collections / shards on each node. This means that it may report some nodes as "cold" (underutilized) because it ignores other, perhaps more active, replicas belonging to other collections. Also, nodes that don’t host any of the monitored replicas or those that are explicitly excluded by node configuration property won’t be reported at all.

In addition to the parameters described at Trigger Configuration, this trigger supports the following configuration properties:

{
 "set-trigger": {
  "name" : "search_rate_trigger",
  "event" : "searchRate",
  "collections" : "test",
  "metric" : "QUERY./select.requestTimes:5minRate",
  "aboveRate" : 100.0,
  "belowRate" : 0.01,
  "belowNodeRate" : 0.01,
  "belowNodeOp" : "DELETENODE",
  "minReplicas" : 1,
  "waitFor" : "20m",
  "enabled" : true,
  "actions" : [
   {
    "name" : "compute_plan",
    "class": "solr.ComputePlanAction"
   },
   {
    "name" : "execute_plan",
    "class": "solr.ExecutePlanAction"
   }
  ]
 }
}

Scheduled Trigger

The Scheduled trigger generates events according to a fixed rate schedule.

In addition to the parameters described at Trigger Configuration, this trigger supports the following configuration:

This trigger applies the every date math expression on the startTime or the last event time to derive the next scheduled time and if current time is greater than next scheduled time but within graceTime then an event is generated.

Apart from the common event properties described in the Event Types section, the trigger adds an additional actualEventTime event property which has the actual event time as opposed to the scheduled time.

For example, if the scheduled time was 2018-01-31T15:30:00Z and grace time was +15MINUTES then an event may be fired at 2018-01-31T15:45:00Z. Such an event will have eventTime as 2018-01-31T15:30:00Z, the scheduled time, but the actualEventTime property will have a value of 2018-01-31T15:45:00Z, the actual time.

Auto-add Replicas Trigger

The default configuration and functionality of this trigger is described in detail in the section titled Automatically Adding Replicas.

Scheduled Maintenance Trigger

This is a Scheduled Trigger named .scheduled_maintenance and it’s configured to run once per day. It executes the following actions:

solr.ExecutePlanAction

This action simply executes any collection admin requests generated by other actions - in particular, in the default configuration it executes DELETESHARD requests produced by solr.InactiveShardPlanAction, as described above.