SolrCloud Autoscaling Listeners

Trigger Listeners allow users to configure additional behavior related to trigger events as they are being processed.

For example, users may want to record autoscaling events to an external system, or notify an administrator when a particular type of event occurs or when its processing reaches certain stage (e.g., failed).

Listener configuration always refers to a specific trigger configuration because a listener is notified of events generated by that specific trigger. Several (or none) named listeners can be registered for a trigger, and they will be notified in the order in which they were defined.

Listener configuration can specify what processing stages are of interest, and when an event enters this processing stage the listener will be notified. Currently the following stages are recognized:

  • STARTED - when an event has been generated by a trigger and its processing is starting.
  • ABORTED - when event was being processed while the source trigger closed.
  • BEFORE_ACTION - when a TriggerAction is about to be invoked. Action name and the current ActionContext are passed to the listener.
  • AFTER_ACTION - after a TriggerAction has been successfully invoked. Action name, ActionContext and the list of action names invoked so far are passed to the listener.
  • FAILED - when event processing failed (or when a TriggerAction failed)
  • SUCCEEDED - when event processing completes successfully

Listener configuration can also specify what particular actions are of interest, both before and/or after they are invoked.

Listener Configuration

Currently the following listener configuration properties are supported:

name
(string, required) A unique listener configuration name.
trigger
(string, required) The name of an existing trigger configuration.
class
(string, required) A listener implementation class name.
stage
(list of strings, optional, ignored case) A list of processing stages that this listener should be notified. Default is empty list.
beforeAction
(list of strings, optional) A list of action names (as defined in trigger configuration) before which the listener will be notified. Default is empty list.
afterAction
(list of strings, optional) A list of action names after which the listener will be notified. Default is empty list.
Additional implementation-specific properties may be provided, depending on the listener implementation.

Note: when both stage and beforeAction / afterAction lists are non-empty then the listener will be notified both when a specified stage is entered and before / after specified actions.

Managing Listener Configurations

Listener configurations are managed using the Autoscaling Write API, and using set-listener and remove-listener commands.

For example:

{
 "set-listener": {
    "name": "foo",
    "trigger": "node_lost_trigger",
    "stage": ["STARTED", "ABORTED", "SUCCEEDED", "FAILED"],
    "class": "solr.SystemLogListener"
 }
}
{
 "remove-listener": {
    "name": "foo"
 }
}

Listener Implementations

Trigger listeners must implement the TriggerListener interface. Solr provides some implementations of trigger listeners, which cover common use cases. These implementations are described below, together with their configuration parameters.

SystemLogListener

This trigger listener sends trigger events and processing context as documents for indexing in SolrCloud .system collection.

When a trigger configuration is first created, a corresponding trigger listener configuration that uses SystemLogListener is also automatically created, to make sure that all events and actions related to the autoscaling framework are logged to the .system collection.

Supported configuration properties:

collection

(string, optional) Specifies the target collection where documents are sent. Default value is .system. If the target collection is missing the listener will silently discard events.

In rare situations when the target collection is in an unstable state (e.g., when some leader replicas were just lost and the leader election hasn’t finished running yet), the listener may not be able to index some events. In such cases a WARN message with the details of the event(s) will be added to the regular logs.
enabled
(boolean, optional) Enables the listener when true. Default value is true.

Documents created by this listener have several predefined fields:

  • id - time-based random id
  • type - always set to autoscaling_event
  • source_s - always set to SystemLogListener
  • timestamp - current time when document was created
  • stage_s - current stage of event processing
  • action_s - current action name, if available
  • message_t - optional additional message
  • error.message_t - message from Throwable, if available
  • error.details_t - stacktrace from Throwable, if available
  • before.actions_ss - list of action names to be invoked so far
  • after.actions_ss - list of action names that have been successfully invoked so far
  • event_str - JSON representation of all event properties
  • context_str - JSON representation of all ActionContext properties, if available

The following fields are created using the information from trigger event:

  • event.id_s - event id
  • event.type_s - event type
  • event.source_s - event source (trigger name)
  • event.time_l - Unix time when the event was created (may significantly differ from the time when it was actually processed)
  • event.property.* - additional fields that represent other arbitrary event properties. These fields use either _s or _ss suffix depending on whether the property value is a collection (values inside collection are treated as strings, there’s no recursive flattening)

The following configuration is used for the automatically created listener (in this case for a trigger named foo):

{
 "name" : "foo.system",
 "trigger" : "solr.SystemLogListener",
 "stage" : ["STARTED", "ABORTED", "SUCCEEDED", "FAILED", "BEFORE_ACTION", "AFTER_ACTION"]
}

HttpTriggerListener

This listener uses HTTP POST to send a representation of the event and context to a specified URL. The URL, payload, and headers may contain property substitution patterns, which are then replaced with values taken from the current event or context properties.

Templates use the same syntax as property substitution in Solr configuration files, e.g., ${foo.bar:baz} means that the value of foo.bar property should be taken, and baz should be used if the value is absent.

Supported configuration properties:

url
(string, required) A URL template.
payload
(string, optional) A payload template. If absent, a JSON map of all properties listed above will be used.
contentType
(string, optional) A payload content type. If absent then application/json will be used.
header.*
(string, optional) A header template(s). The name of the property without "header." prefix defines the literal header name.
timeout
(int, optional) Connection and socket timeout in milliseconds. Default is 60000 milliseconds (60 seconds).
followRedirects
(boolean, optional) Allows following redirects. Default is false.

The following properties are available in context and can be referenced from templates:

  • config.* - listener configuration properties
  • event.* - current event properties
  • stage - current stage of event processing
  • actionName - optional current action name
  • context.* - optional ActionContext properties
  • error - optional error string (from Throwable.toString())
  • message - optional message
Example HttpTriggerListener
{
 "name": "foo",
 "trigger": "node_added_trigger",
 "class": "solr.HttpTriggerListener",
 "url": "http://foo.com/${config.name:invalidName}/${config.properties.xyz:invalidXyz}/${event.eventType}",
 "xyz": "foobar",
 "header.X-Trigger": "${config.trigger}",
 "payload": "actionName=${actionName}, source=${event.source}, type=${event.eventType}",
 "contentType": "text/plain",
 "stage": ["STARTED", "ABORTED", "SUCCEEDED", "FAILED"],
 "beforeAction": ["compute_plan", "execute_plan"],
 "afterAction": ["compute_plan", "execute_plan"]
}

This configuration specifies that each time one of the listed stages is reached, or before and after each of the listed actions is executed, the listener will send the templated payload to a URL that also depends on the configuration and the current event, and with a custom header that indicates the trigger name.