Package org.apache.solr.cloud

Class Overseer

  • All Implemented Interfaces:
    Closeable, AutoCloseable, SolrCloseable
    public class Overseer
extends Object
implements SolrCloseable
    Cluster leader. Responsible for processing state updates, node assignments, creating/deleting collections, shards, replicas and setting various properties.

    The Overseer is a single elected node in the SolrCloud cluster that is in charge of interactions with ZooKeeper that require global synchronization.

    The Overseer deals with:

    The nodes in the cluster communicate with the Overseer over queues implemented in ZooKeeper. There are essentially two queues:

    1. The state update queue, through which nodes request the Overseer to update the state.json file of a Collection in ZooKeeper. This queue is in Zookeeper at /overseer/queue,
    2. A queue shared between Collection API and Config Set API requests. This queue is in Zookeeper at /overseer/collection-queue-work.

    An example of the steps involved in the Overseer processing a Collection creation API call:

    1. Client uses the Collection API with CREATE action and reaches a node of the cluster,
    2. The node (via CollectionsHandler) enqueues the request into the /overseer/collection-queue-work queue in ZooKeepeer,
    3. The OverseerCollectionConfigSetProcessor running on the Overseer node dequeues the message and using an executor service with a maximum pool size of OverseerTaskProcessor.MAX_PARALLEL_TASKS hands it for processing to OverseerCollectionMessageHandler,
    4. Command CreateCollectionCmd then executes and does:
      1. Update some state directly in ZooKeeper (creating collection znode),
      2. Compute replica placement on available nodes in the cluster,
      3. Enqueue a state change request for creating the state.json file for the collection in ZooKeeper. This is done by enqueuing a message in /overseer/queue ,
      4. The command then waits for the update to be seen in ZooKeeper...
    5. The Overseer.ClusterStateUpdater (also running on the Overseer node) dequeues the state change message and creates the state.json file in ZooKeeper for the Collection. All the work of the cluster state updater (creations, updates, deletes) is done sequentially for the whole cluster by a single thread.
    6. The CreateCollectionCmd sees the state change in ZooKeeper and:
      1. Builds and sends requests to each node to create the appropriate cores for all the replicas of all shards of the collection. Nodes create the replicas and set them to Replica.State.ACTIVE.
    7. The collection creation command has succeeded from the Overseer perspective,
    8. CollectionsHandler checks the replicas in Zookeeper and verifies they are all Replica.State.ACTIVE,
    9. The client receives a success return.