Package org.apache.solr.cloud

Class Overseer

  • All Implemented Interfaces:
    Closeable, AutoCloseable, org.apache.solr.common.SolrCloseable
    public class Overseer
extends Object
implements org.apache.solr.common.SolrCloseable
    Cluster leader. Responsible for processing state updates, node assignments, creating/deleting collections, shards, replicas and setting various properties.

    The Overseer is a single elected node in the SolrCloud cluster that is in charge of interactions with ZooKeeper that require global synchronization.

    The Overseer deals with:

    The nodes in the cluster communicate with the Overseer over queues implemented in ZooKeeper. There are essentially two queues:

    1. The state update queue, through which nodes request the Overseer to update the state.json file of a Collection in ZooKeeper. This queue is in Zookeeper at /overseer/queue,
    2. A queue shared between Collection API and Config Set API requests. This queue is in Zookeeper at /overseer/collection-queue-work.

    An example of the steps involved in the Overseer processing a Collection creation API call:

    1. Client uses the Collection API with CREATE action and reaches a node of the cluster,
    2. The node (via CollectionsHandler) enqueues the request into the /overseer/collection-queue-work queue in ZooKeepeer,
    3. The OverseerCollectionConfigSetProcessor running on the Overseer node dequeues the message and using an executor service with a maximum pool size of OverseerTaskProcessor.MAX_PARALLEL_TASKS hands it for processing to OverseerCollectionMessageHandler,
    4. Command CreateCollectionCmd then executes and does:
      1. Update some state directly in ZooKeeper (creating collection znode),
      2. Compute replica placement on available nodes in the cluster,
      3. Enqueue a state change request for creating the state.json file for the collection in ZooKeeper. This is done by enqueuing a message in /overseer/queue ,
      4. The command then waits for the update to be seen in ZooKeeper...
    5. The Overseer.ClusterStateUpdater (also running on the Overseer node) dequeues the state change message and creates the state.json file in ZooKeeper for the Collection. All the work of the cluster state updater (creations, updates, deletes) is done sequentially for the whole cluster by a single thread.
    6. The CreateCollectionCmd sees the state change in ZooKeeper and:
      1. Builds and sends requests to each node to create the appropriate cores for all the replicas of all shards of the collection. Nodes create the replicas and set them to Replica.State.ACTIVE.
    7. The collection creation command has succeeded from the Overseer perspective,
    8. CollectionsHandler checks the replicas in Zookeeper and verifies they are all Replica.State.ACTIVE,
    9. The client receives a success return.

    • Field Detail

      • STATE_UPDATE_DELAY

        public static final int STATE_UPDATE_DELAY

      • STATE_UPDATE_BATCH_SIZE

        public static final int STATE_UPDATE_BATCH_SIZE

    • Method Detail

      • start

        public void start​(String id)

      • systemCollectionCompatCheck

        public void systemCollectionCompatCheck​(BiConsumer<String,​Object> consumer)

      • getSolrCloudManager

        public org.apache.solr.client.solrj.cloud.SolrCloudManager getSolrCloudManager()

      • getUpdaterThread

        public Overseer.OverseerThread getUpdaterThread()
        For tests.
        Returns:
        state updater thread
        NOTE: This API is for internal purposes only and might change in incompatible ways in the next release.

      • isClosed

        public boolean isClosed()
        Specified by:
        isClosed in interface org.apache.solr.common.SolrCloseable

      • getZkStateReader

        public org.apache.solr.common.cloud.ZkStateReader getZkStateReader()

      • submit

        public void submit​(Overseer.Message message)
        Submit an intra-process message which will be picked up and executed when Overseer.ClusterStateUpdater's loop runs next time

      • sendQuitToOverseer

        public void sendQuitToOverseer​(String overseerId)
                        throws org.apache.zookeeper.KeeperException,
                               InterruptedException
        This method enqueues a QUIT message to the overseer of given id. Effect is similar to building the message then calling offerStateUpdate(byte[]) but this method can legitimately be called when cluster state update is distributed (and Overseer cluster state updater not really used) while offerStateUpdate(byte[]) is not. Note that sending "QUIT" to overseer is not a cluster state update and was likely added to this queue because it was simpler.
        Throws:
        org.apache.zookeeper.KeeperException
        InterruptedException