Class Overseer

  • All Implemented Interfaces:
    Closeable, AutoCloseable, org.apache.solr.common.SolrCloseable

    public class Overseer
    extends Object
    implements org.apache.solr.common.SolrCloseable
    Cluster leader. Responsible for processing state updates, node assignments, creating/deleting collections, shards, replicas and setting various properties.

    The Overseer is a single elected node in the SolrCloud cluster that is in charge of interactions with ZooKeeper that require global synchronization.

    The Overseer deals with:

    The nodes in the cluster communicate with the Overseer over queues implemented in ZooKeeper. There are essentially two queues:

    1. The state update queue, through which nodes request the Overseer to update the state.json file of a Collection in ZooKeeper. This queue is in Zookeeper at /overseer/queue,
    2. A queue shared between Collection API and Config Set API requests. This queue is in Zookeeper at /overseer/collection-queue-work.

    An example of the steps involved in the Overseer processing a Collection creation API call:

    1. Client uses the Collection API with CREATE action and reaches a node of the cluster,
    2. The node (via CollectionsHandler) enqueues the request into the /overseer/collection-queue-work queue in ZooKeepeer,
    3. The OverseerCollectionConfigSetProcessor running on the Overseer node dequeues the message and using an executor service with a maximum pool size of OverseerTaskProcessor.MAX_PARALLEL_TASKS hands it for processing to OverseerCollectionMessageHandler,
    4. Command CreateCollectionCmd then executes and does:
      1. Update some state directly in ZooKeeper (creating collection znode),
      2. Compute replica placement on available nodes in the cluster,
      3. Enqueue a state change request for creating the state.json file for the collection in ZooKeeper. This is done by enqueuing a message in /overseer/queue ,
      4. The command then waits for the update to be seen in ZooKeeper...
    5. The Overseer.ClusterStateUpdater (also running on the Overseer node) dequeues the state change message and creates the state.json file in ZooKeeper for the Collection. All the work of the cluster state updater (creations, updates, deletes) is done sequentially for the whole cluster by a single thread.
    6. The CreateCollectionCmd sees the state change in ZooKeeper and:
      1. Builds and sends requests to each node to create the appropriate cores for all the replicas of all shards of the collection. Nodes create the replicas and set them to Replica.State.ACTIVE.
    7. The collection creation command has succeeded from the Overseer perspective,
    8. CollectionsHandler checks the replicas in Zookeeper and verifies they are all Replica.State.ACTIVE,
    9. The client receives a success return.
    • Field Detail

      • STATE_UPDATE_DELAY

        public static final int STATE_UPDATE_DELAY
      • STATE_UPDATE_BATCH_SIZE

        public static final int STATE_UPDATE_BATCH_SIZE
      • STATE_UPDATE_MAX_QUEUE

        public static final int STATE_UPDATE_MAX_QUEUE
    • Method Detail

      • start

        public void start​(String id)
      • systemCollectionCompatCheck

        public void systemCollectionCompatCheck​(BiConsumer<String,​Object> consumer)
      • getSolrCloudManager

        public org.apache.solr.client.solrj.cloud.SolrCloudManager getSolrCloudManager()
      • getUpdaterThread

        public Overseer.OverseerThread getUpdaterThread()
        For tests.
        Returns:
        state updater thread
        NOTE: This API is for internal purposes only and might change in incompatible ways in the next release.
      • isClosed

        public boolean isClosed()
        Specified by:
        isClosed in interface org.apache.solr.common.SolrCloseable
      • getZkStateReader

        public org.apache.solr.common.cloud.ZkStateReader getZkStateReader()
      • offerStateUpdate

        public void offerStateUpdate​(org.apache.solr.common.MapWriter mw)
                              throws org.apache.zookeeper.KeeperException,
                                     InterruptedException
        Throws:
        org.apache.zookeeper.KeeperException
        InterruptedException
      • submit

        public void submit​(Overseer.Message message)
        Submit an intra-process message which will be picked up and executed when Overseer.ClusterStateUpdater's loop runs next time