Package org.apache.solr.cloud

Class Overseer

java.lang.Object
org.apache.solr.cloud.Overseer
All Implemented Interfaces:
Closeable, AutoCloseable, org.apache.solr.common.SolrCloseable
public class Overseer extends Object implements org.apache.solr.common.SolrCloseable
Cluster leader. Responsible for processing state updates, node assignments, creating/deleting collections, shards, replicas and setting various properties.

The Overseer is a single elected node in the SolrCloud cluster that is in charge of interactions with ZooKeeper that require global synchronization.

The Overseer deals with:

The nodes in the cluster communicate with the Overseer over queues implemented in ZooKeeper. There are essentially two queues:

  1. The state update queue, through which nodes request the Overseer to update the state.json file of a Collection in ZooKeeper. This queue is in Zookeeper at /overseer/queue,
  2. A queue shared between Collection API and Config Set API requests. This queue is in Zookeeper at /overseer/collection-queue-work.

An example of the steps involved in the Overseer processing a Collection creation API call:

  1. Client uses the Collection API with CREATE action and reaches a node of the cluster,
  2. The node (via CollectionsHandler) enqueues the request into the /overseer/collection-queue-work queue in ZooKeepeer,
  3. The OverseerCollectionConfigSetProcessor running on the Overseer node dequeues the message and using an executor service with a maximum pool size of OverseerTaskProcessor.MAX_PARALLEL_TASKS hands it for processing to OverseerCollectionMessageHandler,
  4. Command CreateCollectionCmd then executes and does:
    1. Update some state directly in ZooKeeper (creating collection znode),
    2. Compute replica placement on available nodes in the cluster,
    3. Enqueue a state change request for creating the state.json file for the collection in ZooKeeper. This is done by enqueuing a message in /overseer/queue ,
    4. The command then waits for the update to be seen in ZooKeeper...
  5. The Overseer.ClusterStateUpdater (also running on the Overseer node) dequeues the state change message and creates the state.json file in ZooKeeper for the Collection. All the work of the cluster state updater (creations, updates, deletes) is done sequentially for the whole cluster by a single thread.
  6. The CreateCollectionCmd sees the state change in ZooKeeper and:
    1. Builds and sends requests to each node to create the appropriate cores for all the replicas of all shards of the collection. Nodes create the replicas and set them to Replica.State.ACTIVE.
  7. The collection creation command has succeeded from the Overseer perspective,
  8. CollectionsHandler checks the replicas in Zookeeper and verifies they are all Replica.State.ACTIVE,
  9. The client receives a success return.

  • Field Details

    • QUEUE_OPERATION

      public static final String QUEUE_OPERATION
      See Also:

    • STATE_UPDATE_DELAY

      public static final int STATE_UPDATE_DELAY

    • STATE_UPDATE_BATCH_SIZE

      public static final int STATE_UPDATE_BATCH_SIZE

    • STATE_UPDATE_MAX_QUEUE

      public static final int STATE_UPDATE_MAX_QUEUE

    • NUM_RESPONSES_TO_STORE

      public static final int NUM_RESPONSES_TO_STORE
      See Also:

    • OVERSEER_ELECT

      public static final String OVERSEER_ELECT
      See Also:

  • Constructor Details

  • Method Details

    • start

      public void start(String id)

    • getStats

      public Stats getStats()
      Stop ClusterSingleton plugins when we lose leadership.

    • getCoreContainer

      public CoreContainer getCoreContainer()

    • getSolrCloudManager

      public org.apache.solr.client.solrj.cloud.SolrCloudManager getSolrCloudManager()

    • getDistributedClusterStateUpdater

      public DistributedClusterStateUpdater getDistributedClusterStateUpdater()

    • getUpdaterThread

      public Overseer.OverseerThread getUpdaterThread()
      For tests.
      Returns:
      state updater thread
      NOTE: This API is for internal purposes only and might change in incompatible ways in the next release.

    • close

      public void close()
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable

    • isClosed

      public boolean isClosed()
      Specified by:
      isClosed in interface org.apache.solr.common.SolrCloseable

    • getZkStateReader

      public org.apache.solr.common.cloud.ZkStateReader getZkStateReader()

    • offerStateUpdate

      public void offerStateUpdate(org.apache.solr.common.MapWriter mw) throws org.apache.zookeeper.KeeperException, InterruptedException
      Throws:
      org.apache.zookeeper.KeeperException
      InterruptedException

    • offerStateUpdate

      public void offerStateUpdate(byte[] data) throws org.apache.zookeeper.KeeperException, InterruptedException
      Throws:
      org.apache.zookeeper.KeeperException
      InterruptedException

    • submit

      public void submit(Overseer.Message message)
      Submit an intra-process message which will be picked up and executed when Overseer.ClusterStateUpdater's loop runs next time

    • sendQuitToOverseer

      public void sendQuitToOverseer(String overseerId) throws org.apache.zookeeper.KeeperException, InterruptedException
      This method enqueues a QUIT message to the overseer of given id. Effect is similar to building the message then calling offerStateUpdate(org.apache.solr.common.MapWriter) but this method can legitimately be called when cluster state update is distributed (and Overseer cluster state updater not really used) while offerStateUpdate(org.apache.solr.common.MapWriter) is not. Note that sending "QUIT" to overseer is not a cluster state update and was likely added to this queue because it was simpler.
      Throws:
      org.apache.zookeeper.KeeperException
      InterruptedException