Package org.apache.solr.cloud
Class Overseer
- java.lang.Object
-
- org.apache.solr.cloud.Overseer
-
- All Implemented Interfaces:
Closeable
,AutoCloseable
,org.apache.solr.common.SolrCloseable
public class Overseer extends Object implements org.apache.solr.common.SolrCloseable
Cluster leader. Responsible for processing state updates, node assignments, creating/deleting collections, shards, replicas and setting various properties.The Overseer is a single elected node in the SolrCloud cluster that is in charge of interactions with ZooKeeper that require global synchronization.
The Overseer deals with:
- Cluster State updates, i.e. updating Collections'
state.json
files in ZooKeeper, seeOverseer.ClusterStateUpdater
, - Collection API implementation, see
OverseerCollectionConfigSetProcessor
andOverseerCollectionMessageHandler
(and the example below), - Updating Config Sets, see
OverseerCollectionConfigSetProcessor
andOverseerConfigSetMessageHandler
,
The nodes in the cluster communicate with the Overseer over queues implemented in ZooKeeper. There are essentially two queues:
- The state update queue, through which nodes request the Overseer to update the
state.json
file of a Collection in ZooKeeper. This queue is in Zookeeper at/overseer/queue
, - A queue shared between Collection API and Config Set API requests. This queue is in
Zookeeper at
/overseer/collection-queue-work
.
An example of the steps involved in the Overseer processing a Collection creation API call:
- Client uses the Collection API with
CREATE
action and reaches a node of the cluster, - The node (via
CollectionsHandler
) enqueues the request into the/overseer/collection-queue-work
queue in ZooKeepeer, - The
OverseerCollectionConfigSetProcessor
running on the Overseer node dequeues the message and using an executor service with a maximum pool size ofOverseerTaskProcessor.MAX_PARALLEL_TASKS
hands it for processing toOverseerCollectionMessageHandler
, - Command
CreateCollectionCmd
then executes and does:- Update some state directly in ZooKeeper (creating collection znode),
- Compute replica placement on available nodes in the cluster,
- Enqueue a state change request for creating the
state.json
file for the collection in ZooKeeper. This is done by enqueuing a message in/overseer/queue
, - The command then waits for the update to be seen in ZooKeeper...
- The
Overseer.ClusterStateUpdater
(also running on the Overseer node) dequeues the state change message and creates thestate.json
file in ZooKeeper for the Collection. All the work of the cluster state updater (creations, updates, deletes) is done sequentially for the whole cluster by a single thread. - The
CreateCollectionCmd
sees the state change in ZooKeeper and:- Builds and sends requests to each node to create the appropriate cores for all the
replicas of all shards of the collection. Nodes create the replicas and set them to
Replica.State.ACTIVE
.
- Builds and sends requests to each node to create the appropriate cores for all the
replicas of all shards of the collection. Nodes create the replicas and set them to
- The collection creation command has succeeded from the Overseer perspective,
CollectionsHandler
checks the replicas in Zookeeper and verifies they are allReplica.State.ACTIVE
,- The client receives a success return.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static interface
Overseer.Message
static class
Overseer.OverseerThread
-
Field Summary
Fields Modifier and Type Field Description static int
NUM_RESPONSES_TO_STORE
static String
OVERSEER_ELECT
static String
QUEUE_OPERATION
static int
STATE_UPDATE_BATCH_SIZE
static int
STATE_UPDATE_DELAY
static int
STATE_UPDATE_MAX_QUEUE
-
Constructor Summary
Constructors Constructor Description Overseer(HttpShardHandler shardHandler, UpdateShardHandler updateShardHandler, String adminPath, org.apache.solr.common.cloud.ZkStateReader reader, ZkController zkController, CloudConfig config)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description void
close()
CoreContainer
getCoreContainer()
DistributedClusterStateUpdater
getDistributedClusterStateUpdater()
org.apache.solr.client.solrj.cloud.SolrCloudManager
getSolrCloudManager()
Stats
getStats()
StopClusterSingleton
plugins when we lose leadership.Overseer.OverseerThread
getUpdaterThread()
For tests.org.apache.solr.common.cloud.ZkStateReader
getZkStateReader()
boolean
isClosed()
void
offerStateUpdate(byte[] data)
void
offerStateUpdate(org.apache.solr.common.MapWriter mw)
void
sendQuitToOverseer(String overseerId)
This method enqueues a QUIT message to the overseer of given id.void
start(String id)
void
submit(Overseer.Message message)
Submit an intra-process message which will be picked up and executed whenOverseer.ClusterStateUpdater
's loop runs next timevoid
systemCollectionCompatCheck(BiConsumer<String,Object> consumer)
-
-
-
Field Detail
-
QUEUE_OPERATION
public static final String QUEUE_OPERATION
- See Also:
- Constant Field Values
-
STATE_UPDATE_DELAY
public static final int STATE_UPDATE_DELAY
-
STATE_UPDATE_BATCH_SIZE
public static final int STATE_UPDATE_BATCH_SIZE
-
STATE_UPDATE_MAX_QUEUE
public static final int STATE_UPDATE_MAX_QUEUE
-
NUM_RESPONSES_TO_STORE
public static final int NUM_RESPONSES_TO_STORE
- See Also:
- Constant Field Values
-
OVERSEER_ELECT
public static final String OVERSEER_ELECT
- See Also:
- Constant Field Values
-
-
Constructor Detail
-
Overseer
public Overseer(HttpShardHandler shardHandler, UpdateShardHandler updateShardHandler, String adminPath, org.apache.solr.common.cloud.ZkStateReader reader, ZkController zkController, CloudConfig config) throws org.apache.zookeeper.KeeperException, InterruptedException
- Throws:
org.apache.zookeeper.KeeperException
InterruptedException
-
-
Method Detail
-
start
public void start(String id)
-
systemCollectionCompatCheck
public void systemCollectionCompatCheck(BiConsumer<String,Object> consumer)
-
getStats
public Stats getStats()
StopClusterSingleton
plugins when we lose leadership.
-
getCoreContainer
public CoreContainer getCoreContainer()
-
getSolrCloudManager
public org.apache.solr.client.solrj.cloud.SolrCloudManager getSolrCloudManager()
-
getDistributedClusterStateUpdater
public DistributedClusterStateUpdater getDistributedClusterStateUpdater()
-
getUpdaterThread
public Overseer.OverseerThread getUpdaterThread()
For tests.- Returns:
- state updater thread
- NOTE: This API is for internal purposes only and might change in incompatible ways in the next release.
-
close
public void close()
- Specified by:
close
in interfaceAutoCloseable
- Specified by:
close
in interfaceCloseable
-
isClosed
public boolean isClosed()
- Specified by:
isClosed
in interfaceorg.apache.solr.common.SolrCloseable
-
getZkStateReader
public org.apache.solr.common.cloud.ZkStateReader getZkStateReader()
-
offerStateUpdate
public void offerStateUpdate(org.apache.solr.common.MapWriter mw) throws org.apache.zookeeper.KeeperException, InterruptedException
- Throws:
org.apache.zookeeper.KeeperException
InterruptedException
-
offerStateUpdate
public void offerStateUpdate(byte[] data) throws org.apache.zookeeper.KeeperException, InterruptedException
- Throws:
org.apache.zookeeper.KeeperException
InterruptedException
-
submit
public void submit(Overseer.Message message)
Submit an intra-process message which will be picked up and executed whenOverseer.ClusterStateUpdater
's loop runs next time
-
sendQuitToOverseer
public void sendQuitToOverseer(String overseerId) throws org.apache.zookeeper.KeeperException, InterruptedException
This method enqueues a QUIT message to the overseer of given id. Effect is similar to building the message then callingofferStateUpdate(org.apache.solr.common.MapWriter)
but this method can legitimately be called when cluster state update is distributed (and Overseer cluster state updater not really used) whileofferStateUpdate(org.apache.solr.common.MapWriter)
is not. Note that sending "QUIT" to overseer is not a cluster state update and was likely added to this queue because it was simpler.- Throws:
org.apache.zookeeper.KeeperException
InterruptedException
-
-