org.apache.solr.cloud.autoscaling.sim

Class SimClusterStateProvider

java.lang.Object

org.apache.solr.cloud.autoscaling.sim.SimClusterStateProvider

All Implemented Interfaces:

Closeable, AutoCloseable, ClusterStateProvider, SolrCloseable

public class SimClusterStateProvider
extends Object
implements ClusterStateProvider

Simulated ClusterStateProvider.

The following behaviors are supported:

• using autoscaling policy for replica placements
• maintaining and up-to-date list of /live_nodes and nodeAdded / nodeLost markers
• running a simulated leader election on collection changes (with throttling), when needed
• maintaining an up-to-date /clusterstate.json (single file format), which also tracks replica states, leader election changes, replica property changes, etc. Note: this file is only written, but never read by the framework!
• maintaining an up-to-date /clusterprops.json. Note: this file is only written, but never read by the framework!

Field Summary

Fields

Modifier and Type	Field and Description
static long	DEFAULT_DOC_SIZE_BYTES

Constructor Summary

Constructors

Constructor and Description
SimClusterStateProvider(LiveNodesSet liveNodes, SimCloudManager cloudManager) The instance needs to be initialized using the sim* methods in order to ensure proper behavior, otherwise it will behave as a cluster with zero replicas.

Method Summary

Modifier and Type	Method and Description
void	close()
void	connect()
void	copyFrom(ClusterStateProvider other)
void	createSystemCollection()
Map<String,String>	getAliasProperties(String alias)
Map<String,Object>	getClusterProperties()
ClusterState	getClusterState()
Set<String>	getLiveNodes()
String	getPolicyNameByCollection(String coll)
ClusterState.CollectionRef	getState(String collection)
boolean	isClosed()
List<String>	resolveAlias(String alias)
void	simAddNode(String nodeId) Add a new node to the cluster.
void	simAddReplica(String nodeId, ReplicaInfo replicaInfo, boolean runLeaderElection) Add a replica.
void	simAddReplica(ZkNodeProps message, NamedList results) Add a new replica.
void	simCreateCollection(ZkNodeProps props, NamedList results) Create a new collection.
void	simCreateShard(ZkNodeProps message, NamedList results) Create a new shard.
void	simDeleteAllCollections() Remove all collections.
void	simDeleteCollection(String collection, String async, NamedList results) Delete a collection
void	simDeleteShard(ZkNodeProps message, NamedList results) Delete a shard.
Map<String,Map<String,Object>>	simGetCollectionStats()
String	simGetOverseerLeader()
String	simGetRandomNode() Get random node id.
String	simGetRandomNode(Random random) Get random node id.
ReplicaInfo	simGetReplicaInfo(String collection, String coreNode)
List<ReplicaInfo>	simGetReplicaInfos(String node) Return all replica infos for a node.
List<ReplicaInfo>	simGetReplicaInfos(String collection, String shard)
List<String>	simListCollections() List collections.
void	simMoveReplica(ZkNodeProps message, NamedList results) Move replica.
QueryResponse	simQuery(QueryRequest req)
void	simRemoveDeadNodes() Remove all replica information related to dead nodes.
boolean	simRemoveNode(String nodeId) Remove node from a cluster.
void	simRemoveReplica(String nodeId, String collection, String coreNodeName) Remove replica.
void	simResetLeaderThrottles() Reset the leader election throttles.
boolean	simRestoreNode(String nodeId) Restore a previously removed node.
void	simSetClusterProperties(Map<String,Object> properties) Set all cluster properties.
void	simSetClusterProperty(String key, Object value) Set a cluster property.
void	simSetClusterState(ClusterState initialState) Initialize from an existing cluster state
void	simSetCollectionProperties(String coll, Map<String,Object> properties) Set collection properties.
void	simSetCollectionProperty(String coll, String key, String value) Set collection property.
void	simSetCollectionValue(String collection, String key, Object value) Set per-collection value (eg.
void	simSetCollectionValue(String collection, String key, Object value, boolean delta, boolean divide) Set per-collection value (eg.
void	simSetReplicaValues(String node, Map<String,Map<String,List<ReplicaInfo>>> source, boolean overwrite)
void	simSetShardValue(String collection, String shard, String key, Object value) Set per-collection value (eg.
void	simSetShardValue(String collection, String shard, String key, Object value, boolean delta, boolean divide) Set per-collection value (eg.
void	simSetSliceProperties(String coll, String slice, Map<String,Object> properties) Set slice properties.
void	simSplitShard(ZkNodeProps message, NamedList results) Split a shard.
UpdateResponse	simUpdate(UpdateRequest req) Simulate an update by modifying replica metrics.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.solr.client.solrj.impl.ClusterStateProvider

getClusterProperty, getClusterProperty, getCollection, isRoutedAlias, resolveSimpleAlias

Field Detail

DEFAULT_DOC_SIZE_BYTES

public static final long DEFAULT_DOC_SIZE_BYTES

See Also:

Constant Field Values

Constructor Detail

SimClusterStateProvider

public SimClusterStateProvider(LiveNodesSet liveNodes,
                               SimCloudManager cloudManager)
                        throws Exception

The instance needs to be initialized using the sim* methods in order to ensure proper behavior, otherwise it will behave as a cluster with zero replicas.

Throws:

Exception

Method Detail

copyFrom

public void copyFrom(ClusterStateProvider other)
              throws Exception

Throws:

Exception

simSetClusterState

public void simSetClusterState(ClusterState initialState)
                        throws Exception

Initialize from an existing cluster state

Parameters:

initialState - initial cluster state

Throws:

Exception

simResetLeaderThrottles

public void simResetLeaderThrottles()

Reset the leader election throttles.

simGetRandomNode

public String simGetRandomNode()

Get random node id.

Returns:

one of the live nodes

simGetRandomNode

public String simGetRandomNode(Random random)

Get random node id.

Parameters:

random - instance of random.

Returns:

one of the live nodes

simAddNode

public void simAddNode(String nodeId)
                throws Exception

Add a new node to the cluster.

Parameters:

nodeId - unique node id

Throws:

Exception

simRemoveNode

public boolean simRemoveNode(String nodeId)
                      throws Exception

Remove node from a cluster. This is equivalent to a situation when a node is lost. All replicas that were assigned to this node are marked as DOWN.

Parameters:

nodeId - node id

Returns:

true if a node existed and was removed

Throws:

Exception

simRemoveDeadNodes

public void simRemoveDeadNodes()
                        throws Exception

Remove all replica information related to dead nodes.

Throws:

Exception

simGetOverseerLeader

public String simGetOverseerLeader()

simRestoreNode

public boolean simRestoreNode(String nodeId)
                       throws Exception

Restore a previously removed node. This also simulates a short replica recovery state.

Parameters:

nodeId - node id to restore

Returns:

true when this operation restored any replicas, false otherwise (empty node).

Throws:

Exception

simAddReplica

public void simAddReplica(ZkNodeProps message,
                          NamedList results)
                   throws Exception

Add a new replica. Note that if any details of a replica (node, coreNodeName, SolrCore name, etc) are missing they will be filled in using the policy framework.

Parameters:

message - replica details

results - result of the operation

Throws:

Exception

simAddReplica

public void simAddReplica(String nodeId,
                          ReplicaInfo replicaInfo,
                          boolean runLeaderElection)
                   throws Exception

Add a replica. Note that all details of the replica must be present here, including node, coreNodeName and SolrCore name.

Parameters:

nodeId - node id where the replica will be added

replicaInfo - replica info

runLeaderElection - if true then run a leader election after adding the replica.

Throws:

Exception

simRemoveReplica

public void simRemoveReplica(String nodeId,
                             String collection,
                             String coreNodeName)
                      throws Exception

Remove replica.

Parameters:

nodeId - node id

coreNodeName - coreNodeName

Throws:

Exception

simCreateCollection

public void simCreateCollection(ZkNodeProps props,
                                NamedList results)
                         throws Exception

Create a new collection. This operation uses policy framework for node and replica assignments.

Parameters:

props - collection details

results - results of the operation.

Throws:

Exception

simDeleteCollection

public void simDeleteCollection(String collection,
                                String async,
                                NamedList results)
                         throws Exception

Delete a collection

Parameters:

collection - collection name

async - async id

results - results of the operation

Throws:

Exception

simDeleteAllCollections

public void simDeleteAllCollections()
                             throws Exception

Remove all collections.

Throws:

Exception

simMoveReplica

public void simMoveReplica(ZkNodeProps message,
                           NamedList results)
                    throws Exception

Move replica. This uses a similar algorithm as MoveReplicaCmd moveNormalReplica(...) method.

Parameters:

message - operation details

results - operation results.

Throws:

Exception

simCreateShard

public void simCreateShard(ZkNodeProps message,
                           NamedList results)
                    throws Exception

Create a new shard. This uses a similar algorithm as CreateShardCmd.

Parameters:

message - operation details

results - operation results

Throws:

Exception

simSplitShard

public void simSplitShard(ZkNodeProps message,
                          NamedList results)
                   throws Exception

Split a shard. This uses a similar algorithm as SplitShardCmd, including simulating its quirks, and leaving the original parent slice in place.

Parameters:

message - operation details

results - operation results.

Throws:

Exception

simDeleteShard

public void simDeleteShard(ZkNodeProps message,
                           NamedList results)
                    throws Exception

Delete a shard. This uses a similar algorithm as DeleteShardCmd

Parameters:

message - operation details

results - operation results

Throws:

Exception

createSystemCollection

public void createSystemCollection()
                            throws IOException

Throws:

IOException

simUpdate

public UpdateResponse simUpdate(UpdateRequest req)
                         throws SolrException,
                                InterruptedException,
                                IOException

Simulate an update by modifying replica metrics. The following core metrics are updated:

• SEARCHER.searcher.numDocs - increased by added docs, decreased by deleteById and deleteByQuery
• SEARCHER.searcher.deletedDocs - decreased by deleteById and deleteByQuery by up to numDocs
• SEARCHER.searcher.maxDoc - always increased by the number of added docs.

IMPORTANT limitations:

• document replacements are always counted as new docs
• delete by ID always succeeds (unless numDocs == 0)
• deleteByQuery is not supported unless the query is *:*

Parameters:

req - update request. This request MUST have the collection param set.

Returns:

UpdateResponse

Throws:

SolrException - on errors, such as nonexistent collection or unsupported deleteByQuery

InterruptedException

IOException

simQuery

public QueryResponse simQuery(QueryRequest req)
                       throws SolrException,
                              InterruptedException,
                              IOException

Throws:

SolrException

InterruptedException

IOException

simSetClusterProperties

public void simSetClusterProperties(Map<String,Object> properties)
                             throws Exception

Set all cluster properties. This also updates the clusterprops.json data in DistribStateManager

Parameters:

properties - properties to set

Throws:

Exception

simSetClusterProperty

public void simSetClusterProperty(String key,
                                  Object value)
                           throws Exception

Set a cluster property. This also updates the clusterprops.json data in DistribStateManager

Parameters:

key - property name

value - property value

Throws:

Exception

simSetCollectionProperties

public void simSetCollectionProperties(String coll,
                                       Map<String,Object> properties)
                                throws Exception

Set collection properties.

Parameters:

coll - collection name

properties - properties

Throws:

Exception

simSetCollectionProperty

public void simSetCollectionProperty(String coll,
                                     String key,
                                     String value)
                              throws Exception

Set collection property.

Parameters:

coll - collection name

key - property name

value - property value

Throws:

Exception

simSetSliceProperties

public void simSetSliceProperties(String coll,
                                  String slice,
                                  Map<String,Object> properties)
                           throws Exception

Set slice properties.

Parameters:

coll - collection name

slice - slice name

properties - slice properties

Throws:

Exception

simSetCollectionValue

public void simSetCollectionValue(String collection,
                                  String key,
                                  Object value)
                           throws Exception

Set per-collection value (eg. a metric). This value will be applied to each replica.

Parameters:

collection - collection name

key - property name

value - property value

Throws:

Exception

simSetCollectionValue

public void simSetCollectionValue(String collection,
                                  String key,
                                  Object value,
                                  boolean delta,
                                  boolean divide)
                           throws Exception

Set per-collection value (eg. a metric). This value will be applied to each replica.

Parameters:

collection - collection name

key - property name

value - property value

divide - if the value is a Number and this param is true, then the value will be evenly divided by the number of replicas.

Throws:

Exception

simSetShardValue

public void simSetShardValue(String collection,
                             String shard,
                             String key,
                             Object value)
                      throws Exception

Set per-collection value (eg. a metric). This value will be applied to each replica in a selected shard.

Parameters:

collection - collection name

shard - shard name. If null then all shards will be affected.

key - property name

value - property value

Throws:

Exception

simSetShardValue

public void simSetShardValue(String collection,
                             String shard,
                             String key,
                             Object value,
                             boolean delta,
                             boolean divide)
                      throws Exception

Set per-collection value (eg. a metric). This value will be applied to each replica in a selected shard.

Parameters:

collection - collection name

shard - shard name. If null then all shards will be affected.

key - property name

value - property value

delta - if true then treat the numeric value as delta to add to the existing value (or set the value to delta if missing)

divide - if the value is a Number and this is true, then the value will be evenly divided by the number of replicas.

Throws:

Exception

simSetReplicaValues

public void simSetReplicaValues(String node,
                                Map<String,Map<String,List<ReplicaInfo>>> source,
                                boolean overwrite)

simGetReplicaInfos

public List<ReplicaInfo> simGetReplicaInfos(String node)

Return all replica infos for a node.

Parameters:

node - node id

Returns:

copy of the list of replicas on that node, or empty list if none

simGetReplicaInfos

public List<ReplicaInfo> simGetReplicaInfos(String collection,
                                            String shard)

simGetReplicaInfo

public ReplicaInfo simGetReplicaInfo(String collection,
                                     String coreNode)

simListCollections

public List<String> simListCollections()
                                throws InterruptedException

List collections.

Returns:

list of existing collections.

Throws:

InterruptedException

simGetCollectionStats

public Map<String,Map<String,Object>> simGetCollectionStats()
                                                     throws IOException,
                                                            InterruptedException

Throws:

IOException

InterruptedException

getState

public ClusterState.CollectionRef getState(String collection)

Specified by:

getState in interface ClusterStateProvider

getLiveNodes

public Set<String> getLiveNodes()

Specified by:

getLiveNodes in interface ClusterStateProvider

resolveAlias

public List<String> resolveAlias(String alias)

Specified by:

resolveAlias in interface ClusterStateProvider

getAliasProperties

public Map<String,String> getAliasProperties(String alias)

Specified by:

getAliasProperties in interface ClusterStateProvider

getClusterState

public ClusterState getClusterState()
                             throws IOException

Specified by:

getClusterState in interface ClusterStateProvider

Throws:

IOException

getClusterProperties

public Map<String,Object> getClusterProperties()

Specified by:

getClusterProperties in interface ClusterStateProvider

getPolicyNameByCollection

public String getPolicyNameByCollection(String coll)

Specified by:

getPolicyNameByCollection in interface ClusterStateProvider

connect

public void connect()

Specified by:

connect in interface ClusterStateProvider

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

isClosed

public boolean isClosed()

Specified by:

isClosed in interface SolrCloseable