IndexConfig in SolrConfig

ramBufferSizeMB

Once accumulated document updates exceed this much memory space (defined in megabytes), then the pending updates are flushed. This can also create new segments or trigger a merge. Using this setting is generally preferable to maxBufferedDocs. If both maxBufferedDocs and ramBufferSizeMB are set in solrconfig.xml, then a flush will occur when either limit is reached. The default is 100Mb.

<ramBufferSizeMB>100</ramBufferSizeMB>

maxBufferedDocs

Sets the number of document updates to buffer in memory before they are flushed as a new segment. This may also trigger a merge. The default Solr configuration sets to flush by RAM usage (ramBufferSizeMB).

<maxBufferedDocs>1000</maxBufferedDocs>

useCompoundFile

Controls whether newly written (and not yet merged) index segments should use the Compound File Segments format. The default is false.

<useCompoundFile>false</useCompoundFile>

ramPerThreadHardLimitMB

Sets the maximum memory (defined in megabytes) consumption per thread triggering a forced flush if exceeded. The given value must be greater than 0 and less than 2GB (2048 MB).

<ramPerThreadHardLimitMB>1945</ramPerThreadHardLimitMB>

mergePolicyFactory

Defines how merging segments is done.

The default in Solr is to use a TieredMergePolicy, which merges segments of approximately equal size, subject to an allowed number of segments per tier.

Other policies available are the LogByteSizeMergePolicy, LogDocMergePolicy, and UninvertDocValuesMergePolicy. For more information on these policies, please see the MergePolicy javadocs.

<mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory">
  <int name="maxMergeAtOnce">10</int>
  <int name="segmentsPerTier">10</int>
  <double name="forceMergeDeletesPctAllowed">10.0</double>
  <double name="deletesPctAllowed">33.0</double>

</mergePolicyFactory>

Controlling Segment Sizes

The most common adjustment users make to the configuration of TieredMergePolicy (or LogByteSizeMergePolicy) are the "merge factors" to change how many segments should be merged at one time and, in the TieredMergePolicy case, the maximum size of an merged segment.

For TieredMergePolicy, this is controlled by setting the maxMergeAtOnce (default 10), segmentsPerTier (default 10) and maxMergedSegmentMB (default 5000) options.

LogByteSizeMergePolicy has a single mergeFactor option (default 10).

To understand why these options are important, consider what happens when an update is made to an index using LogByteSizeMergePolicy: Documents are always added to the most recently opened segment. When a segment fills up, a new segment is created and subsequent updates are placed there.

If creating a new segment would cause the number of lowest-level segments to exceed the mergeFactor value, then all those segments are merged together to form a single large segment. Thus, if the merge factor is 10, each merge results in the creation of a single segment that is roughly ten times larger than each of its ten constituents. When there are 10 of these larger segments, then they in turn are merged into an even larger single segment. This process can continue indefinitely.

When using TieredMergePolicy, the process is the same, but instead of a single mergeFactor value, the segmentsPerTier setting is used as the threshold to decide if a merge should happen, and the maxMergeAtOnce setting determines how many segments should be included in the merge.

Choosing the best merge factors is generally a trade-off of indexing speed vs. searching speed. Having fewer segments in the index generally accelerates searches, because there are fewer places to look. It also can also result in fewer physical files on disk. But to keep the number of segments low, merges will occur more often, which can add load to the system and slow down updates to the index.

Conversely, keeping more segments can accelerate indexing, because merges happen less often, making an update is less likely to trigger a merge. But searches become more computationally expensive and will likely be slower, because search terms must be looked up in more index segments. Faster index updates also means shorter commit turnaround times, which means more timely search results.

Controlling Deleted Document Percentages

When a document is deleted or updated, the document is marked as deleted but it not removed from the index until the segment is merged. There are two parameters that can can be adjusted when using the default TieredMergePolicy that influence the number of deleted documents in an index.

Customizing Merge Policies

If the configuration options for the built-in merge policies do not fully suit your use case, you can customize them: either by creating a custom merge policy factory that you specify in your configuration, or by configuring a merge policy wrapper which uses a wrapped.prefix configuration option to control how the factory it wraps will be configured:

<mergePolicyFactory class="org.apache.solr.index.SortingMergePolicyFactory">
  <str name="sort">timestamp desc</str>
  <str name="wrapped.prefix">inner</str>
  <str name="inner.class">org.apache.solr.index.TieredMergePolicyFactory</str>
  <int name="inner.maxMergeAtOnce">10</int>
  <int name="inner.segmentsPerTier">10</int>
</mergePolicyFactory>

The example above shows Solr’s SortingMergePolicyFactory being configured to sort documents in merged segments by "timestamp desc", and wrapped around a TieredMergePolicyFactory configured to use the values maxMergeAtOnce=10 and segmentsPerTier=10 via the inner prefix defined by SortingMergePolicyFactory 's wrapped.prefix option. For more information on using SortingMergePolicyFactory, see the segmentTerminateEarly parameter.

mergeScheduler

The merge scheduler controls how merges are performed. The default ConcurrentMergeScheduler performs merges in the background using separate threads. The alternative, SerialMergeScheduler, does not perform merges with separate threads.

The ConcurrentMergeScheduler has the following configurable attributes:

The defaults for the above attributes are dynamically set based on whether the underlying disk drive is rotational disk or not. Refer to the Dynamic defaults for ConcurrentMergeScheduler section for more details.

<mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler"/>

<mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler">
  <int name="maxMergeCount">9</int>
  <int name="maxThreadCount">4</int>
</mergeScheduler>

mergedSegmentWarmer

When using Solr in for Near Real Time Searching a merged segment warmer can be configured to warm the reader on the newly merged segment, before the merge commits. This is not required for near real-time search, but will reduce search latency on opening a new near real-time reader after a merge completes.

<mergedSegmentWarmer class="org.apache.lucene.index.SimpleMergedSegmentWarmer"/>

lockType

The LockFactory options specify the locking implementation to use.

The set of valid lock type options depends on the DirectoryFactory you have configured.

The values listed below are are supported by StandardDirectoryFactory (the default):

<lockType>native</lockType>

writeLockTimeout

The maximum time to wait for a write lock on an IndexWriter. The default is 1000, expressed in milliseconds.

<writeLockTimeout>1000</writeLockTimeout>