Major Changes in Solr 8

When JSON data is sent to Solr with nested child documents split using the split parameter, the child documents will now be associated to their parents by the field/label string used in the JSON instead of anonymously.

Most users probably won’t notice the distinction since the label is lost unless special fields are in the schema. This choice used to be toggleable with an internal/expert anonChildDocs parameter flag, which has been removed.

Deleting (or updating) documents by their uniqueKey is now scoped to only consider root documents, not child/nested documents. Thus a delete-by-id won’t work on a child document (it will fail silently), and an attempt to update a child document by providing a new document with the same ID would add a new document (which will probably be erroneous).

Both these actions were and still are problematic. In-place-updates are safe though. If you want to delete certain child documents and if you know they don’t themselves have nested children then you must do so with a delete-by-query technique.

If you explicitly use BM25SimilarityFactory in your schema, the absolute scoring will be lower since Lucene changed the calculation of BM25 to remove a multiplication factor (for technical details, see LUCENE-8563 or SOLR-13025). Ordering of documents will not change in the normal case. Use LegacyBM25SimilarityFactory if you need to force the old 6.x/7.x scoring.

Note that if you have not specified any similarityFactory in the schema, or use the default SchemaSimilarityFactory, then LegacyBM25Similarity is automatically selected when the value for luceneMatchVersion is lower than 8.0.0.

See also the section Similarity for more information.

Memory codecs have been removed from Lucene (MemoryPostings, MemoryDocValues) and are no longer available in Solr. If you used postingsFormat="Memory" or docValuesFormat="Memory" on any field or field type configuration then either remove that setting to use the default or experiment with one of the other options.

For more information on defining a codec, see the section Codec Factory; for more information on field properties, see the section Field Type Definitions and Properties.

Index-time boosts were removed from Lucene in version 7.0, and in Solr 7.x the syntax was still allowed (although it logged a warning in the logs). The syntax was similar to:

{"id":"1", "val_s":{"value":"foo", "boost":2.0}}

This syntax has been removed entirely and if sent to Solr it will now produce an error. This was done in conjunction with the improvements for nested document support.

The date format patterns used by ParseDateFieldUpdateProcessorFactory (used by default in "schemaless mode") are now interpreted by Java 8’s java.time.DateTimeFormatter instead of Joda Time. The pattern language is very similar but not the same. Typically, simply update the pattern by changing an uppercase 'Z' to lowercase 'z' and that’s it.

For the current recommended set of patterns in schemaless mode, see the section Schemaless Mode, or simply examine the _default configSet (found in server/solr/configsets).

Also note that the default set of date patterns (formats) have expanded from previous releases to subsume those patterns previously handled by the "extract" contrib (Solr Cell / Tika).

The default replica placement strategy used in Solr has been reverted to the "legacy" policy used by Solr 7.4 and previous versions. This is due to multiple bugs in the autoscaling based replica placement strategy that was made default in Solr 7.5 which causes multiple replicas of the same shard to be placed on the same node in addition to the maxShardsPerNode and createNodeSet parameters being ignored.

Although the default has changed, autoscaling will continue to be used if a cluster policy or preference is specified or a collection level policy is in use.

The default replica placement strategy can be changed to use autoscaling again by setting a cluster property:

curl -X POST -H 'Content-type:application/json' --data-binary '
  {
    "set-obj-property": {
      "defaults" : {
        "cluster": {
          "useLegacyReplicaAssignment":false
        }
      }
    }
  }' http://$SOLR_HOST:$SOLR_PORT/api/cluster

A new command-line option is available via bin/solr autoscaling to calculate autoscaling policy suggestions and diagnostic information outside of the running Solr cluster. This option can use the existing autoscaling policy, or test the impact of a new one from a file located on the server filesystem.

These options have been documented in the section Testing Autoscaling Configuration and Suggestions.

The Admin UI now presents a login screen for any users with authentication enabled on their cluster. Clusters with Basic Authentication will prompt users to enter a username and password. On clusters configured to use Kerberos Authentication, authentication is handled transparently by the browser as before, but if authentication fails, users will be directed to configure their browser to provide an appropriate Kerberos ticket.

The login screen’s purpose is cosmetic only - Admin UI-triggered Solr requests were subject to authentication prior to 7.7 and still are today. The login screen changes only the user experience of providing this authentication.

The shards parameter, used to manually select the shards and replicas that receive distributed requests, now checks nodes against a whitelist of acceptable values for security reasons.

In SolrCloud mode this whitelist is automatically configured to contain all live nodes. In standalone mode the whitelist is empty by default. Upgrading users who use the shards parameter in standalone mode can correct this value by setting the shardsWhitelist property in any shardHandler configurations in their solrconfig.xml file.

For more information, see the Distributed Request documentation.

The JSON parameter to set cluster-wide default cluster properties with the CLUSTERPROP command has changed.

The old syntax nested the defaults into a property named clusterDefaults. The new syntax uses only defaults. The command to use is still set-obj-property.

An example of the new syntax is:

{
  "set-obj-property": {
    "defaults" : {
      "collection": {
        "numShards": 2,
        "nrtReplicas": 1,
        "tlogReplicas": 1,
        "pullReplicas": 1
      }
    }
  }
}

The old syntax will be supported until at least Solr 9, but users are advised to begin using the new syntax as soon as possible.

An autoscaling policy is now used as the default strategy for selecting nodes on which new replicas or replicas of new collections are created.

A default policy is now in place for all users, which will sort nodes by the number of cores and available freedisk, which means by default a node with the fewest number of cores already on it and the highest available freedisk will be selected for new core creation.

1. It removes the restriction against using maxShardsPerNode when an autoscaling policy is in place. This parameter can now always be set when creating a collection.

2. It removes the default setting of maxShardsPerNode=1 when an autoscaling policy is in place. It will be set correctly (if required) regardless of whether an autoscaling policy is in place or not.

   The default value of maxShardsPerNode is still 1. It can be set to -1 if the old behavior of unlimited maxShardsPerNode is desired.

Lucene has introduced the ByteBuffersDirectoryFactory as a replacement for the RAMDirectoryFactory, which will be removed in Solr 9.

While most users are still encouraged to use the NRTCachingDirectoryFactory, which allows Lucene to select the best directory factory to use, if you have explicitly configured Solr to use the RAMDirectoryFactory, you are encouraged to switch to the new implementation as soon as possible before Solr 9 is released.

For more information about the new directory factory, see the Jira issue LUCENE-8438.

For more information about the directory factory configuration in Solr, see the section DataDir and DirectoryFactory in SolrConfig.

When using the TieredMergePolicy, the default merge policy for Solr, optimize and expungeDeletes now respect the maxMergedSegmentMB configuration parameter, which defaults to 5000 (5GB).

If it is absolutely necessary to control the number of segments present after optimize, specify maxSegments as a positive integer. Setting maxSegments higher than 1 are honored on a "best effort" basis.

The TieredMergePolicy will also reclaim resources from segments that exceed maxMergedSegmentMB more aggressively than earlier.

The old Leader-In-Recovery implementation (implemented in Solr 4.9) is now deprecated and replaced. Solr will support rolling upgrades from old 7.x versions of Solr to future 7.x releases until the last release of the 7.x major version.

This means to upgrade to Solr 8 in the future, you will need to be on Solr 7.3 or higher.

Starting a query string with local parameters {!myparser …​} is used to switch from one query parser to another, and is intended for use by Solr system developers, not end users doing searches. To reduce negative side-effects of unintended hack-ability, Solr now limits the cases when local parameters will be parsed to only contexts in which the default parser is "lucene" or "func".

So, if defType=edismax then q={!myparser …​} won’t work. In that example, put the desired query parser into the defType parameter.

Another example is if deftype=edismax then hl.q={!myparser …​} won’t work for the same reason. In this example, either put the desired query parser into the hl.qparser parameter or set hl.qparser=lucene. Most users won’t run into these cases but some will need to change.

If you must have full backwards compatibility, use luceneMatchVersion=7.1.0 or an earlier version.

The eDisMax parser by default no longer allows subqueries that specify a Solr parser using either local parameters, or the older _query_ magic field trick.

For example, {!prefix f=myfield v=enterp} or _query_:"{!prefix f=myfield v=enterp}" are not supported by default any longer. If you want to allow power-users to do this, set uf=* query or some other value that includes _query_.

If you need full backwards compatibility for the time being, use luceneMatchVersion=7.1.0 or something earlier.

The feature to automatically add replicas if a replica goes down, previously available only when storing indexes in HDFS, has been ported to the autoscaling framework. Due to this, autoAddReplicas is now available to all users even if their indexes are on local disks.

Existing users of this feature should not have to change anything. However, they should note these changes:

• Behavior: Changing the autoAddReplicas property from disabled (false) to enabled (true) using MODIFYCOLLECTION API no longer replaces down replicas for the collection immediately. Instead, replicas are only added if a node containing them went down while autoAddReplicas was enabled. The parameters autoReplicaFailoverBadNodeExpiration and autoReplicaFailoverWorkLoopDelay are no longer used.

• Deprecations: Enabling/disabling autoAddReplicas cluster-wide with the API will be deprecated; use suspend/resume trigger APIs with name=".auto_add_replicas" instead.

  More information about the changes to this feature can be found in the section SolrCloud Automatically Adding Replicas.

• If a reporter configures the group="shard" attribute then please also configure the class="org.apache.solr.metrics.reporters.solr.SolrShardReporter" attribute.

• If a reporter configures the group="cluster" attribute then please also configure the class="org.apache.solr.metrics.reporters.solr.SolrClusterReporter" attribute.

  See the section Shard and Cluster Reporters for more information.