Major Changes in Solr 8

When JSON data is sent to Solr with nested child documents split using the split parameter, the child documents will now be associated to their parents by the field/label string used in the JSON instead of anonymously.

Most users probably won’t notice the distinction since the label is lost unless special fields are in the schema. This choice used to be toggleable with an internal/expert anonChildDocs parameter flag, which has been removed.

Deleting (or updating) documents by their uniqueKey is now scoped to only consider root documents, not child/nested documents. Thus a delete-by-id won’t work on a child document (it will fail silently), and an attempt to update a child document by providing a new document with the same ID would add a new document (which will probably be erroneous).

Both these actions were and still are problematic. In-place-updates are safe though. If you want to delete certain child documents and if you know they don’t themselves have nested children then you must do so with a delete-by-query technique.

Solr has a new field in the _default configset, called _nest_path_. This field stores the path of the document in the hierarchy for non-root documents.

If you explicitly use BM25SimilarityFactory in your schema, the absolute scoring will be lower since Lucene changed the calculation of BM25 to remove a multiplication factor (for technical details, see LUCENE-8563 or SOLR-13025). Ordering of documents will not change in the normal case. Use LegacyBM25SimilarityFactory if you need to force the old 6.x/7.x scoring.

Note that if you have not specified any similarityFactory in the schema, or use the default SchemaSimilarityFactory, then LegacyBM25Similarity is automatically selected when the value for luceneMatchVersion is lower than 8.0.0.

See also the section Similarity for more information.

Memory codecs have been removed from Lucene (MemoryPostings, MemoryDocValues) and are no longer available in Solr. If you used postingsFormat="Memory" or docValuesFormat="Memory" on any field or field type configuration then either remove that setting to use the default or experiment with one of the other options.

For more information on defining a codec, see the section Codec Factory; for more information on field properties, see the section Field Type Definitions and Properties.

The LowerCaseTokenizer has been deprecated and is likely to be removed in Solr 9. Users are encouraged to use the LetterTokenizer and the LowerCaseFilter instead.

The _default configset now includes a ignored_* dynamic field rule.

Index-time boosts were removed from Lucene in version 7.0, and in Solr 7.x the syntax was still allowed (although it logged a warning in the logs). The syntax was similar to:

{"id":"1", "val_s":{"value":"foo", "boost":2.0}}

This syntax has been removed entirely and if sent to Solr it will now produce an error. This was done in conjunction with the improvements for nested document support.

The date format patterns used by ParseDateFieldUpdateProcessorFactory (used by default in "schemaless mode") are now interpreted by Java 8’s java.time.DateTimeFormatter instead of Joda Time. The pattern language is very similar but not the same. Typically, simply update the pattern by changing an uppercase 'Z' to lowercase 'z' and that’s it.

For the current recommended set of patterns in schemaless mode, see the section Schemaless Mode, or simply examine the _default configset (found in server/solr/configsets).

Also note that the default set of date patterns (formats) have expanded from previous releases to subsume those patterns previously handled by the "extract" contrib (Solr Cell / Tika).

The extraction contrib (Solr Cell) no longer does any date parsing, and thus no longer supports the date.formats parameter. To ensure date strings are properly parsed, use the ParseDateFieldUpdateProcessorFactory in your update chain. This update request processor is found by default with the "parse-date" update processor when running Solr in "schemaless mode".

The LanguageIdentifierUpdateProcessor base class in the langid contrib (found in contrib/langid) changed some method signatures. If you have a custom language identifier implementation you will need to adapt your code. See the Jira issue SOLR-11774 for details of the changes.

The Unified Highlighter parameter hl.weightMatches now defaults to true. See the section Highlighting for more information about Highlighter parameters.

The eDisMax query parser will now thrown an error when the qf parameter refers to a nonexistent field.

The Function Query Parser now returns scores that are equal to zero (0) when a negative value is produced. This change is due to the fact that Lucene now requires scores to be positive.

Authentication plugins can now intercept internode requests on a per-request basis.

The Basic Authentication plugin now has an option forwardCredentials to let Basic Auth headers be forwarded on inter-node requests in case of distributed search, instead of falling back to PKI.

Metrics are now reported for authentication requests.

The Radial Graph view of a Solr cluster when running in SolrCloud mode has been removed.

The Nodes view introduced in Solr 7.5 is now the default when choosing the "Cloud" tab in the left navigation menu.

The default replica placement strategy used in Solr has been reverted to the "legacy" policy used by Solr 7.4 and previous versions. This is due to multiple bugs in the autoscaling based replica placement strategy that was made default in Solr 7.5 which causes multiple replicas of the same shard to be placed on the same node in addition to the maxShardsPerNode and createNodeSet parameters being ignored.

Although the default has changed, autoscaling will continue to be used if a cluster policy or preference is specified or a collection level policy is in use.

The default replica placement strategy can be changed to use autoscaling again by setting a cluster property:

curl -X POST -H 'Content-type:application/json' --data-binary '
  {
    "set-obj-property": {
      "defaults" : {
        "cluster": {
          "useLegacyReplicaAssignment":false
        }
      }
    }
  }' http://$SOLR_HOST:$SOLR_PORT/api/cluster

A new command-line option is available via bin/solr autoscaling to calculate autoscaling policy suggestions and diagnostic information outside of the running Solr cluster. This option can use the existing autoscaling policy, or test the impact of a new one from a file located on the server filesystem.

These options have been documented in the section Testing Autoscaling Configuration and Suggestions.

All Hadoop dependencies have been upgraded to Hadoop 3.2.0 (from 2.7.2).

The Admin UI now presents a login screen for any users with authentication enabled on their cluster. Clusters with Basic Authentication will prompt users to enter a username and password. On clusters configured to use Kerberos Authentication, authentication is handled transparently by the browser as before, but if authentication fails, users will be directed to configure their browser to provide an appropriate Kerberos ticket.

The login screen’s purpose is cosmetic only - Admin UI-triggered Solr requests were subject to authentication prior to 7.7 and still are today. The login screen changes only the user experience of providing this authentication.

The shards parameter, used to manually select the shards and replicas that receive distributed requests, now checks nodes against a whitelist of acceptable values for security reasons.

In SolrCloud mode this whitelist is automatically configured to contain all live nodes. In standalone mode the whitelist is empty by default. Upgrading users who use the shards parameter in standalone mode can correct this value by setting the shardsWhitelist property in any shardHandler configurations in their solrconfig.xml file.

For more information, see the Distributed Request documentation.

The JSON parameter to set cluster-wide default cluster properties with the CLUSTERPROP command has changed.

The old syntax nested the defaults into a property named clusterDefaults. The new syntax uses only defaults. The command to use is still set-obj-property.

An example of the new syntax is:

{
  "set-obj-property": {
    "defaults" : {
      "collection": {
        "numShards": 2,
        "nrtReplicas": 1,
        "tlogReplicas": 1,
        "pullReplicas": 1
      }
    }
  }
}

The old syntax will be supported until at least Solr 9, but users are advised to begin using the new syntax as soon as possible.

The parameter min_rf has been deprecated and no longer needs to be provided in order to see the achieved replication factor. This information will now always be returned to the client with the response.

An autoscaling policy is now used as the default strategy for selecting nodes on which new replicas or replicas of new collections are created.

A default policy is now in place for all users, which will sort nodes by the number of cores and available freedisk, which means by default a node with the fewest number of cores already on it and the highest available freedisk will be selected for new core creation.

The change described above has two additional impacts on the maxShardsPerNode parameter:

1. It removes the restriction against using maxShardsPerNode when an autoscaling policy is in place. This parameter can now always be set when creating a collection.

2. It removes the default setting of maxShardsPerNode=1 when an autoscaling policy is in place. It will be set correctly (if required) regardless of whether an autoscaling policy is in place or not.

   The default value of maxShardsPerNode is still 1. It can be set to -1 if the old behavior of unlimited maxShardsPerNode is desired.

Lucene has introduced the ByteBuffersDirectoryFactory as a replacement for the RAMDirectoryFactory, which will be removed in Solr 9.

While most users are still encouraged to use the NRTCachingDirectoryFactory, which allows Lucene to select the best directory factory to use, if you have explicitly configured Solr to use the RAMDirectoryFactory, you are encouraged to switch to the new implementation as soon as possible before Solr 9 is released.

For more information about the new directory factory, see the Jira issue LUCENE-8438.

For more information about the directory factory configuration in Solr, see the section DataDir and DirectoryFactory in SolrConfig.

Since Solr 7.0, Solr’s schema field-guessing has created _str fields for all _txt fields, and returned those by default with queries. As of 7.5, _str fields will no longer be returned by default. They will still be available and can be requested with the fl parameter on queries. See also the section on field guessing for more information about how schema field guessing works.

The Standard Filter, which has been non-operational since at least Solr v4, has been removed.

When using the TieredMergePolicy, the default merge policy for Solr, optimize and expungeDeletes now respect the maxMergedSegmentMB configuration parameter, which defaults to 5000 (5GB).

If it is absolutely necessary to control the number of segments present after optimize, specify maxSegments as a positive integer. Setting maxSegments higher than 1 are honored on a "best effort" basis.

The TieredMergePolicy will also reclaim resources from segments that exceed maxMergedSegmentMB more aggressively than earlier.

The UIMA contrib has been removed from Solr and is no longer available.

Solr’s logging configuration file is now located in server/resources/log4j2.xml by default.

A bug for Windows users has been corrected. When using Solr’s examples (bin/solr start -e) log files will now be put in the correct location (example/ instead of server). See also Solr Examples and Solr Control Script Reference for more information.

Solr now uses Log4j v2.11. The Log4j configuration is now in log4j2.xml rather than log4j.properties files. This is a server side change only and clients using SolrJ won’t need any changes. Clients can still use any logging implementation which is compatible with SLF4J. We now let Log4j handle rotation of Solr logs at startup, and bin/solr start scripts will no longer attempt this nor move existing console or garbage collection logs into logs/archived either. See Configuring Logging for more details about Solr logging.

Configuring slowQueryThresholdMillis now logs slow requests to a separate file named solr_slow_requests.log. Previously they would get logged in the solr.log file.

In the master-slave model of scaling Solr, a slave no longer commits an empty index when a completely new index is detected on master during replication. To return to the previous behavior pass false to skipCommitOnMasterVersionZero in the slave section of replication handler configuration, or pass it to the fetchindex command.

Collections created without specifying a configset name have used a copy of the _default configset since Solr 7.0. Before 7.3, the copied configset was named the same as the collection name, but from 7.3 onwards it will be named with a new ".AUTOCREATED" suffix. This is to prevent overwriting custom configset names.

The rq parameter used with Learning to Rank rerank query parsing no longer considers the defType parameter. See Running a Rerank Query for more information about this parameter.

The behaviour of the autoscaling system will now pause all triggers from execution between the start of actions and the end of a cool down period. The triggers will resume after the cool down period expires. Previously, the cool down period was a fixed period started after actions for a trigger event completed and during this time all triggers continued to run but any events were rejected and tried later.

The throttling mechanism used to limit the rate of autoscaling events processed has been removed. This deprecates the actionThrottlePeriodSeconds setting in the set-properties Autoscaling API which is now non-operational. Use the triggerCooldownPeriodSeconds parameter instead to pause event processing.

The default value of autoReplicaFailoverWaitAfterExpiration, used with the AutoAddReplicas feature, has increased to 120 seconds from the previous default of 30 seconds. This affects how soon Solr adds new replicas to replace the replicas on nodes which have either crashed or shutdown.

The default Solr log file size and number of backups have been raised to 32MB and 10 respectively. See the section Configuring Logging for more information about how to configure logging.

The old Leader-In-Recovery implementation (implemented in Solr 4.9) is now deprecated and replaced. Solr will support rolling upgrades from old 7.x versions of Solr to future 7.x releases until the last release of the 7.x major version.

This means to upgrade to Solr 8 in the future, you will need to be on Solr 7.3 or higher.

Replicas which are not up-to-date are no longer allowed to become leader. Use the FORCELEADER command of the Collections API to allow these replicas become leader.

If you are using the spatial JTS library with Solr, you must upgrade to 1.15.0. This new version of JTS is now dual-licensed to include a BSD style license. See the section on Spatial Search for more information.

The top-level <highlighting> element in solrconfig.xml is now officially deprecated in favour of the equivalent <searchComponent> syntax. This element has been out of use in default Solr installations for several releases already.

Starting a query string with local parameters {!myparser …​} is used to switch from one query parser to another, and is intended for use by Solr system developers, not end users doing searches. To reduce negative side-effects of unintended hack-ability, Solr now limits the cases when local parameters will be parsed to only contexts in which the default parser is "lucene" or "func".

So, if defType=edismax then q={!myparser …​} won’t work. In that example, put the desired query parser into the defType parameter.

Another example is if deftype=edismax then hl.q={!myparser …​} won’t work for the same reason. In this example, either put the desired query parser into the hl.qparser parameter or set hl.qparser=lucene. Most users won’t run into these cases but some will need to change.

If you must have full backwards compatibility, use luceneMatchVersion=7.1.0 or an earlier version.

The eDisMax parser by default no longer allows subqueries that specify a Solr parser using either local parameters, or the older _query_ magic field trick.

For example, {!prefix f=myfield v=enterp} or _query_:"{!prefix f=myfield v=enterp}" are not supported by default any longer. If you want to allow power-users to do this, set uf=* query or some other value that includes _query_.

If you need full backwards compatibility for the time being, use luceneMatchVersion=7.1.0 or something earlier.

The feature to automatically add replicas if a replica goes down, previously available only when storing indexes in HDFS, has been ported to the autoscaling framework. Due to this, autoAddReplicas is now available to all users even if their indexes are on local disks.

Existing users of this feature should not have to change anything. However, they should note these changes:

• Behavior: Changing the autoAddReplicas property from disabled (false) to enabled (true) using MODIFYCOLLECTION API no longer replaces down replicas for the collection immediately. Instead, replicas are only added if a node containing them went down while autoAddReplicas was enabled. The parameters autoReplicaFailoverBadNodeExpiration and autoReplicaFailoverWorkLoopDelay are no longer used.

• Deprecations: Enabling/disabling autoAddReplicas cluster-wide with the API will be deprecated; use suspend/resume trigger APIs with name=".auto_add_replicas" instead.

  More information about the changes to this feature can be found in the section SolrCloud Automatically Adding Replicas.

Shard and cluster metric reporter configuration now require a class attribute.

• If a reporter configures the group="shard" attribute then please also configure the class="org.apache.solr.metrics.reporters.solr.SolrShardReporter" attribute.

• If a reporter configures the group="cluster" attribute then please also configure the class="org.apache.solr.metrics.reporters.solr.SolrClusterReporter" attribute.

  See the section Shard and Cluster Reporters for more information.

All Stream Evaluators in solrj.io.eval have been refactored to have a simpler and more robust structure. This simplifies and condenses the code required to implement a new Evaluator and makes it much easier for evaluators to handle differing data types (primitives, objects, arrays, lists, and so forth).

In the ReplicationHandler, the master.commitReserveDuration sub-element is deprecated. Instead please configure a direct commitReserveDuration element for use in all modes (master, slave, cloud).

The RunExecutableListener was removed for security reasons. If you want to listen to events caused by updates, commits, or optimize, write your own listener as native Java class as part of a Solr plugin.

In the XML query parser (defType=xmlparser or {!xmlparser …​ }) the resolving of external entities is now disallowed by default.