Major Changes in Solr 8

When JSON data is sent to Solr with nested child documents split using the split parameter, the child documents will now be associated to their parents by the field/label string used in the JSON instead of anonymously.

Deleting (or updating) documents by their uniqueKey is now scoped to only consider root documents, not child/nested documents. Thus a delete-by-id won’t work on a child document (it will fail silently), and an attempt to update a child document by providing a new document with the same ID would add a new document (which will probably be erroneous).

Solr has a new field in the _default configset, called _nest_path_. This field stores the path of the document in the hierarchy for non-root documents.

If you explicitly use BM25SimilarityFactory in your schema, the absolute scoring will be lower since Lucene changed the calculation of BM25 to remove a multiplication factor (for technical details, see LUCENE-8563 or SOLR-13025). Ordering of documents will not change in the normal case. Use LegacyBM25SimilarityFactory if you need to force the old 6.x/7.x scoring.

Memory codecs have been removed from Lucene (MemoryPostings, MemoryDocValues) and are no longer available in Solr. If you used postingsFormat="Memory" or docValuesFormat="Memory" on any field or field type configuration then either remove that setting to use the default or experiment with one of the other options.

The LowerCaseTokenizer has been deprecated and is likely to be removed in Solr 9. Users are encouraged to use the LetterTokenizer and the LowerCaseFilter instead.

The _default configset now includes a ignored_* dynamic field rule.

Index-time boosts were removed from Lucene in version 7.0, and in Solr 7.x the syntax was still allowed (although it logged a warning in the logs). The syntax was similar to:

{"id":"1", "val_s":{"value":"foo", "boost":2.0}}

The date format patterns used by ParseDateFieldUpdateProcessorFactory (used by default in "schemaless mode") are now interpreted by Java 8’s java.time.DateTimeFormatter instead of Joda Time. The pattern language is very similar but not the same. Typically, simply update the pattern by changing an uppercase 'Z' to lowercase 'z' and that’s it.

The extraction module Solr Cell) no longer does any date parsing, and thus no longer supports the date.formats parameter. To ensure date strings are properly parsed, use the ParseDateFieldUpdateProcessorFactory in your update chain. This update request processor is found by default with the "parse-date" update processor when running Solr in schemaless mode".

The LanguageIdentifierUpdateProcessor base class in the langid module (found in modules/langid) changed some method signatures. If you have a custom language identifier implementation you will need to adapt your code. See the Jira issue SOLR-11774 for details of the changes.

The Unified Highlighter parameter hl.weightMatches now defaults to true. See the section Highlighting for more information about Highlighter parameters.

The eDisMax query parser will now thrown an error when the qf parameter refers to a nonexistent field.

The Function Query Parser now returns scores that are equal to zero (0) when a negative value is produced. This change is due to the fact that Lucene now requires scores to be positive.

Authentication plugins can now intercept internode requests on a per-request basis.

The Basic Authentication plugin now has an option forwardCredentials to let Basic Auth headers be forwarded on inter-node requests in case of distributed search, instead of falling back to PKI.

Metrics are now reported for authentication requests.

The Radial Graph view of a Solr cluster when running in SolrCloud mode has been removed.

The Nodes view introduced in Solr 7.5 is now the default when choosing the "Cloud" tab in the left navigation menu.

The default replica placement strategy used in Solr has been reverted to the "legacy" policy used by Solr 7.4 and previous versions. This is due to multiple bugs in the autoscaling based replica placement strategy that was made default in Solr 7.5 which causes multiple replicas of the same shard to be placed on the same node in addition to the maxShardsPerNode and createNodeSet parameters being ignored.

curl -X POST -H 'Content-type:application/json' --data-binary '
  {
    "set-obj-property": {
      "defaults" : {
        "cluster": {
          "useLegacyReplicaAssignment":false
        }
      }
    }
  }' http://$SOLR_HOST:$SOLR_PORT/api/cluster

A new command-line option is available via bin/solr autoscaling to calculate autoscaling policy suggestions and diagnostic information outside of the running Solr cluster. This option can use the existing autoscaling policy, or test the impact of a new one from a file located on the server filesystem.

All Hadoop dependencies have been upgraded to Hadoop 3.2.0 (from 2.7.2).

The Admin UI now presents a login screen for any users with authentication enabled on their cluster. Clusters with Basic Authentication will prompt users to enter a username and password. On clusters configured to use Kerberos Authentication, authentication is handled transparently by the browser as before, but if authentication fails, users will be directed to configure their browser to provide an appropriate Kerberos ticket.

The shards parameter, used to manually select the shards and replicas that receive distributed requests, now checks nodes against an allow-list of acceptable values for security reasons.

The JSON parameter to set cluster-wide default cluster properties with the CLUSTERPROP command has changed.

{
  "set-obj-property": {
    "defaults" : {
      "collection": {
        "numShards": 2,
        "nrtReplicas": 1,
        "tlogReplicas": 1,
        "pullReplicas": 1
      }
    }
  }
}

The parameter min_rf has been deprecated and no longer needs to be provided in order to see the achieved replication factor. This information will now always be returned to the client with the response.

An autoscaling policy is now used as the default strategy for selecting nodes on which new replicas or replicas of new collections are created.

The change described above has two additional impacts on the maxShardsPerNode parameter:

Lucene has introduced the ByteBuffersDirectoryFactory as a replacement for the RAMDirectoryFactory, which will be removed in Solr 9.

Since Solr 7.0, Solr’s schema field-guessing has created _str fields for all _txt fields, and returned those by default with queries. As of 7.5, _str fields will no longer be returned by default. They will still be available and can be requested with the fl parameter on queries. See also the section on field guessing for more information about how schema field guessing works.

The Standard Filter, which has been non-operational since at least Solr v4, has been removed.

When using the TieredMergePolicy, the default merge policy for Solr, optimize and expungeDeletes now respect the maxMergedSegmentMB configuration parameter, which defaults to 5000 (5GB).

The UIMA module has been removed from Solr and is no longer available.

Solr’s logging configuration file is now located in server/resources/log4j2.xml by default.

A bug for Windows users has been corrected. When using Solr’s examples (bin/solr start -e) log files will now be put in the correct location (example/ instead of server). See also Solr Examples and Solr Control Script Reference for more information.

Solr now uses Log4j v2.11. The Log4j configuration is now in log4j2.xml rather than log4j.properties files. This is a server side change only and clients using SolrJ won’t need any changes. Clients can still use any logging implementation which is compatible with SLF4J. We now let Log4j handle rotation of Solr logs at startup, and bin/solr start scripts will no longer attempt this nor move existing console or garbage collection logs into logs/archived either. See Configuring Logging for more details about Solr logging.

Configuring slowQueryThresholdMillis now logs slow requests to a separate file named solr_slow_requests.log. Previously they would get logged in the solr.log file.

In the leader-follower model of scaling Solr, a follower no longer commits an empty index when a completely new index is detected on leader during replication. To return to the previous behavior pass false to skipCommitOnLeaderVersionZero in the follower section of replication handler configuration, or pass it to the fetchindex command.

Collections created without specifying a configset name have used a copy of the _default configset since Solr 7.0. Before 7.3, the copied configset was named the same as the collection name, but from 7.3 onwards it will be named with a new ".AUTOCREATED" suffix. This is to prevent overwriting custom configset names.

The rq parameter used with Learning to Rank rerank query parsing no longer considers the defType parameter. See Running a Rerank Query for more information about this parameter.

The behaviour of the autoscaling system will now pause all triggers from execution between the start of actions and the end of a cool down period. The triggers will resume after the cool down period expires. Previously, the cool down period was a fixed period started after actions for a trigger event completed and during this time all triggers continued to run but any events were rejected and tried later.

The throttling mechanism used to limit the rate of autoscaling events processed has been removed. This deprecates the actionThrottlePeriodSeconds setting in the Autoscaling API which is now non-operational. Use the triggerCooldownPeriodSeconds parameter instead to pause event processing.

The default value of autoReplicaFailoverWaitAfterExpiration, used with the AutoAddReplicas feature, has increased to 120 seconds from the previous default of 30 seconds. This affects how soon Solr adds new replicas to replace the replicas on nodes which have either crashed or shutdown.

The default Solr log file size and number of backups have been raised to 32MB and 10 respectively. See the section Configuring Logging for more information about how to configure logging.

The old Leader-In-Recovery implementation (implemented in Solr 4.9) is now deprecated and replaced. Solr will support rolling upgrades from old 7.x versions of Solr to future 7.x releases until the last release of the 7.x major version.

Replicas which are not up-to-date are no longer allowed to become leader. Use the FORCELEADER command of the Collections API to allow these replicas become leader.

If you are using the spatial JTS library with Solr, you must upgrade to 1.15.0. This new version of JTS is now dual-licensed to include a BSD style license. See the section on Spatial Search for more information.

The top-level <highlighting> element in solrconfig.xml is now officially deprecated in favour of the equivalent <searchComponent> syntax. This element has been out of use in default Solr installations for several releases already.

Starting a query string with Local Params {!myparser …​} is used to switch from one query parser to another, and is intended for use by Solr system developers, not end users doing searches. To reduce negative side-effects of unintended hack-ability, Solr now limits the cases when local params will be parsed to only contexts in which the default parser is lucene or func.

The eDisMax parser by default no longer allows subqueries that specify a Solr parser using either local params, or the older _query_ magic field trick.

The feature to automatically add replicas if a replica goes down, previously available only when storing indexes in HDFS, has been ported to the autoscaling framework. Due to this, autoAddReplicas is now available to all users even if their indexes are on local disks.

Shard and cluster metric reporter configuration now require a class attribute.

All stream evaluators in solrj.io.eval have been refactored to have a simpler and more robust structure. This simplifies and condenses the code required to implement a new Evaluator and makes it much easier for evaluators to handle differing data types (primitives, objects, arrays, lists, and so forth).

In the ReplicationHandler, the leader.commitReserveDuration sub-element is deprecated. Instead please configure a direct commitReserveDuration element for use in all modes (leader, follower, cloud).

The RunExecutableListener was removed for security reasons. If you want to listen to events caused by updates, commits, or optimize, write your own listener as native Java class as part of a Solr plugin.

In the XML query parser (defType=xmlparser or {!xmlparser …​ }) the resolving of external entities is now disallowed by default.