Major Changes in Solr 9
Solr 9.0 is a major new release of Solr.
This page highlights the most important changes including new features and changes in default behavior as well as previously deprecated features that have now been removed.
Solr 9 Upgrade Planning
Before starting an upgrade to Solr 9, please take the time to review all information about changes from the version you are currently on up to Solr 9.
You should also consider all changes that have been made to Solr in any version you have not upgraded to already. For example, if you are currently using Solr 8.1, you should review changes made in all subsequent 8.x releases in addition to changes for 9.0.
A thorough review of the list in Major Changes in Earlier 8.x Versions as well as the CHANGES.txt in your Solr instance will help you plan your migration to Solr 9.
Upgrade Prerequisites
Solr 9 requires Java 11 as minimum Java version and is also tested with Java 17.
If using CloudSolrClient
to connect to your SolrCloud cluster, SolrJ must be upgraded in all your client applications to version 8.10 or higher (8.x), before upgrading your SolrCloud cluster to version 9.0. Otherwise, SolrJ will not be able to connect to the cluster once it has upgraded to Solr 9.
If you have an old collection that was initially created with a Solr version prior to 5.0, Solr may keep that collection’s cluster state in the /clusterstate.json
file at the root of Zookeeper. Solr 9 no longer supports this file. You must upgrade such collections to the new per-collection state.json
format before upgrading to Solr 9. This is done by calling the Collection API MIGRATESTATEFORMAT action while still using Solr 8.x.
If you are using Solr in standalone mode with the Query Elevation Component with its elevation file in the data directory, you’ll have to move it to the configset folder instead.
If you rely on metrics, alerts, or monitors on Solr KPIs that use the "master" or "slave" terminology, please update your system for those metrics to now show up with "leader" and "follower" terminology.
Rolling Upgrades
If you are planning to upgrade your cluster using a rolling upgrade model (upgrade each node in succession, as opposed to standing up a brand new 9.x cluster), please read the following carefully.
Rolling upgrades from Solr 8 to Solr 9 require upgrading from Solr 8.7 or newer. If you run an 8.x. version prior to 8.7 we recommend that you first upgrade to 8.11.
Certain commands, like a restore of a collection, might not complete properly if performed during an upgrade. They may need to be re-attempted after.
PKI Authentication
Internode communication secured by PKI Authentication has changed formats. For detailed information, see PKI Authentication Plugin.
A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart sequence:
-
Upgrade to Solr 9 and set system properties:
solr.pki.sendVersion=v1
andsolr.pki.acceptVersions=v1,v2
. This will allow Solr 9 nodes to send messages to Solr 8 nodes while the upgrade is in progress. -
Restart with
solr.pki.sendVersion=v2
(default, can be unset) andsolr.pki.acceptVersions=v1,v2
. This will force all nodes to send the new header. -
(Optional) Restart with system property
solr.pki.acceptVersions=v2
(default, can be unset) to prevent outdated nodes from connecting to your cluster.
Reindexing After Upgrade
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the Reindexing section, which covers several strategies for how to reindex.
In Solr 8, it was possible to add docValues to a schema without re-indexing via UninvertDocValuesMergePolicy
, an advanced/expert utility.
Due to changes in Lucene 9, that isn’t possible any more.
Solr 9.8
Configuration
In solrconfig.xml, the numVersionBuckets
and versionBucketLockTimeoutMs
settings are now obsolete and ignored; a warning will be logged if specified.
<lib/>
directives have been made into an "opt-in" feature in Solr 9.8 and must be explicitly enabled if desired (using a system property or environment variable like -Dsolr.config.lib.enabled=true
)
If not explicitly enabled, Solr will log a warning but otherwise quietly ignore any <lib/>
tags in configsets.
This is a breaking change, and we apologize for the disruption, but it’s a necessary response to a number of security gaps that have come to light around this feature.
<lib/>
directives should be considered deprecated and will be removed entirely in Solr 10.0.
Current users of <lib/>
have a number of options to ensure their libraries remain on Solr’s classpath, including: SOLR_MODULES
, core "lib" directories, the solr.xml <sharedLib>
tag, Solr’s package manager, etc.
Partial Results
When query limits are in use and partial results are not desirable (i.e. reporting or quantitative usages of search)
users may pass partialResults=false
.
This feature has been improved to eliminate some wasted processing.
In the presence of this parameter, queries encountering limits will return zero results, and to make it clear when this has happened, the partialResults
response header will be set to "omitted"
Solr 9.7
SchemaVersion upgraded to 1.7
The default schemaVersion is now 1.7. With the new schema version, most fields and fieldTypes that support docValues will have docValues enabled by default. See the Enabling Doc Values Section for a complete list.
Also, fields will be unable to be uninverted by default. (uninvertible=false
)
In order to upgrade from schemaVersion 1.6 to 1.7, a re-index of all data will generally be required. This re-index cannot be in-place. Lucene does not allow adding docValues to existing fields.
However, if any of the following is true, a re-index is not required:
-
All applicable fields already have docValues enabled in the original schema
-
The schema is updated to explicitly disable docValues for all fields and fieldTypes that did not have an explicit
docValues
default provided.
The uninvertible=false
default has no impact on the index, so is unrelated to re-indexing concerns.
Solr 9.6
Dependency upgrades
Lucene is upgraded to 9.10.0 and a variety of library dependencies have been upgraded. See https://solr.apache.org/docs/9_6_0/changes/Changes.html for specific upgraded libraries.
SolrJ
-
The
Http2SolrClient
classes powered by the Jetty Http library are no longer marked as experimental. -
Solrj now offers a lightweight client based on java.net.http.HttpClient:
HttpJdkSolrClient
-
SolrJ now offers an improved async request API
Solr 9.5
Dependency Upgrades
Solr 8.2 recommended using Zookeeper 3.5.5 and now with Curator 5.5.0 requires Zookeeper 3.5.x or higher. This primarily affects users of hadoop-auth
, but usage of Curator could affect other parts of Solr.
Global Circuit Breakers
-
Circuit breakers can now be configured globally, not only per collection. See Configuring Circuit Breakers for more information.
Configuration and Ease-of-Use
-
Solr will now automatically resolve all environment variables with
SOLR_
prefix, and set the corresponding system property. This is useful for configuring more aspects of Solr through environment variables, such as for containers. Underscores are replaced with dots and strings are lowercased. For example, while you earlier had to set the system property-Dsolr.clustering.enabled=true
to enable clustering, you can now set the equivalent environment variableSOLR_CLUSTERING_ENABLED=true
instead. -
Collection and Replica Properties may now be used as property substitution variables in configuration files (e.g. solrconfig.xml).
-
Solr now auto-reloads updated keystore and truststore files when TLS is enabled.
-
Tracing support has received a number of quality-of-life improvements, including improved tracking of distributed collection commands and increased coverage for internal requests made with the Apache and Jetty HTTP clients.
Solr 9.4
The Built-In Config Sets
-
The build in ConfigSets (
_default
andsample_techproducts_configs
), now use a defaultautoSoftCommit
time of 3 seconds, meaning that documents will be searchable within 3 seconds of uploading them. Please refer to the Soft Commit documentation for more information on how this change will effect you. Upgrading existing clouds and use-cases that have custom configSets will not be affected by this change.
Replica Placements
-
The default
minimalFreeDiskGB
value has been lowered from20GB
to5GB
when using the AffinityPlacementPlugin. Therefore, when using the default settings, nodes that were previously excluded from Replica placements due to low available disk space may be selected after upgrading.
Embedded Zookeeper
-
The Embedded Zookeeper can now be configured to listen to (or bind to) more hosts than just
localhost
, see the Network Configuration documentation for more information.
Circuit Breaker
-
The Circuit Breakers are now pluggable, and you can define multiple circuit breakers including custom ones. The existing
CircuitBreakerManager
is deprecated, and users are encouraged to switch to the new plugins. While the oldCircuitBreakerManager
returned HTTP 503 when a circuit breaker was tripped, the new plugins return HTTP 429.
Security
-
Since Solr 8.4.1/8.5.0, the
solr.jetty.ssl.verifyClientHostName
sysProp andSOLR_SSL_CLIENT_HOSTNAME_VERIFICATION
envVar have been used incorrectly. It has instead been used to override thesolr.ssl.checkPeerName
sysProp in theHTTP2SolrClient
. This has been fixed, and the setting once again tells the server to check the originating client hostname against the client certificate when doing mTLS. This option is still enabled by default. -
The
solr.jetty.ssl.sniHostCheck
option now defaults to the value ofSOLR_SSL_CHECK_PEER_NAME
, if it is provided. This will enable client and server hostName check settings to be governed by the same environment variable. If users want separate client/server settings, they can manually override thesolr.jetty.ssl.sniHostCheck
option inSOLR_OPTS
.
Deprecations
-
The
rid
request id query parameter has been deprecated in favor of the always-on trace generation. To disable therid
generation set the system propertysolr.disableRequestId
totrue
. To disable the always-on trace generation set the system propertysolr.alwaysOnTraceId
tofalse
.
Solr 9.3
Binary Releases
-
Solr now comes with two binary releases and docker images, full and slim. The Installing Solr page provides information on what is included in each. Please refer to the Solr Downloads site for information on how to download these offerings.
Shard Management
-
Solr now provides an "Install Shard" API to allow users who have built (per-shard) indices offline to import them into SolrCloud shards.
Solr CLI
-
The
bin/solr -i
andbin/solr -info
options were removed in favour of thebin/solr status
command.
Security
-
Use of
stream.file
,stream.url
andstream.body
params are no longer enabled via configuration in solrconfig.xml, nor dynamic equivalents with the config API. Older configuration now does nothing. Instead, set an env var: SOLR_ENABLE_REMOTE_STREAMING or SOLR_ENABLE_STREAM_BODY or system property equivalents. -
The method for specifying sysProps that contain sensitive information has been streamlined. Now the sysProp
-Dsolr.hiddenSysProps
or the envVarSOLR_HIDDEN_SYS_PROPS
are available to provide a comma-separated list of patterns to match sysProps that should be hidden or redacted. Please see the hiddenSysProps section for more information.The sysProp
-Dsolr.redaction.system.pattern
has been deprecated, use the above options instead.The
<hiddenSysProps>
solr.xml element under<metrics>
has been deprecated. Instead, use the <hiddenSysProps> tag under<solr>
, which accepts a comma-separated string. -
The Schema Designer now utilizes the same trust model as the ConfigSet Upload API.
Official Docker Image
-
The customization of the Official Solr Dockerfile has been changed. The customization options
SOLR_DOWNLOAD_URL
,SOLR_CLOSER_URL
,SOLR_DIST_URL
andSOLR_ARCHIVE_URL
, have been removed. The only way to specify a custom location for the Solr binaries is throughSOLR_DOWNLOAD_SERVER
. If the server URL containsapache.org
, then the Dockerfile will check gpg signature files. If the server URL does not containapache.org
, then the gpg signature checking will be skipped.It is still strongly recommended to use the Dockerfile included in the Solr binary TGZs if you want to build Solr images with custom versions of Solr. The custom version of Solr will include this Dockerfile when it is built.
Deletion of unknown cores is now disabled by default
-
When Solr loads a core from a filesystem it will check for corresponding cluster state in ZooKeeper. Prior to Solr 9.3, if no corresponding entry existed the core was deleted automatically to remove the orphaned files. As of Solr 9.3 that behaviour is no longer enabled by default. See Unknown core deletion.
use of timeAllowed
-
Query timeouts with
timeAllowed
are implemented differently. It should be faster albeit have less fidelity – will not timeout a query outside of core query processing (e.g. won’t cancel spellcheck or faceting). Usesolr.useExitableDirectoryReader
to use the previous behavior. Please share your experience with Solr developers! The previous behavior should not be enabled if timeAllowed isn’t used because unfortunately its performance tax is now imposed on all queries, even those without timeAllowed.
v2 API
-
Solr’s experimental "v2" API has seen a number of improvements in the 9.3 release.
It is now approaching parity with the functionality offered by Solr’s v1 API. In particular 9.3 adds v2 "CRUD" endpoints for interacting with alias properties and collection snapshots. Several lower-level "replication" APIs now also offer v2 equivalents.
Additionally, the v2 API as a whole is being redesigned to be more REST-ful and intuitive. To this end, 9.3 introduces backwards-incompatible changes to the v2 shard, replica, and backup creation and deletion APIs. A number of lower-level "command" APIs have also been tweaked, including: collection reloading, collection renaming, "force leader", "balance shard unique", and all of Solr’s "log level" and "log watcher" APIs. Clients using Solr’s v2 endpoints should check their usage against the full list of modified APIs in CHANGES.txt prior to upgrading. Details on each are available in the 9.3 reference guide.
Solr 9.2
Upgrade to Jetty 10.x
-
Solr upgraded to Jetty 10.x from 9.x due to Jetty 9.x is now end of life. Jetty 10.x has a Java 11 minimum and matches Solr 9 minimum Java version. Jetty logging has been replaced with slf4j again matching Solr. See https://webtide.com/jetty-10-and-11-have-arrived/ for additional Jetty 10.x highlights.
Jetty Configuration
-
Solr no longer duplicates certain Jetty "server" library dependencies between
server/lib
andWEB-INF/lib
(jetty-util, jetty-io, etc.). This is an improvement to the binary release artifact, but Jetty does not allow web-apps (Solr) to share these libraries by default. Theserver/contexts/solr-jetty-context.xml
now explicitly removes these restrictions, allowing Solr to share these "server" jars which now live inserver/lib/ext
. -
The "Transient Cores" feature is now deprecated.
SSL Configuration
-
When using Solr (or SolrJ) with an SSL-enabled Solr cluster using HTTP2, the default
-Dsolr.ssl.checkPeerName
value is now true. This is what has been documented in Enabling SSL, and matches the functionality of the originalHttpSolrClient
.
Tracing
-
A new
opentelemetry
module is added, with support for OTEL tracing inOTLP
format using gRPC.
Docker
-
The OS version of the official Docker image and provided Dockerfile has been upgraded to Ubuntu 22 (jammy) from Ubuntu 20 (focal). Solr’s Docker image now requires Docker engine version 20.10.10 or newer.
Users who cannot upgrade their Docker engine will need to specify the docker commandline option --security-opt seccomp=unconfined when starting the container.
|
Streaming Expressions
-
Streaming Expressions have been moved out of Solrj core into its own module called
solrj-streaming
. This change will only affect users that have used streaming expression classes in clients outside of Solr. Streaming expressions sent to the /stream handler will still operate exactly as before. External clients using streaming expression classes will need to update their depenencies to reference thesolrj-streaming
artifacts under theorg.apache.solr
groupId.
Deprecations
-
Loading solr.xml from Zookeeper is deprecated. See Configuring solr.xml.
-
The Analytics Component has been deprecated. Consider using JSON Facet API as a substitute. Please notify the project if there’s functionality you need that isn’t currently covered by JSON facets.
-
The
jaegertracer-configurator
module is deprecated for removal in Solr 10. Users should start migrating to the newopentelemetry
module.
Solr 9.1.1
-
Solr no longer accepts all file types for configSets. Please see ConfigSet Forbidden File Types for more information.
Solr 9.1
Querying and Indexing
-
Added Lucene91HnswVectorsFormat codec for DenseVectorField. In order to use the new codec, reindex is necessary.
SolrJ
SolrJ is beginning to be split up.
If you use ZooKeeper coordinates to create a CloudSolrClient
, you will need to add a dependency on solrj-zookeeper
.
If you use SolrJ’s Maven POM to depend on SolrJ, then this should happen automatically through transitive resolution.
Instead of depending on ZooKeeper, consider migrating to use of specifying a list of Solr URLs in the client’s builder.
Not only does this reduce dependencies, but it improves security by being able to limit ZooKeeper access.
Zookeeper
Zookeeper Credentials support now follows a new paradigm.
Old classes, such as VMParamsAllAndReadonlyDigestZkACLProvider
and VMParamsSingleSetCredentialsDigestZkCredentialsProvider
, are deprecated but still supported until at least 10.0
.
Users are encouraged to upgrade to the non-deprecated classes before the next major version release.
Please refer to deployment-guide:zookeeper-access-control.adoc#solr-to-zookeeper-acls-workflow for more information.
Solr 9.0
Querying and Indexing
-
Dense Vector "Neural" Search through
DenseVectorField
fieldType and K-Nearest-Neighbor (KNN) Query Parser. -
Admin UI support for SQL Querying.
-
New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
-
New NorwegianNormalizationFilter
-
Implicit
/terms
handler now returns terms across all shards in SolrCloud instead of only the local core. Users/apps may be assuming the old behavior. A request can be modified via the standarddistrib=false
param to only use the local core receiving the request. -
SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section SQL Query Language.
-
JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
Facet count in Json Facet module always returns a
long
value, irrespective of number of shards. -
MacroExpander
will no longer will expand URL parameters inside of theexpr
parameter (used by streaming expressions). Additionally, users are advised to use theInjectionDefense
class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding theexpr
parameter can be reinstated with-DStreamingExpressionMacros=true
passed to the JVM at startup -
The response format for field values serialized as raw XML (via the
[xml]
raw value DocTransformer andwt=xml
) has changed. Previously, values were dropped in directly as top-level child elements of each<doc>
, obscuring associated field names and yielding inconsistent<doc>
structure. As of version 9.0, raw values are wrapped in a<raw name="field_name">[…]</raw>
element at the top level of each<doc>
(or within an enclosing<arr name="field_name"><raw>[…]</raw></arr>
element for multi-valued fields). Existing clients that parse field values serialized in this way will need to be updated accordingly. -
Highlighting:
hl.method=unified
is the new default. Usehl.method=original
to switch back if needed. -
solr.xml
maxBooleanClauses
is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings usingedismax
with manyqf
andpf
fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value ofmaxBooleanClauses
if warranted. -
Atomic/partial updates to nested documents now require the
_root_
field to clearly show the document isn’t a root document. Solr 8 would fallback on the_route_
param but no longer.
Security
-
New Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
-
Improved security when using PKI Authentication plugin.
-
Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
-
All request handlers support security permissions. Users may have to adapt their
security.json
. -
Ability to disable admin UI through a system property.
-
The property
blockUnknown
in theBasicAuthPlugin
and theJWTAuthPlugin
now defaults totrue
instead offalse
. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly setblockUnknown:false
insecurity.json
. -
Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
-
Solr now binds to localhost network interface by default for better out of the box security. Administrators that need Solr exposed more broadly can change the
SOLR_JETTY_HOST
property in their Solr include (solr.in.sh
/solr.in.cmd
) file. -
Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production. If you rely upon the previous behavior, then you can change the
clientPortAddress
insolr/server/solr/zoo.cfg
-
Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs. This may require some more disk space for logs than was the case in version 8.x. See Configuring Logging for how to change this.
-
Hadoop authentication support has been moved to the new
hadoop-auth
module. Users need to add the modulehadoop-auth
to classpath. The plugins has also changed package name toorg.apache.solr.security.hadoop
, but can still be loaded as shortformclass="solr.HadoopAuthPlugin"
,class="solr.ConfigurableInternodeAuthHadoopPlugin"
orclass="solr.KerberosPlugin"
- see the section Hadoop Authentication Plugin. -
JWTAuthPlugin has been moved to a module. Users need to add the module
jwt-auth
to classpath. The plugin has also changed package name toorg.apache.solr.security.jwt
, but can still be loaded as shortformclass="solr.JWTAuthPlugin"
. -
Dependency updates - A lot of dependency updates removes several security issues from dependencies, and thus make Solr more secure.
-
The allow-list defining allowed URLs for the
shards
parameter is not in theshardHandler
configuration anymore. It is defined by theallowUrls
top-level property of thesolr.xml
file. For more information, see Format of solr.allowUrls documentation. -
To improve security,
StatelessScriptUpdateProcessorFactory
has been renamed asScriptUpdateProcessorFactory
and moved to thescripting
Module instead of shipping as part of Solr core. This module needs to be enabled explicitly. -
To improve security,
XSLTResponseWriter
has been moved to thescripting
Module instead of shipping as part of Solr core. This module needs to be enabled explicitly.
Stability and Scalability
-
Rate limiting provides a way to throttle update and search requests based on usage metrics.
-
A new Task management interface allows declaring tasks as cancellable and trackable.
-
Ability to specify node roles in Solr. This release supports
overseer
anddata
roles out of the box. -
New API for pluggable Replica Placement Plugins that replaces the auto-scaling framework.
-
Support for distributed processing of cluster state updates and collection API calls, without relying on the Overseer.
Build and Docker
-
Solr is now built and released independently of Lucene (separate Apache projects).
-
Build system switched to Gradle, no longer uses Ant + Ivy.
-
Docker image creation is now a part of the Apache Solr GitHub repo.
-
Docker image documentation is now a part of the reference guide.
-
Official Docker image upgraded to use JDK17 (by Eclipse Temurin) and ability to create functionally identical local image.
Logging and Metrics
-
Metrics handler only depends on SolrJ instead of core and has its own
log4j2.xml
and no longer shares Solr’s logging config. -
Only
SearchHandler
and subclasses have "local" metrics now. It’s now tracked as if it’s another handler with a "[shard]" suffix, e.g. "/select[shard]". There are no longer ".distrib." named metrics; all metrics are assumed to be such except "[shard]". The default Prometheus exporter config splits that component to a new label named "internal". The sample Grafana dashboard now filters to include or exclude this. -
The default port of "Prometheus exporter" has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
Logging is now asynchronous by default. There’s a small window where log messages may be lost in the event of some hard crash. Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
Log4J configuration & Solr MDC values - MDC values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc…) have been modified to now be "bare" values, without the special single character prefixes that were included in past version. The default
log4j2.xml
configuration file for Solr has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the<PatternLayout/>
. Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See SOLR-15630 for more details. -
Jetty Request log is now enabled by default, i.e. logging every request.
-
The prometheus-exporter is no longer packaged as a Solr Module. It can be found under
solr/prometheus-exporter/
. -
Solr modules (formerly known as contribs) can now easily be enabled by an environment variable (e.g. in
solr.in.sh
orsolr.in.cmd
) or as a system property (e.g. inSOLR_OPTS
). Example:SOLR_MODULES=extraction,ltr
.
Deprecations and Removals
-
The Data Import Handler (DIH) is an independent project now; it is no longer a part of Solr.
-
No more support for
clusterstate.json
andMIGRATESTATE
API has been removed. If your collections useclusterstate.json
you will need to take some steps, described elsewhere in this document. -
Auto-scaling framework has been removed. Please refer to Replica Placement Plugins for alternate options.
-
LegacyBM25SimilarityFactory
has been removed. -
Legacy SolrCache implementations (LRUCache, LFUCache, FastLRUCache) have been removed. Users have to modify their existing configurations to use CaffeineCache instead.
-
VelocityResponseWriter
is an independent project now; it is no longer a part of Solr. This encompasses all previously included/browse
andwt=velocity
examples. -
Cross Data Center Replication has been removed.
-
SolrJ clients like
HttpSolrClient
andLBHttpSolrClient
that lacked HTTP2 support have been deprecated. The old CloudSolrClient has been renamed as CloudLegacySolrClient and deprecated. -
SimpleFSDirectoryFactory is removed in favor of NIOFSDirectoryFactory
-
Removed the deprecated
HttpSolrClient.RemoteSolrException
andHttpSolrClient.RemoteExecutionException
. All the usages are replaced byBaseHttpSolrClient.RemoteSolrException
andBaseHttpSolrClient.RemoteExecutionException
. -
maxShardsPerNode
parameter has been removed because it was broken and inconsistent with other replica placement strategies. Other relevant placement strategies should be used instead, such as autoscaling policy or rules-based placement. -
The binary distribution no longer contains test-framework jars.
-
Deprecated BlockJoinFacetComponent and BlockJoinDocSetFacetComponent are removed. Users are encouraged to migrate to uniqueBlock() in JSON Facet API.
-
Core level admin API endpoints
/admin/threads
,/admin/properties
,/admin/logging
are now only available at the node level.
Other
-
Contrib modules are now just "modules". You can easily enable module(s) through environment variable
SOLR_MODULES
. -
Features lifted out as separate modules are: HDFS, Hadoop-Auth, SQL, Scripting, and JWT-Auth.
-
The "dist" folder in the release has been removed. Please update your
<lib>
entries in yoursolrconfig.xml
to use the new location.-
The
solr-core
andsolr-solrj
jars can be found underserver/solr-webapp/webapp/WEB-INF/lib/
. -
The Solr Module jars and their dependencies can be found in
modules/<module-name>/lib
, packaged individually for each module. -
The
solrj-deps
(SolrJ Dependencies) are no longer separated out from the other Server jars. -
Please refer to the SolrJ Maven artifact to see the exact dependencies you need to include from
server/solr-webapp/webapp/WEB-INF/lib/
andserver/lib/ext/
if you are loading in SolrJ manually. If you plan on using SolrJ as a JDBC driver, please refer to the JDBC documentation -
More information can be found in the Libs documentation.
-
-
SolrJ class
CloudSolrClient
now supports HTTP2. It has a new Builder. SeeCloudLegacySolrClient
for the 8.x version of this class. -
In Backup request responses, the
response
key now uses a map to return information instead of a list. This is only applicable for users returning information in JSON format, which is the default behavior. -
SolrMetricProducer
/SolrInfoBean
APIs have changed and third-party components that implement these APIs need to be updated. -
Use of blacklist/whitelist terminology has been completely removed. JWTAuthPlugin parameter
algWhitelist
is nowalgAllowlist
. The old parameter will still work in 9.x. Environment variablesSOLR_IP_WHITELIST
andSOLR_IP_BLACKLIST
are no longer supported, but replaced withSOLR_IP_ALLOWLIST
andSOLR_IP_DENYLIST
. -
Solr Backups - Async responses for backups now correctly aggregate and return information. For collection’s snapshot backup request responses additional fields
indexVersion
,indexFileCount
, etc. were added similar to incremental backup request responses. -
If you are using the HDFS backup repository, you need to change the repository class to
org.apache.solr.hdfs.backup.repository.HdfsBackupRepository
- see the HDFS Backup Repository section. -
HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related changes, however the module needs to be installed - see the section Solr on HDFS.
-
The folder
$SOLR_HOME/userfiles
, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder. -
Solr no longer requires a
solr.xml
in$SOLR_HOME
. If one is not found, Solr will instead use the default one from$SOLR_TIP/server/solr/solr.xml
. You can revert to the pre-9.0 behaviour by setting environment variableSOLR_SOLRXML_REQUIRED=true
or system property-Dsolr.solrxml.required=true
. Solr also does not require azoo.cfg
in$SOLR_HOME
if started with embedded zookeeper. -
base_url
has been removed from stored cluster state. If you’re able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set-Dsolr.storeBaseUrl=false
(introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, thebase_url
will no longer be persisted in stored state. However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set-Dsolr.storeBaseUrl=true
so that Solr will continue to store thebase_url
in Zookeeper. For background, see: SOLR-12182 and SOLR-15145. Support for thesolr.storeBaseUrl
system property will be removed in Solr 10.x andbase_url
will no longer be stored. -
Analyzer components can now be looked up by their SPI names based on the field type configuration.
-
The
solr-extraction
module has been cleaned up to producesolr-extraction-
jar instead ofsolr-cell-
jars. -
Extra lucene libraries used in modules are no longer packaged in
lucene-libs/
under module directories in the binary release. Instead, these libraries will be included with all other module dependencies inlib/
.
Major Changes in Earlier 8.x Versions
The following is a list of major changes released between Solr 8.1 and 8.11.
Please be sure to review this list so you understand what may have changed between the version of Solr you are currently running and Solr 9.0.
Solr 8.11
See the 8.11 Release Notes for an overview of the main new features of Solr 8.11.
When upgrading to 8.11.x users should be aware of the following major changes from 8.10.
Support for Multiple Authentication Schemes
Two new authentication and authorization plugins provide support for configuring multiple authentication schemes.
The MultiAuthPlugin
allows combining two or more authentication approaches, such as JWT and Basic authentication.
The MultiAuthRuleBasedAuthorizationPlugin
is used when the MultiAuthPlugin
is also in use, and combines the various roles defined for all plugins to determine the proper role assignment for the user account.
For information on configuring these plugins, see the following sections:
ZooKeeper chroot
It’s now possible to create the ZooKeeper chroot at startup if it does not already exist. See the section Using the -z Parameter with bin/solr for an example.
Other Changes
A few other minor changes are worth noting:
-
The
config-read
pre-defined permission now correctly governs access for various configuration-related APIs. See also Predefined Permissions. -
The S3BackupRepository supports configuring the AWS Profile, if necessary. See also S3BackupRepository.
-
Additionally, backups will now properly succeed after SPLITSHARD operations, and will correctly handle incremental backup purges.
-
SolrJ now supports uploading configsets.
Solr 8.10
See the 8.10 Release Notes for an overview of the main new features of Solr 8.10.
When upgrading to 8.10.x users should be aware of the following major changes from 8.9.
Schema Designer UI
A new screen has been added to the Admin UI that allows you to interactively design a Solr schema using your documents.
The designer screen provides a safe environment for you to:
-
Upload or paste sample documents to identify fields.
-
Get a "first" guess at what Solr thinks the field types in the fields should be.
-
Edit fields, field types, dynamic fields, and supporting files.
-
See how a field’s analysis will impact your text.
-
Test how schema changes will impact query-time behavior.
-
Save your changes to a configset to use with a new collection.
See the section Schema Designer for full details.
Backups in S3
Following the redesign of backups in Solr 8.8 that allowed storage of incremental backups in Google Cloud environments, Solr 8.10 provides support for storing backups in Amazon S3 buckets.
See the section S3BackupRepository for how to configure.
Security Admin UI
Solr’s Admin UI also got a new screen to support management of users, roles, and permissions.
The new UI works when authentication and/or authorization has been enabled with bin/solr auth
or by manually installing a security.json
file.
Before this, it provides a warning that your Solr instance is unsecured.
See the section Security UI for details.
Solr SQL Improvements
A number of improvements have been made in Solr’s SQL functionality:
-
Support added for
LIKE
,IS NOT NULL
,IS NULL
, and wildcards (for simplisticLIKE
functionality). -
Two new aggregation functions,
COUNT(DISTINCT field)
andAPPROX_COUNT_DISTINCT(field)
, have been added. -
Queries using an
ORDER BY
clause can supportOFFSET
andFETCH
operations. -
Multi-valued fields can now be returned.
-
User permissions have been simplified so access to query endpoints
/sql
,/select
, and/export
is sufficient for full access for all SQL queries.
shards.preference
A new option for the shards.preference
parameter allows selection of nodes based on whether or not the replica is a leader.
Now adding shards.preference=replica.leader:false
will limit queries only to replicas which are not currently their shard’s leader.
See the section shards.preference Parameter for details and examples.
Metrics & Prometheus Exporter
A new expr
option in the Metrics API allows for more advanced filtering of metrics based on regular expressions.
See the section Metrics API for examples.
The Prometheus Exporter’s default solr-exporter.config
has been improved to use the new expr
option in the Metrics API to get a smaller set of metrics.
The default metrics exported still include most metrics, but the configuration will be easier to trim as needed.
This should help provide performance improvements in busy clusters being monitored by Prometheus.
ZooKeeper Credentials
ZooKeeper credentials can now be stored in a file whose location is defined with a system property instead of being passed in plain-text. See Out of the Box Credential Implementations for how to set this up.
Solr 8.9
See the 8.9 Release Notes for an overview of the main new features of Solr 8.9.
When upgrading to 8.9.x users should be aware of the following major changes from 8.8.
Backup and Restore
Solr 8.9 introduces extensive changes to Solr’s backup and restore support.
A new backup format has been introduced in Solr 8.9 which replaces the previous snapshot-based backup. This new format enables ‘incremental’ backups. Repeated backups to a given location will take advantage of the data stored by their predecessors and will only operate on files that have changed since the previous backup. This is supported by default, simply by storing each backup file in the same location.
The old and new formats are not compatible, although backups in the old format, a full snapshot of all files, can still be used to restore to Solr for the time-being. The old format is officially deprecated, and support for it is likely to be removed in Solr 9.0.
For the time-being the old format can be created by defining a parameter incremental=false
.
Again, though, this support is likely to be removed in Solr 9.0.
More documentation on backups is available at Backup and Restore.
New Collections API commands for backups:
-
LISTBACKUP: Lists information about each backup stored at the specified repository location. See List Backups for more details.
-
DELETEBACKUP: Deletes specified backups from the repository. See Delete Backups for more details.
A new option for backup repository is also available in 8.9, which is to use Google Cloud Storage (GCS).
This is a module (located in modules/gcs-repository
).
See GCSBackupRepository for configuration details.
The Solr community is working to add support for S3 buckets in the near future.
Nested Docs
Child Doc Transformer’s childFilter
parameter no longer applies query syntax
escaping because it’s inconsistent with the rest of Solr and was limiting.
This refers to [child childFilter='field:value']
.
There was no escaping here prior to 8.0 either.
Collapse and Expand
-
BlockCollapse: If documents have been (or could be) indexed in a way where documents with the same collapse key have been indexed contiguously in the index, a new "block collapse" provides a significant speed improvement over traditional collapse.
See Block Collapsing for details.
-
Expand Null Groups: A new parameter
expand.nullGroup
allows an expanded group to be returned containing document with no value in the expanded field. See Expand Component for details.
In-Place Updates
A new request parameter update.partial.requireInPlace=true
allows telling Solr to "fail fast" if all of the necessary conditions are not satisfied to allow an in-place update to succeed.
See also In-Place Updates.
Metrics History
The Metrics History feature, which allowed long-term storage and aggregation of Solr’s metrics, has been deprecated and will be removed in 9.0.
Embedded Solr Server
When using EmbeddedSolrServer, it will no longer close CoreContainer instances that were passed to it.
Solr 8.8
When upgrading to 8.8.x users should be aware of the following major changes from 8.7.
Nested Documents
-
When doing atomic/partial updates to a child document:
-
Supply the
_root_
field (the ID of the root document) so that Solr understands you are manipulating a child document and not a root document. In its absence, Solr looks at the_route_
parameter but this may change in the future because it’s not an ideal substitute. If neither are present, Solr assumes you are updating a root document. If this assumption is false, Solr will do a cheap check that usually detects the problem and will throw an exception to alert you of the need to specify the Root ID. This backwards incompatible change was done to increase performance and robustness. -
This feature no longer requires
stored=true
ordocValues=true
on the_root_
field. You might have it for other purposes though (e.g., foruniqueBlock(…)
). -
This feature no longer requires the
_nest_path_
field, although you probably ought to continue to define it as it’s useful for other things.
-
Removed Modules
-
The search results clustering module (Carrot2) has been removed from 8.x Solr due to lack of Java 1.8 compatibility in the dependency that provides online clustering of search results. The module will be re-introduced in Solr 9.0.
Learning to Rank
-
Interleaving support has been added to Learning to Rank (LTR). Currently only the Team Draft Interleaving algorithm is supported. For examples using this feature, see the section Running a Rerank Query Interleaving Two Models.
Metrics
-
Two metrics have been added for SolrCloud’s Overseer:
-
solr_metrics_overseer_stateUpdateQueueSize
-
solr_metrics_overseer_collectionWorkQueueSize
-
Prometheus Exporter
-
The
./bin
scripts included with the Prometheus Exporter now allow use of custom java options with environment variables. See the section Environment Variable Options for more details. -
The default Grafana dashboards now include panels for query performance monitoring. The default Prometheus Exporter configuration includes metrics like queries-per-second (QPS) and 95th percentiles (P95) to populate the new panels.
-
The default Prometheus Exporter configuration also includes the two new metrics mentioned in the Metrics above.
Solr Home
-
The internal logic for identifying 'Solr Home' (
solr.solr.home
) has been refactored to make testing less error-prone. Plugin developers usingSolrPaths.locateSolrHome()
ornew SolrResourceLoader
should check deprecation warnings as existing some existing functionality will be removed in 9.0. SOLR-14934 has more technical details about this change for those concerned.
base_url removed from stored state
As of Solr 8.8.0, the base_url
property was removed from the stored state for replicas (SOLR-12182).
If you’re able to upgrade SolrJ to 8.8.x
for all of your client applications, then you can set -Dsolr.storeBaseUrl=false
(introduced in Solr 8.8.1) to better align the stored state
in ZooKeeper with future versions of Solr.
However, if you are not able to upgrade SolrJ to 8.8.x for all client applications,
then leave the default -Dsolr.storeBaseUrl=true
so that Solr will continue to store the base_url
in ZooKeeper.
You may also see some NPE in collection state updates during a rolling upgrade to 8.8.0 from a previous version of Solr. After upgrading all nodes in your cluster to 8.8.0, collections should fully recover. Trigger another rolling restart if there are any replicas that do not recover after the upgrade to re-elect leaders.
Solr 8.7
See the 8.7 Release Notes for an overview of the main new features of Solr 8.7.
When upgrading to 8.7.x users should be aware of the following major changes from 8.6.
Autoscaling
-
If upgrading from 8.6.0, please see the 8.6.1 Upgrade notes below for information on performance degradations introduced in 8.6.0 that require some intervention to resolve. If you are already on 8.6.1 or higher, you can ignore these instructions.
Deprecations
-
The autoscaling framework is now formally deprecated and will be removed in Solr 9.0. The Solr community is working on pluggable API to replace this functionality, with the goal for it to be ready by the time 9.0 is released. Deprecations include: autoscaling policy, triggers,
withCollection
support, simulation framework, autoscaling suggestions tab in the UI,autoAddReplicas
andUTILIZENODE
command. -
Similarly, rule-based replica placement strategy has been deprecated and will be replaced in Solr 9.0 by APIs for replica placement and cluster events, with plugin-based implementations.
-
Support for detecting spinning disks has been removed in LUCENE-9576. Corresponding
spins
metrics in Solr still exist but now they always returnfalse
and will be removed in Solr 9.0.
User-Managed Cluster Terminology Updated
-
Solr has replaced the terms "master" and "slave" in the codebase and all documentation with "leader" and "follower".
This functionality has only changed in terms of parameter names changed, and we do not expect any back-compatibility issues on upgrade to 8.7 or even 9.0 later.
However, users should update their
solrconfig.xml
files after completing the upgrade on all nodes of a cluster. Comparing your configuration to the updated configuration examples in User-Managed Index Replication Index Replication will show examples of what needs to change, but here are the main changes:-
On the replication leader, in the definition of the
/replication
request handler:-
Replace "master" with "leader".
-
Replace "slave" with "follower" if the former term is used in the name of any follower
solrconfig.xml
file definitions. This file can be named anything, so you can change it to whatever you’d like to call it if you’d like. -
Replace "slave" with "follower" if the former term is used in a replication repeater configuration.
-
-
On the replication follower, in the definition of the
/replication
request handler:-
Replace "masterUrl" with "leaderUrl".
-
Replace "slave" with "follower" if the former term is used in a repeater configuration.
-
-
JSON Facets
-
Performance enhancements for the
relatedness()
statistics function are included with 8.7. These yield the highest benefits with high-cardinality fields and can be disabled if working with lower cardinality fields with a newsweep_collection
parameter. See the section relatedness() Options for details.
solr.in.sh / solr.in.cmd
-
Solr has relied on the
SOLR_STOP_WAIT
parameter defined insolr.in.sh
orsolr.in.cmd
to determine how long to wait on startup. A new parameterSOLR_START_WAIT
allows defining how long Solr should wait for start up to complete.If the time set by this parameter is exceeded, Solr will exit the startup process and return the last few lines of the
solr.log
file to the terminal.By default, this parameter is set to the same value as
SOLR_STOP_WAIT
. -
The default ZooKeeper client timeout (
ZK_CLIENT_TIMEOUT
) is now 30 seconds (30000
milliseconds) instead of 15.
Configsets
-
It’s now possible to overwrite an existing configset when uploading changes by supplying the
overwrite=true
parameter to the Configset API.A related parameter is
cleanup=true
, which allows deleting any files from the old configset that are left behind after the overwrite.The default for both of these parameters is
false
. -
When deleting a collection that has an automatically created configset (i.e., the configset was copied from the
_default
collection when the collection was created), the configset will also be deleted if it is not in use by any other collection.
Logging
-
A request ID (
rid
) is now logged for all distributed search requests (in SolrCloud) which can be used to correlate query events across the system. A parameterdisableRequestId=true
can be added to disable this if desired.
Solr 8.6.1
See the 8.6.1 Release Notes for an overview of the fixes included in Solr 8.6.1.
When upgrading to 8.6.1 users should be aware of the following major changes from 8.6.0.
Autoscaling
-
As mentioned in the 8.6 upgrade notes, a default autoscaling policy was provided starting in 8.6.0. This default autoscaling policy resulted in increasingly slow collection creation calls in large clusters (50+ collections).
In 8.6.1 the default autoscaling policy has been removed, and clusters will not use autoscaling unless a policy has explicitly been created. If your cluster is running 8.6.0 and not using an explicit autoscaling policy, upgrade to 8.6.1 and remove the default cluster policy and preferences via the following command.
Replace
localhost:8983
with your Solr endpoint.curl -X POST -H 'Content-type:application/json' -d '{set-cluster-policy : [], set-cluster-preferences : []}' http://localhost:8983/api/cluster/autoscaling
This information is only relevant for users upgrading from 8.6.0. If upgrading from an earlier version to 8.6.1+, this warning can be ignored.
Solr 8.6
See the 8.6 Release Notes for an overview of the main new features of Solr 8.6.
When upgrading to 8.6.x users should be aware of the following major changes from 8.5.
Support for Block-Max WAND
Lucene added support for Block-Max WAND in 8.0, and 8.6 makes this available for Solr also.
This can provide significant performance enhancements by not calculating the score for results which are not likely to appear in the top set of results.
It is enabled when using a new query parameter minExactCount
.
This parameter tells Solr to accurately count the number of hits accurately until at least this value.
Once this value is reached, Solr can skip over documents that don’t have a score high enough to be in the top set of documents, which has the potential for greatly speeding up searches.
It’s important to note that when using this parameter, the hit count of searches may not be accurate.
It is guaranteed that the hit count is accurate up to the value of minExactCount
, but any returned hit count higher than that may be an approximation.
A new boolean attribute numFoundExact
is included in all responses to indicate if the hit count in the response is expected to be exact or not.
More information about this new feature is available in the section minExactCount Parameter.
Autoscaling
-
NOTE: The default autoscaling policy has been removed as of 8.6.1
Solr now includes a default autoscaling policy. This policy can be overridden with your custom rules or by specifying an empty policy to replace the default.
-
The ComputePlan action now supports a collection selector to identify collections based on collection properties to determine which collections should be operated on.
Security
-
Prior to Solr 8.6 Solr APIs which take a file system location, such as core creation, backup, restore, and others, did not validate the path and Solr would allow any absolute or relative path. Starting in 8.6 only paths that are relative to
SOLR_HOME
,SOLR_DATA_HOME
andcoreRootDir
are allowed by default.If you need to create a core or store a backup outside the default paths, you will need to tell Solr which paths to allow. A new element in
solr.xml
calledallowPaths
takes a comma-separated list of allowed paths.When using the
solr.xml
file that ships with 8.6, you can configure the list of paths to allow through the system propertysolr.allowPaths
. Please seebin/solr.in.sh
orbin\solr.in.cmd
for example usage. Using the value*
will allow any path as in earlier versions.For more on this, see the section Solr.xml Parameters.
Windows SMB shares on the UNC format, such as
\\myhost\myshare\mypath
are now always disallowed. Please use drive letter mounts instead, i.e.,S:\mypath
. -
A new authorization plugin
ExternalRoleRuleBasedAuthorizationPlugin
is now available. This plugin allows an authentication plugin (such as JWT) to supply a user’s roles instead of maintaining a user-to-role mapping inside Solr. -
When authentication is enabled, the Admin UI Dashboard (main screen) now includes a panel that shows the authentication and authorization plugins in use, the logged in username, and the roles assigned to this user. A new link will also appear in the left-hand navigation to allow a user to log out.
Streaming Expressions
-
The
/export
handler now supports streaming expressions to allow limiting the output of the export to only matching documents.For more information about how to use this, see the section Specifying the Local Streaming Expression.
-
The
stats
,facet
, andtimeseries
expressions now support percentiles and standard deviation aggregations.
Highlighting
For the Unified Highlighter: The setting hl.fragsizeIsMinimum
now defaults to false
because true
was found to be a significant performance regression when highlighting lots of text.
This will yield longer highlights on average compared to Solr 8.5 but relatively unchanged compared to previous releases.
Furthermore, if your application highlights lots of text, you may want to experiment with lowering hl.fragAlignRatio
to trade ideal fragment alignment for better performance.
Deprecations
A primary focus of the community is improving Solr’s stability and supportability. With the addition of the package manager system in 8.4, we now have the ability to move some features into plugins maintained by third-parties with the expertise to properly develop and support them. Our goal is to make running Solr easier and less prone to outages and other headaches. In this spirit, the following features have been deprecated and are slated to be removed in Solr 9.0.
-
Cross Data Center Replication (CDCR), in its current form, is deprecated and is scheduled to be removed in 9.0. This feature is unlikely to be replaced by an identical plugin. However, the community is working on figuring out a replacement feature for disaster recovery and failover.
-
The Data Import Handler (DIH) is deprecated and is scheduled to be removed in 9.0. Work to replace DIH with a community-supported plugin is underway and may be available soon.
-
Support to store indexes and backups in HDFS is deprecated and is scheduled to be removed in 9.0. A community-supported version of this may be available as a plugin in the future. For more details, please see SOLR-14021.
Users interested in maintaining a feature as a plugin are encouraged to join the developer mailing list to find out more about how to help.
Solr 8.5
See the 8.5 Release Notes for an overview of the main new features of Solr 8.5.
When upgrading to 8.5.x users should be aware of the following major changes from 8.4.
Note: an index incompatibility warning was retroactively added below to 8.4 for users choosing a non-default postings format (e.g., "FST50").
Considerations for a SolrCloud Upgrade
Solr 8.5 introduces a change in the format used for the elements in the Overseer queues and maps (see SOLR-14095 for technical discussion of the change). This queue is used internally by the Overseer to reliably handle operations, to communicate operation results between the Overseer and the coordinator node, and by the REQUESTSTATUS API for displaying information about async Collection operations.
This change won’t require you to change any client-side code you should see no differences on the client side. However, it does require some care when upgrading an existing SolrCloud cluster depending on your upgrade strategy.
If you are upgrading Solr with an atomic restart strategy:
-
If you don’t use async or REQUESTSTATUS operations, you should be able to restart and not see any issues.
-
If you do use Collection API operations:
-
Pause Collection API operations.
-
Cleanup queues (See the section DELETESTATUS for examples) if you use async operations.
-
Upgrade and restart the nodes.
-
Resume all normal operations.
-
If you are upgrading Solr with a rolling restart strategy:
-
If you don’t use Collection API operations, you should be able to do a rolling restart and not see any issues.
-
If you do use Collection API operations, but you can pause their use during the restart the easiest way is to:
-
Pause Collection API operations.
-
Upgrade and restart all nodes.
-
Cleanup queues (See the section DELETESTATUS for examples) if you use async operations.
-
Resume all normal operations.
-
If you use Collection API operations and can’t pause them during the upgrade:
-
Start 8.5 nodes with the system property:
-Dsolr.useUnsafeOverseerResponse=deserialization
. Ensure the Overseer node is upgraded last. -
Once all nodes are in 8.5 and once you don’t need to read old status anymore, restart again removing the system property.
If you prefer to keep the old (but insecure) serialization strategy, you can start your nodes using the system
property: -Dsolr.useUnsafeOverseerResponse=true
.
Keep in mind that this will be removed in future version of Solr.
Security Manager
Solr now has the ability to run with a Java security manager enabled.
To enable this, set the property SOLR_SECURITY_MANAGER_ENABLED=true
in solr.in.sh
or solr.in.cmd
.
Note that if you are using HDFS to store indexes, you cannot enable the security manager.
In Solr 9.0, this will be the default.
Block/Allow Specific IPs
Solr has two new parameters to allow you to restrict access to Solr using IP addresses.
Use SOLR_IP_WHITELIST
to configure a whitelist, and SOLR_IP_BLACKLIST
to configure a blacklist.
These properties are defined in solr.in.sh
or solr.in.cmd
.
See also the section Enable IP Access Control.
BlockJoin Facet Deprecation
The BlockJoinFacetComponent is marked for deprecation and will be removed in 9.0.
Users are encouraged to migrate to uniqueBlock()
in JSON Facet API.
More information about this is available in the section Block Join Domain Changes.
Caching with the Boolean Query Parser
By default, the Boolean Query Parser caches queries in Solr’s filterCache.
It’s now possible to disable this with the local param cache=false
.
Indexing Log Files
Solr now includes a command line tool, bin/postlogs
which will index Solr’s log files into a collection.
This provides an easy way to use Solr or visualization tools (such as Zeppelin) to troubleshoot problems with the system.
See the documentation for more details at Log Analytics.
Highlighting
Solr’s Unified Highlighter now has two parameters to help control passage sizing, hl.fragAlignRatio
and hl.fragsizeIsMinimum
.
See the section Unified Highlighter for details about these new parameters.
Regardless of the settings, the passages may be sized differently than before.
Warning: These default settings were found to be a significant performance regression for apps that highlight lots of text with the default sentence break iterator.
See the 8.6 upgrade notes for advise you can apply in 8.5.
Shared Library System Parameter
Solr’s solr.xml
file has long had support for a sharedLib
parameter, which allows you to define a common location for .jar files that may need to be in the path for all cores.
This property can now be defined in solr.in.sh
or solr.in.cmd
as a system property (-Dsolr.sharedLib=/path/to/lib
) added to SOLR_OPTS
(see solr.in.sh
or solr.in.cmd
for details).
Solr 8.4
See the 8.4 Release Notes for an overview of the main new features of Solr 8.4.
When upgrading to 8.4.x users should be aware of the following major changes from 8.3.
Reminder: If you set the postingsFormat
or docValuesFormat
in the schema in order to use a non-default option, you risk preventing yourself from upgrading your Lucene/Solr software at future versions.
Multiple non-default postings formats changed in 8.4, thus rendering the index data from a previous index.
This includes "FST50" which was recommended by the Solr TaggerHandler for performance reasons.
There is now improved documentation to navigate this trade-off choice.
Package Management System
Version 8.4 introduces a package management system to Solr. The goals of the system are to allow hot (live) deployment of plugins, provide packaging guidelines for plugins, and standardize Solr’s approach by following familiar concepts used in other package management systems.
The system is designed to eventually replace use of the <lib ../>
directive,
the Blob Store, and other methods of deploying plugins and custom components
to Solr.
The system is currently considered experimental, so use with caution. It must be enabled with a system parameter passed at start up before it can be used. For details, please see the section Package Management.
With this feature Solr’s Blob Store functionality is now deprecated and will likely be removed in 9.0.
Security
The follow mix of changes were all made with the intention of making Solr more secure out of the box.
-
The
solrconfig.xml
file in Solr’s_default
configset has been trimmed of the following previously pre-configured items:-
All
<lib …/>
directives. This means that Solr Cell (aka Tika), Learning to Rank, Clustering (with Carrot2), language identification, and Velocity (for the/browse
sample search interface) are no longer enabled out of the box. -
The
/browse
,/tvrh
, and/update/extract
request handlers. -
The Term Vector Component.
-
The XSLT and Velocity response writers.
All of these items can be added to your Solr implementation by manually editing
solrconfig.xml
to add them back in, or use the Config API.The
sample_techproducts_configs
and the examples found in./example
are unchanged.
-
-
Configsets that have been uploaded with an unsecured Configset API (i.e., when authentication is not enabled) are considered "Untrusted Configsets".
In order to bolster Solr’s out-of-the-box security, these untrusted configsets are no longer allowed to use the
<lib …/>
directive to implement modules or custom Jars.When upgrading to 8.4, if you are using untrusted configsets that contain
<lib ../>
directives, their corresponding collections will not load (they will cease to work). You have a few options in this case:-
You can secure your Solr instance with authentication and re-upload the configset (using the
bin/solr zk upconfig …
Solr CLI command); -
You can put your custom Jars in Solr’s classpath instead of
lib
directories; -
You can try the new package management system to manage your custom Jars.
See the section Upload a Configset for more details about trusted vs. untrusted configsets.
-
-
Our default Jetty configuration has been updated to now set a Content-Security-Policy (CSP) by default. See
./server/etc/jetty.xml
for details about how it is configured.As a result of this change, any custom HTML served by Solr’s HTTP server that contains inline Javascript will no longer execute in modern browsers. The options for you are:
-
Change your JavaScript code to not run inline any longer;
-
Edit
jetty.xml
to remove CSP (creating weaker security protection); -
Remove/alter the headers with a reverse proxy.
-
-
Solr’s Blob Store and runtime libs functionality are now deprecated and are planned to be removed from Solr in version 9.0. It has been replaced with the new package management system.
-
The Velocity response writer is also now deprecated and is planned to be removed from Solr in version 9.0.
Using Collapse with Group Disallowed
Using the CollapsingQueryParser with Result Grouping has never been supported as it causes inconsistent behavior and NullPointerException errors. We have now explicitly disallowed this combination to prevent these errors. If you are using these together, you will need to modify your queries.
SolrJ
-
SolrJ now supports the
shards.preference
parameter for single-shard scenarios to ensure multi-shard and single-shard request routing works in the same way.See Cloud Request Routing and shards.preference Parameter for details.
-
QueryResponse.getExplainMap()
type has changed fromMap<String, String>
toMap<String, Object>
in order to support structured explanations.This change is expected to be mostly back-compatible. Compiled third-party components will work the same due to type erasure, but source code changes may be required.
-
Replica routing code has been moved to SolrJ, making those classes available to clients if necessary.
Streaming Expressions
-
A new DBSCAN clustering streaming evaluator has been added.
-
The
precision
stream evaluator can now operate on matrices. -
The
random
streaming expression can now create the x-axis.
JSON Facets
-
Two new aggregations have been added:
missing
andcountvals
. -
Several aggregations now support multi-valued fields:
min
,max
,avg
,sum
,sumsq
,stddev
,variance
, andpercentile
.
Caches
-
After the addition of
CaffeineCache
in 8.3, legacy SolrCache implementations are deprecated and likely to be removed in 9.0.Users are encouraged to transition their cache configurations to use
org.apache.solr.search.CaffeineCache
as soon as feasible.
Solr 8.3
See the 8.3 Release Notes for an overview of the main new features of Solr 8.3.
When upgrading to 8.3.x users should be aware of the following major changes from 8.2.
JWT Authentication
JWT Authentication now supports multiple identity providers.
To allow this, the parameter jwkUrl
has been deprecated and replaced with jwksUrl
.
Implementations using jwkUrl
will continue to work as normal, but users
should plan to transition their configurations to use jwksUrl
instead as
soon as feasible.
Caches
-
Solr has a new cache implementation,
CaffeineCache
, which is now recommended over other caches. This cache is expected to generally provide most users lower memory footprint, higher hit ratio, and better multi-threaded performance.Since caching has a direct impact on the performance of your Solr implementation, before switching to any new cache implementation in production, take care to test for your environment and traffic patterns so you fully understand the ramifications of the change.
-
A new parameter,
maxIdleTime
, allows automatic eviction of cache items that have not been used in the defined amount of time. This allows the cache to release some memory and should aid those who want or need to fine-tune their caches.
See the section Caches and Query Warming for more details about these and other cache options and parameters.
Solr 8.2
See the 8.2 Release Notes for an overview of the main new features of Solr 8.2.
When upgrading to 8.2.x, users should be aware of the following major changes from v8.1.
ZooKeeper 3.5.5
Solr 8.2 updates the version of ZooKeeper included with Solr to v3.5.5.
It is recommended that external ensembles set up to work with Solr also be updated to ZooKeeper 3.5.5.
This ZooKeeper release includes many new security features.
In order for Solr’s Admin UI to work with 3.5.5, the zoo.cfg
file must allow access to ZooKeeper’s "four-letter commands".
At a minimum, ruok
, conf
, and mntr
must be enabled, but other commands can optionally be enabled if you choose.
See the section Configuration for a ZooKeeper Ensemble for details.
Until 8.3, SOLR-13672 causes the ZK Status screen in the Admin UI to not be able to report status. This only impacts the UI, ZooKeeper still operates correctly. |
Routed Aliases
-
Routed aliases now use collection properties to identify collections that belong to the alias; prior to 8.2, these aliases used core properties.
This is backward-compatible and aliases created with prior versions will continue to work. However, new collections will no longer add the
routedAliasName
property to thecore.properties
file so any external code depending on this location will need to be updated.
-
Time-routed aliases now include a
TRA
infix in the collection name, in the pattern<alias>_TRA_<timestamp>
.
Collections created with older versions will continue to work.
Distributed Tracing Support
This release adds support for tracing requests in Solr. Please review the section Distributed Tracing for details on how to configure this feature.
Solr 8.1
See the 8.1 Release Notes for an overview of the main new features of Solr 8.1.
When upgrading to 8.1.x, users should be aware of the following major changes from v8.0.
Global maxBooleanClauses Parameter
-
The behavior of the
maxBooleanClauses
parameter has changed to reduce the risk of exponential query expansion when dealing with pathological query strings.A default upper limit of 1024 clauses is now enforced at the node level. This was the default prior to 7.0, and it can be overridden with a new global parameter in
solr.xml
. This limit will be enforced for all queries whether explicitly defined by the user (or client), or created by Solr and Lucene internals.An identical parameter is available in
solrconfig.xml
for limiting the size of queries explicitly defined by the user (or client), but this per-collection limit will still be restricted by the global limit set insolr.xml
.If your use case demands that you a lot of OR or AND clauses in your queries, upon upgrade to 8.1 you may need to adjust the global
maxBooleanClauses
parameter since between 7.0 and 8.1 the limit was effectively unbounded.For more information about the new parameter, see the section maxBooleanClauses.
Security
-
JSON Web Tokens (JWT) are now supported for authentication. These allow Solr to assert a user is already authenticated via an external identity provider, such as an OpenID Connect-enabled IdP. For more information, see the section JWT Authentication Plugin.
-
A new security plugin for audit logging has been added. A default class
SolrLogAuditLoggerPlugin
is available and configurable insecurity.json
. The base class is also extendable for adding custom audit plugins if needed. See the section Audit Logging for more information.
Collections API
-
The output of the REQUESTSTATUS command in the Collections API will now include internal asynchronous requests (if any) in the "success" or "failed" keys.
-
The CREATE command will now return the appropriate status code (4xx, 5xx, etc.) when the command has failed. Previously, it always returned
0
, even in failure. -
The MODIFYCOLLECTION command now accepts an attribute to set a collection as read-only. This can be used to block a collection from receiving any updates while still allowing queries to be served. See the section MODIFYCOLLECTION for details on how to use it.
-
A new command RENAME allows renaming a collection by setting up a one-to-one alias using the new name. For more information, see the section RENAME.
-
A new command REINDEXCOLLECTION allows indexing existing stored fields from a source collection into a new collection. For more information, please see the section REINDEXCOLLECTION.
Logging
-
The default Log4j2 logging mode has been changed from synchronous to asynchronous. This will improve logging throughput and reduce system contention at the cost of a slight chance that some logging messages may be missed in the event of abnormal Solr termination.
If even this slight risk is unacceptable, the Log4j configuration file found in
server/resources/log4j2.xml
has the synchronous logging configuration in a commented section and can be edited to re-enable synchronous logging.
Metrics
-
The SolrGangliaReporter has been removed from Solr. The metrics library used by Solr, Dropwizard Metrics, was updated to version 4, and Ganglia support was removed from it due to a dependency on the LGPL license.
Browse UI (Velocity)
-
Velocity and Velocity Tools were both upgraded as part of this release. Velocity upgraded from 1.7 to 2.0. Please see https://velocity.apache.org/engine/2.0/upgrading.html about upgrading. Velocity Tools upgraded from 2.0 to 3.0. For more details, please see https://velocity.apache.org/tools/3.0/upgrading.html for details about the upgrade.
Default Garbage Collector (GC)
-
Solr’s default GC has been changed from CMS to G1. If you prefer to use CMS or any other GC method, you can modify the
GC_TUNE
section ofsolr.in.sh
(*nix) orsolr.in.cmd
(Windows).