Spell Checking

The IndexBasedSpellChecker uses a Solr index as the basis for a parallel index used for spell checking. It requires defining a field as the basis for the index terms; a common practice is to copy terms from some fields (such as title, body, etc.) to another field created for spell checking. Here is an example of configuring IndexBasedSpellChecker in solrconfig.xml:

The first element defines the searchComponent to use the solr.SpellCheckComponent. The classname is the specific implementation of the SpellCheckComponent, in this case solr.IndexBasedSpellChecker. Defining the classname is optional; if not defined, it will default to IndexBasedSpellChecker.

The spellcheckIndexDir defines the location of the directory that holds the spellcheck index, while the field defines the source field (defined in the Schema) for spellcheck terms. When choosing a field for the spellcheck index, it’s best to avoid a heavily processed field to get more accurate results. If the field has many word variations from processing synonyms and/or stemming, the dictionary will be created with those variations in addition to more valid spelling data.

By default, this spellchecker builds its dictionary from the Solr index. This can be changed by specifying sourceLocation - a folder with static Lucene index files to use instead of the Solr index.

The spellchecker can be assigned a descriptive label, name, - which can be helpful if the search component defines multiple spellcheckers. With that, a spellcheck query can identify a subset of spellcheckers that should be consulted (see Spell Check Parameters for more details).

The query analyzer for the field is used to tokenize the spellcheck query. If there’s a need to override that behavior, configure a fieldType and the spellchecker will use the query analyzer for that field type instead.

queryAnalyzerFieldType is a field type from Solr’s schema, and works similarly to the fieldType parameter. The key difference is that Solr uses field or fieldType when it tokenizes the spellcheck query supplied via spellcheck.q, and uses queryAnalyzerFieldType when the query is instead provided via the q parameter.

The field type specified by this parameter should do minimal transformations. It’s usually a best practice to avoid types that aggressively stem or NGram, for instance, since those types of analysis can throw off spell checking.

Common configuration parameters like distanceMeasure, comparatorClass, accuracy, and thresholdTokenFrequency provide control over the returned spellcheck suggestions.

If the distanceMeasure is not specified, Solr will use the Levenshtein metric which is the default metric for other spellchecker implementations as well (except for DirectSolrSpellChecker).

When comparatorClass is configured as "score", the suggestions with lower distance (i.e., higher similarity) scores are considered more relevant. The alternative value is "freq" - this prioritizes suggestions with higher document frequency.

The accuracy setting defines the threshold for a valid suggestion, and the thresholdTokenFrequency setting allows skipping suggestions which have low document frequency in the index.

Finally, buildOnCommit and buildOnOptimize define whether to build the spellcheck index at every commit (that is, every time new documents are added to the index) or at every optimize request. Both are optional, and can be omitted if you would rather set their values to false.

The DirectSolrSpellChecker uses terms from the Solr index without building a parallel index like the IndexBasedSpellChecker. This spellchecker has the benefit of not having to be built regularly, meaning that the terms are always up-to-date with terms in the index. Here is how this might be configured in solrconfig.xml:

When choosing a field to query for this spellchecker, you want one which has relatively little analysis performed on it (particularly analysis such as stemming). Note that you need to specify a field to use for the suggestions, so like the IndexBasedSpellChecker, you may want to copy data from fields like title, body, etc., to a field dedicated to providing spelling suggestions.

Many of the parameters relate to how this spellchecker should query the index for term suggestions. The distanceMeasure defines the metric to use during the spellcheck query - the default value for this spellchecker is "internal", which corresponds to the Damerau-Levenshtein metric.

Because this spellchecker is querying the main index, you may want to limit how often it queries the index to be sure to avoid any performance conflicts with user queries. The accuracy setting defines the threshold for a valid suggestion, while maxEdits defines the number of changes to the term to allow. Since most spelling mistakes are only 1 letter off, setting this to 1 will reduce the number of possible suggestions (the default, however, is 2); the value can only be 1 or 2. minPrefix defines the minimum number of characters the terms should share. Setting this to 1 means that the spelling suggestions will all start with the same letter, for example.

The maxInspections parameter defines the maximum number of possible matches to review before returning results; the default is 5. minQueryLength defines how many characters must be in the query before suggestions are provided; the default is 4. maxQueryLength enables the spellchecker to skip over very long query terms, which can avoid expensive operations or exceptions. There is no limit to term length by default.

At first, spellchecker analyses incoming query words by looking them up in the index. Only query words which are absent from the index, or too rare (equal to or below maxQueryFrequency) are considered as misspelled and used for finding suggestions. Words which are more frequent than maxQueryFrequency bypass spellchecker unchanged. After suggestions for every misspelled word are found they are filtered for enough frequency with thresholdTokenFrequency as boundary value. These parameters (maxQueryFrequency and thresholdTokenFrequency) can be a percentage represented as a decimal value below 1 (such as 0.01 for or 1%) or an absolute value (such as 4).

The FileBasedSpellChecker uses an external file as a spelling dictionary. This can be useful if using Solr as a spelling server, or if spelling suggestions don’t need to be based on actual terms in the index. In solrconfig.xml, you would define the searchComponent as so:

The configuration is very similar to the IndexBasedSpellChecker, and the differences here are the use of the sourceLocation to define the location of the file of terms, and the use of characterEncoding to define the encoding of the terms file.

If the fieldType parameter is specified and matches a type from the Solr schema, Solr will build the spellcheck index by first tokenizing each line from the external file using the fieldType index analyzer, and then adding each token to the index.

If not, Solr will treat each line from the external file as an individual token, and add them to the spellcheck index as is.

WordBreakSolrSpellChecker offers suggestions by combining adjacent query terms and/or breaking terms into multiple words. It is a SpellCheckComponent enhancement, leveraging Lucene’s WordBreakSpellChecker. It can detect spelling errors resulting from misplaced whitespace without the use of shingle-based dictionaries and provides collation support for word-break errors, including cases where the user has a mix of single-word spelling errors and word-break errors in the same query. It also provides shard support.

Here is how it might be configured in solrconfig.xml:

Some of the parameters should be familiar from the discussion of the other spellcheckers, such as name, classname, and field. New for this spellchecker is combineWords, which defines whether words should be combined in a dictionary search (default is true); and breakWords, which defines if words should be broken during a dictionary search (default is true).

maxChanges is an integer which defines how many times the spellchecker should check collation possibilities against the index.

maxCombinedLength allows skipping over the suggestions which are too long. Similarly, minBreakLength instructs the spellchecker to not break the word into parts that are too short.

maxEvaluations defines the maximum number of word combinations to evaluate - a higher value might improve the result quality, while a lower value might improve performance.

minSuggestionFreq sets the minimum frequency a term must have to be included as part of a suggestion.

Finally, the breakSuggestionTieBreaker setting ("max_freq" or "sum_freq") instructs Solr to sort the suggestions by the number of word breaks, and then by the maximum or by the sum of all the component term’s frequencies, respectively.

The spellchecker can be configured together with a traditional checker (i.e., DirectSolrSpellChecker). The results are combined and collations can contain a mix of corrections from both spellcheckers.