MoreLikeThis

MoreLikeThis enables queries for documents similar to a document in their result list.

It does this by using terms from the original document to find similar documents in the index.

There are several ways to use MoreLikeThis. The first, and most common, is to use it as a request handler. In this case, you would send text to the MoreLikeThis request handler as needed (as in when a user clicked on a "similar documents" link).

The second is to use it as a search component. This is less desirable since it performs the MoreLikeThis analysis on every document that matches a user query. This may slow search results.

Another approach is to use it as a request handler but with externally supplied text. This case, also referred to as the MoreLikeThisHandler, will supply information about similar documents in the index based on the text of the input document.

Finally, the MLT query parser can be used. This operates in much the same way as the request handler but since it is a query parser it can be used in filter queries, boost queries, etc., and results can be paginated or highlighted as needed.

How MoreLikeThis Works

MoreLikeThis constructs a Lucene query based on terms in a document. It does this by pulling terms from the list of fields provided with the request.

For best results, the fields should have stored term vectors (termVectors=true), which can be configured in the schema. If term vectors are not stored, MoreLikeThis can generate terms from stored fields. The field used for the uniqueKey must also be stored in order for MoreLikeThis to work properly.

Terms from the original document are filtered using thresholds defined with the MoreLikeThis parameters. Once the terms have been selected, a query is run with any other query parameters as appropriate and a new document set is returned.

MoreLikeThis Handler and Component

The MoreLikeThis request handler and search component share several parameters, but also have some key differences in response and operation, as described below.

Common Handler and Component Parameters

The list below summarizes the MoreLikeThis parameters supported by Solr. These parameters can be used with the MoreLikeThis search component or request handler.

mlt.fl
RequiredDefault: none

Specifies the fields to use for similarity. A list of fields can be provided separated by commas. If possible, the fields should have stored termVectors.

mlt.mintf
OptionalDefault: 2

Specifies the minimum frequency below which terms will be ignored in the source document.

mlt.mindf
OptionalDefault: 5

Specifies the minimum frequency below which terms will be ignored which do not occur in at least this many documents.

mlt.maxdf
OptionalDefault: none

Specifies the maximum frequency above which terms will be ignored which occur in more than this many documents.

mlt.maxdfpct
OptionalDefault: none

Specifies the maximum document frequency using a ratio relative to the number of documents in the index. The value provided must be an integer between 0 and 100. For example, mlt.maxdfpct=75 means the word will be ignored if it occurs in more than 75 percent of the documents in the index.

mlt.minwl
OptionalDefault: none

Sets the minimum word length below which words will be ignored.

mlt.maxwl
OptionalDefault: none

Sets the maximum word length above which words will be ignored.

mlt.maxqt
OptionalDefault: 25

Sets the maximum number of query terms that will be included in any generated query.

mlt.maxntp
OptionalDefault: 5000

Sets the maximum number of tokens to parse in each example document field that is not stored with TermVector support.

mlt.boost
OptionalDefault: false

Specifies if the query will be boosted by the interesting term relevance. Possible values are true or false.

mlt.qf
OptionalDefault: none

Query fields and their boosts using the same format used by the DisMax Query Parser. These fields must also be specified in mlt.fl.

mlt.interestingTerms
OptionalDefault: none

Adds a section in the response that shows the top terms (based on TF/IDF) used for the MoreLikeThis query. It supports three possible values:

  • list lists the terms.

  • none lists no terms (the default).

  • details lists the terms along with the boost value used for each term. Unless mlt.boost=true, all terms will have boost=1.0.

To use this parameter with the search component, the query cannot be distributed. In order to get interesting terms, the query must be sent to a single shard and limited to that shard only (with the shards parameter). Multi-shard support is, however, available with the MoreLikeThis request handler.

MoreLikeThis Request Handler

Request Handler Configuration

The MoreLikeThis request handler is not configured by default and needs to be set up before using it. You can do this by manually editing solrconfig.xml or with the Config API:

Manual Configuration

<requestHandler name="/mlt" class="solr.MoreLikeThisHandler">
  <str name="mlt.fl">body</str>
</requestHandler>

Config API

curl -X POST -H 'Content-type:application/json' -d {
  "add-requesthandler": {
    "name": "/mlt",
    "class": "solr.MoreLikeThisHandler",
    "defaults": {"mlt.fl": "body"}
  }
} http://localhost:8983/solr/<collection>/config

Both of the above examples set the mlt.fl parameter to "body" for the request handler. This means that all requests to the handler will use that value for the parameter unless specifically overridden in an individual request.

For more about request handler configuration in general, see the section RequestHandlers and SearchComponents in Solrconfig.

Request Handler Parameters

The MoreLikeThis request handler supports the following parameters in addition to the common parameters above. It supports faceting, paging, and filtering using common query parameters, but does not work well with alternate query parsers.

mlt.match.include
OptionalDefault: true

Specifies if the response should include the matched document. If set to false, the response will look like a normal select response.

mlt.match.offset
OptionalDefault: none

Specifies an offset into the main query search results to locate the document on which the MoreLikeThis query should operate. By default, the query operates on the first result for the q parameter.

Request Handler Query and Response

Queries to the MoreLikeThis request handler use the name defined when it was configured (/mlt in the above example).

The following example query uses a document (q=id:0553573403) found in Solr’s example document set (./example/exampledocs), and asks that the author field be used to find similar documents (mlt.fl=author).

http://localhost:8983/solr/gettingstarted/mlt?mlt.fl=author&mlt.interestingTerms=details&mlt.match.include=true&mlt.mindf=0&mlt.mintf=0&q=id%3A0553573403

This query also requests interesting terms with their boosts (mlt.interestingTerms=details) and that the original document also be returned (mlt.match.include=true). The minimum term frequency and minimum word document frequency are set to 0.

The response will include a section match, which includes the original document. The response section includes the similar documents. Finally, the interestingTerms section shows the terms from the author field that were used to find the similar documents. Because we did not also specify mlt.boost, the boost values shown for the interesting terms all display 1.0.

{
  "match":{"numFound":1,"start":0,"numFoundExact":true,
    "docs":[
      {
        "id":"0553573403",
        "cat":["book"],
        "name":["A Game of Thrones"],
        "price":[7.99],
        "inStock":[true],
        "author":["George R.R. Martin"],
        "series_t":"A Song of Ice and Fire",
        "sequence_i":1,
        "genre_s":"fantasy",
        "_version_":1693062911089442816}]
  },
  "response":{"numFound":2,"start":0,"numFoundExact":true,
    "docs":[
      {
        "id":"0553579908",
        "cat":["book"],
        "name":["A Clash of Kings"],
        "price":[7.99],
        "inStock":[true],
        "author":["George R.R. Martin"],
        "series_t":"A Song of Ice and Fire",
        "sequence_i":2,
        "genre_s":"fantasy",
        "_version_":1693062911094685696},
      {
        "id":"055357342X",
        "cat":["book"],
        "name":["A Storm of Swords"],
        "price":[7.99],
        "inStock":[true],
        "author":["George R.R. Martin"],
        "series_t":"A Song of Ice and Fire",
        "sequence_i":3,
        "genre_s":"fantasy",
        "_version_":1693062911095734272}]
  },
  "interestingTerms":[
    "author:r.r",1.0,
    "author:george",1.0,
    "author:martin",1.0]}

If we had not requested mlt.match.include=true, the response would not have included the match section.

Streaming External Content to MoreLikeThis

An external document (one not in the index) can be passed to the MoreLikeThis request handler to be used for recommended documents.

This is accomplished with the use of Content Streams. The body of a document can be passed directly to the request handler with the stream.body parameter. Alternatively, if remote streams are enabled, a URL or file could be passed.

http://localhost:8983/solr/mlt?stream.body=electronics%20memory&mlt.fl=manu,cat&mlt.interestingTerms=list&mlt.mintf=0

This query would pass the terms "electronics memory" to the request handler instead of using a document already in the index.

The response in this case would look similar to the response above that used a document already in the index.

MoreLikeThis Search Component

Using MoreLikeThis as a search component returns similar documents for each document in the response set for another query. It’s important to note this could incur a cost to search performance so should only be used when the use case warrants it.

Search Component Configuration

The MoreLikeThis search component is a default search component that works with all search handlers (see also Default Components).

Since it is configured already, it doesn’t need any additional configuration unless you’d like to set parameters for a particular collection that override the MoreLikeThis defaults. To do this, you could configure it like this:

<searchComponent name="mlt" class="solr.MoreLikeThisComponent">
    <str name="mlt">true</str>
    <str name="mlt.fl">body</str>
</searchComponent>

The above example would always enable MoreLikeThis for all queries and will always use the "body" field. This is probably not something you really want! But the example serves to show how you might define whichever parameters you would like to be default for MoreLikeThis.

If you gave the search component a name other than "mlt" as in the above example, you would need to explicitly add it to a request handler as described in the section Search Components. Because the above example uses the same name as the default, the parameters defined there override Solr’s default.

Search Component Parameters

The MoreLikeThis search component supports the following parameters in addition to the common parameters above.

mlt
OptionalDefault: none

If set to true, activates the MoreLikeThis component and enables Solr to return MoreLikeThis results.

mlt.count
OptionalDefault: 5

Specifies the number of similar documents to be returned for each result.

Search Component Query and Response

The response when using MoreLikeThis as a search component is different than when using the request handler.

In this case, we are using the /select request handler and performing a regular query (q=author:martin). We’ve asked for MoreLikeThis to be added to the response (mlt=true), but otherwise the parameters are the same as the earlier example (we’ve asked for interesting terms and set minimum term and document frequencies to 0).

http://localhost:8983/solr/gettingstarted/select?mlt.fl=name&mlt.mindf=0&mlt.mintf=0&mlt=true&q=author%3Amartin

The response includes the results of our query, in this case 3 documents which have the term "martin" in the author field. We’ve changed the field, however, to find documents that are similar to these based on values in the name field (mlt.fl=name).

In the response, a moreLikeThis section has been added. For each document in the results that match our query, a list of document IDs is returned with score values. Each of these documents are similar to the document in the result list to varying degrees.

{
  "response":{"numFound":3,"start":0,"maxScore":0.43659902,"numFoundExact":true,  "docs":[
      {
        "id":"0553573403",
        "cat":["book"],
        "name":["A Game of Thrones"],
        "price":[7.99],
        "inStock":[true],
        "author":["George R.R. Martin"],
        "series_t":"A Song of Ice and Fire",
        "sequence_i":1,
        "genre_s":"fantasy",
        "_version_":1693062911089442816},
      {
        "id":"0553579908",
        "cat":["book"],
        "name":["A Clash of Kings"],
        "price":[7.99],
        "inStock":[true],
        "author":["George R.R. Martin"],
        "series_t":"A Song of Ice and Fire",
        "sequence_i":2,
        "genre_s":"fantasy",
        "_version_":1693062911094685696},
      {
        "id":"055357342X",
        "cat":["book"],
        "name":["A Storm of Swords"],
        "price":[7.99],
        "inStock":[true],
        "author":["George R.R. Martin"],
        "series_t":"A Song of Ice and Fire",
        "sequence_i":3,
        "genre_s":"fantasy",
        "_version_":1693062911095734272}]
  },
  "moreLikeThis":[
    "0553573403",{"numFound":6,"start":0,"maxScore":1.6554483,"numFoundExact":true,
      "docs":[
        {
          "id":"055357342X",
          "score":1.6554483},
        {
          "id":"0553579908",
          "score":1.6554483},
        {
          "id":"0805080481",
          "score":1.3422124},
        {
          "id":"0812550706",
          "score":1.284826},
        {
          "id":"978-1423103349",
          "score":0.7652973}]
    },
    "0553579908",{"numFound":5,"start":0,"maxScore":1.6554483,"numFoundExact":true,
      "docs":[
        {
          "id":"055357342X",
          "score":1.6554483},
        {
          "id":"0553573403",
          "score":1.6554483},
        {
          "id":"0805080481",
          "score":1.3422124},
        {
          "id":"978-1423103349",
          "score":0.7652973},
        {
          "id":"VDBDB1A16",
          "score":0.68205893}]
    },
    "055357342X",{"numFound":5,"start":0,"maxScore":1.6554483,"numFoundExact":true,
      "docs":[
        {
          "id":"0553579908",
          "score":1.6554483},
        {
          "id":"0553573403",
          "score":1.6554483},
        {
          "id":"0805080481",
          "score":1.3422124},
        {
          "id":"978-1423103349",
          "score":0.7652973},
        {
          "id":"VDBDB1A16",
          "score":0.68205893}]
    }]}

MoreLikeThis Query Parser

The mlt query parser provides a mechanism to retrieve documents similar to a specific document, like the request handler.

It uses Lucene’s existing MoreLikeThis logic and also works in SolrCloud mode. The document identifier used here is the document’s uniqueKey value and not the Lucene internal document id. The list of returned documents excludes the queried document.

One benefit of the query parser is that it can be used in various places, not only in a standard q parameter. This allows MoreLikeThis to be added to boost queries, filter queries, function queries, etc.

Query Parser Parameters

This query parser takes the following parameters:

qf
RequiredDefault: none

Defines the fields to use as the basis for similarity analysis.

mintf
OptionalDefault: 2

Defines the minimum frequency below which terms will be ignored in the source document.

mindf
OptionalDefault: 5

Defines the minimum frequency below which terms will be ignored which do not occur in at least this many documents.

maxdf
OptionalDefault: none

Sets the maximum frequency above which terms will be ignored which occur in more than this many documents.

minwl
OptionalDefault: none

Sets the minimum word length below which words will be ignored.

maxwl
OptionalDefault: none

Sets the maximum word length above which words will be ignored.

maxqt
OptionalDefault: 25

Sets the maximum number of query terms that will be included in any generated query.

maxntp
OptionalDefault: 5000

Sets the maximum number of tokens to parse in each example document field that is not stored with TermVector support.

boost
OptionalDefault: false

Specifies if the query will be boosted by the interesting term relevance. It can be either true or false.

Query Parser Query and Response

The structure of a MoreLikeThis query parser request is like a query using local params, as in:

{!mlt qf=name}1

This would use the MoreLikeThis query parser to find documents similar to document "1", based on the "name" field.

Additional parameters would be added inside the brackets, for example if we wanted to specify limits for mintf and mindf:

{!mlt qf=name mintf=2 mindf=3}1

If given a query such as the following based on the example documents provided with Solr:

http://localhost:8983/solr/gettingstarted/select?q={!mlt qf=author mintf=1 mindf=1}0553573403

The query parser response includes only the similar documents sorted by score:

{
  "response":{"numFound":2,"start":0,"maxScore":1.309797,"numFoundExact":true,
    "docs":[
      {
        "id":"0553579908",
        "cat":["book"],
        "name":["A Clash of Kings"],
        "price":[7.99],
        "inStock":[true],
        "author":["George R.R. Martin"],
        "series_t":"A Song of Ice and Fire",
        "sequence_i":2,
        "genre_s":"fantasy",
        "_version_":1693062911094685696},
      {
        "id":"055357342X",
        "cat":["book"],
        "name":["A Storm of Swords"],
        "price":[7.99],
        "inStock":[true],
        "author":["George R.R. Martin"],
        "series_t":"A Song of Ice and Fire",
        "sequence_i":3,
        "genre_s":"fantasy",
        "_version_":1693062911095734272}]
  }}