Other Parsers

Block Join Children Query Parser

This parser wraps a query that matches some parent documents and returns the children of those documents.

The syntax for this parser is: q={!child of=<blockMask>}<someParents>.

The resulting query will match all documents which do not match the <blockMask> query and are children (or descendents) of the documents matched by <someParents>.

Using the example documents above, we can construct a query such as q={!child of="content_type:parent"}title:lucene. We only get one document in response:

<result name="response" numFound="1" start="0">
  <doc>
    <str name="id">4</str>
    <arr name="content_type"><str>child</str></arr>
    <str name="comments">Lots of new features</str>
  </doc>
</result>

Parent query must not match any docs besides parent filter. Combine them as must (+) and must-not (-) clauses to find a problem doc.

?q={!child of=<blockMask> filters=$parentfq excludeTags=certain}<someParents>
&parentfq=BRAND:Foo
&parentfq=NAME:Bar
&parentfq={!tag=certain}CATEGORY:Baz

q={!child of=<blockMask>}+<someParents> +BRAND:Foo +NAME:Bar

Block Join Parent Query Parser

This parser takes a query that matches child documents and returns their parents.

The syntax for this parser is similar to the child parser: q={!parent which=<blockMask>}<someChildren>.

The resulting query will match all documents which do match the <blockMask> query and are parents (or ancestors) of the documents matched by <someChildren>.

Again using the example documents above, we can construct a query such as q={!parent which="content_type:parent"}comments:SolrCloud. We get this document in response:

<result name="response" numFound="1" start="0">
  <doc>
    <str name="id">1</str>
    <arr name="content_type"><str>parent</str></arr>
    <arr name="title"><str>Solr has block join support</str></arr>
  </doc>
</result>

Child query must not match same docs with parent filter. Combine them as must clauses (+) to find a problem doc.

?q={!parent which=<blockMask> filters=$childfq excludeTags=certain}<someChildren>
&childfq=COLOR:Red
&childfq=SIZE:XL
&childfq={!tag=certain}PRINT:Hatched

q={!parent which=<blockMask>}+<someChildren> +COLOR:Red +SIZE:XL

Block Masks: The of and which local params

The purpose of the "Block Mask" query specified as either an of or which parameter (depending on the parser used) is to identy the set of all documents in the index which should be treated as "parents" (or their ancestors) and which documents should be treated as "children". This is important because in the "on disk" index, the relationships are flattened into "blocks" of documents, so the of / which params are needed to serve as a "mask" against the flat document blocks to identify the boundaries of every hierarchical relationship.

In the example queries above, we were able to use a very simple Block Mask of doc_type:parent because our data is very simple: every document is either a parent or a child So this query string easily distinguishes all of our documents.

A common mistake is to try and use a which parameter that is more restrictive then the set of all parent documents, in order to filter the parents that are matched, as in this bad example:

// BAD! DO NOT USE!
q={!parent which="title:join"}comments:support

This type of query will frequently not work the way you might expect. Since the which parameter only identifies some of the "parent" documents, the resulting query can match "parent" documents it should not, because it will mistakenly identify all documents which do not match the which="title:join" Block Mask as children of the next "parent" document in the index (that does match this Mask).

A similar problematic situation can arise when mixing parent/child documents with "simple" documents that have no children and do not match the query used to identify 'parent' documents. For example, if we add the following document to our existing parent/child example documents:

<add>
  <doc>
    <field name="id">0</field>
    <field name="content_type">plain</field>
    <field name="title">Lucene and Solr are cool</field>
  </doc>
</add>

…​then our simple doc_type:parent Block Mask would no longer be adequate. We would instead need to use *:* -doc_type:child or doc_type:(simple parent) to prevent our "simple" document from mistakenly being treated as a "child" of an adjacent "parent" document.

The Searching Nested Documents section contains more detailed examples of specifing Block Mask queries with non trivial hierarchicies of documents.

Boost Query Parser Examples

Creates a query name:foo which is boosted (scores are multiplied) by the function query log(popularity):

q={!boost b=log(popularity)}name:foo

Creates a query name:foo which has it’s scores multiplied by the inverse of the numeric price field — effectively "demoting" documents which have a high price by lowering their final score:

// NOTE: we "add 1" to the denominator to prevent divide by zero
q={!boost b=div(1,add(1,price))}name:foo

The query(…​) function is particularly useful for situations where you want to multiply (or divide) the score for each document matching your main query by the score that document would have from another query.

This example uses local parameter variables to create a query for name:foo which is boosted by the scores from the independently specified query category:electronics:

q={!boost b=query($my_boost)}name:foo
my_boost=category:electronics

Complex Phrase Parser Limitations

Performance is sensitive to the number of unique terms that are associated with a pattern. For instance, searching for "a*" will form a large OR clause (technically a SpanOr with many terms) for all of the terms in your index for the indicated field that start with the single letter 'a'. It may be prudent to restrict wildcards to at least two or preferably three letters as a prefix. Allowing very short prefixes may result in to many low-quality documents being returned.

Notice that it also supports leading wildcards "*a" as well with consequent performance implications. Applying ReversedWildcardFilterFactory in index-time analysis is usually a good idea.

<maxBooleanClauses>4096</maxBooleanClauses>

 q=features:"Stores up to 15,000"

 q=features:"sto* up to 15*"&defType=complexphrase

Graph Query Parameters

Graph Query Limitations

The graph parser only works in single node Solr installations, or with SolrCloud collections that use exactly 1 shard.

Graph Query Examples

To understand how the graph parser works, consider the following Directed Cyclic Graph, containing 8 nodes (A to H) and 9 edges (1 to 9):

One way to model this graph as Solr documents, would be to create one document per node, with mutivalued fields identifying the incoming and outgoing edges for each node:

curl -H 'Content-Type: application/json' 'http://localhost:8983/solr/my_graph/update?commit=true' --data-binary '[
  {"id":"A","foo":  7, "out_edge":["1","9"],  "in_edge":["4","2"]  },
  {"id":"B","foo": 12, "out_edge":["3","6"],  "in_edge":["1"]      },
  {"id":"C","foo": 10, "out_edge":["5","2"],  "in_edge":["9"]      },
  {"id":"D","foo": 20, "out_edge":["4","7"],  "in_edge":["3","5"]  },
  {"id":"E","foo": 17, "out_edge":[],         "in_edge":["6"]      },
  {"id":"F","foo": 11, "out_edge":[],         "in_edge":["7"]      },
  {"id":"G","foo":  7, "out_edge":["8"],      "in_edge":[]         },
  {"id":"H","foo": 10, "out_edge":[],         "in_edge":["8"]      }
]'

With the model shown above, the following query demonstrates a simple traversal of all nodes reachable from node A:

http://localhost:8983/solr/my_graph/query?fl=id&q={!graph+from=in_edge+to=out_edge}id:A

"response":{"numFound":6,"start":0,"docs":[
   { "id":"A" },
   { "id":"B" },
   { "id":"C" },
   { "id":"D" },
   { "id":"E" },
   { "id":"F" } ]
}

We can also use the traversalFilter to limit the graph traversal to only nodes with maximum value of 15 in the foo field. In this case that means D, E, and F are excluded – F has a value of foo=11, but it is unreachable because the traversal skipped D:

http://localhost:8983/solr/my_graph/query?fl=id&q={!graph+from=in_edge+to=out_edge+traversalFilter='foo:[*+TO+15]'}id:A

...
"response":{"numFound":3,"start":0,"docs":[
   { "id":"A" },
   { "id":"B" },
   { "id":"C" } ]
}

The examples shown so far have all used a query for a single document ("id:A") as the root node for the graph traversal, but any query can be used to identify multiple documents to use as root nodes. The next example demonstrates using the maxDepth parameter to find all nodes that are at most one edge away from an root node with a value in the foo field less then or equal to 10:

http://localhost:8983/solr/my_graph/query?fl=id&q={!graph+from=in_edge+to=out_edge+maxDepth=1}foo:[*+TO+10]

...
"response":{"numFound":6,"start":0,"docs":[
   { "id":"A" },
   { "id":"B" },
   { "id":"C" },
   { "id":"D" },
   { "id":"G" },
   { "id":"H" } ]
}

Simplified Models

The Document & Field modeling used in the above examples enumerated all of the outgoing and income edges for each node explicitly, to help demonstrate exactly how the "from" and "to" parameters work, and to give you an idea of what is possible. With multiple sets of fields like these for identifying incoming and outgoing edges, it’s possible to model many independent Directed Graphs that contain some or all of the documents in your collection.

But in many cases it can also be possible to drastically simplify the model used.

For example, the same graph shown in the diagram above can be modeled by Solr Documents that represent each node and know only the ids of the nodes they link to, without knowing anything about the incoming links:

curl -H 'Content-Type: application/json' 'http://localhost:8983/solr/alt_graph/update?commit=true' --data-binary '[
  {"id":"A","foo":  7, "out_edge":["B","C"] },
  {"id":"B","foo": 12, "out_edge":["E","D"] },
  {"id":"C","foo": 10, "out_edge":["A","D"] },
  {"id":"D","foo": 20, "out_edge":["A","F"] },
  {"id":"E","foo": 17, "out_edge":[]        },
  {"id":"F","foo": 11, "out_edge":[]        },
  {"id":"G","foo":  7, "out_edge":["H"]     },
  {"id":"H","foo": 10, "out_edge":[]        }
  ]'

With this alternative document model, all of the same queries demonstrated above can still be executed, simply by changing the “from” parameter to replace the “in_edge” field with the “id” field:

http://localhost:8983/solr/alt_graph/query?fl=id&q={!graph+from=id+to=out_edge+maxDepth=1}foo:[*+TO+10]

...
"response":{"numFound":6,"start":0,"docs":[
   { "id":"A" },
   { "id":"B" },
   { "id":"C" },
   { "id":"D" },
   { "id":"G" },
   { "id":"H" } ]
}

Hash Range Parameters

Hash Range Example

{!hash_range f="field_name" l="0" u="12345"}

Hash Range Cache Config

The hash range query parser uses a special cache to improve the speedup of the queries. The following should be added to the solrconfig.xml for the various fields that you want to perform the hash range query on. Note the name of the cache should be the field name prefixed by "hash_".

<cache name="hash_field_name"
       class="solr.LRUCache"
       size="128"
       initialSize="0"
       regenerator="solr.NoOpRegenerator"/>

Parameters

This query parser takes the following parameters:

Joining Across Single Shard Collections

You can also specify a fromIndex parameter to join with a field from another core or a single shard collection. If running in SolrCloud mode, then the collection specified in the fromIndex parameter must have a single shard and a replica on all Solr nodes where the collection you’re joining to has a replica.

Let’s consider an example where you want to use a Solr join query to filter movies by directors that have won an Oscar. Specifically, imagine we have two collections with the following fields:

movies: id, title, director_id, …​

movie_directors: id, name, has_oscar, …​

To filter movies by directors that have won an Oscar using a Solr join on the movie_directors collection, you can send the following filter query to the movies collection:

fq={!join from=id fromIndex=movie_directors to=director_id}has_oscar:true

Notice that the query criteria of the filter (has_oscar:true) is based on a field in the collection specified using fromIndex. Keep in mind that you cannot return fields from the fromIndex collection using join queries, you can only use the fields for filtering results in the "to" collection (movies).

Next, let’s understand how these collections need to be deployed in your cluster. Imagine the movies collection is deployed to a four node SolrCloud cluster and has two shards with a replication factor of two. Specifically, the movies collection has replicas on the following four nodes:

node 1: movies_shard1_replica1

node 2: movies_shard1_replica2

node 3: movies_shard2_replica1

node 4: movies_shard2_replica2

To use the movie_directors collection in Solr join queries with the movies collection, it needs to have a replica on each of the four nodes. In other words, movie_directors must have one shard and replication factor of four:

node 1: movie_directors_shard1_replica1

node 2: movie_directors_shard1_replica2

node 3: movie_directors_shard1_replica3

node 4: movie_directors_shard1_replica4

At query time, the JoinQParser will access the local replica of the movie_directors collection to perform the join. If a local replica is not available or active, then the query will fail. At this point, it should be clear that since you’re limited to a single shard and the data must be replicated across all nodes where it is needed, this approach works better with smaller data sets where there is a one-to-many relationship between the from collection and the to collection. Moreover, if you add a replica to the to collection, then you also need to add a replica for the from collection.

For more information, Erick Erickson has written a blog post about join performance titled Solr and Joins.

Cross Collection Join

The Cross Collection Join Filter is a method for the join parser that will execute a query against a remote Solr collection to get back a set of join keys that will be used to as a filter query against the local Solr collection.

The crossCollection join query will create an CrossCollectionQuery object. The CrossCollectionQuery will first query a remote Solr collection and get back a streaming expression result of the join keys. As the join keys are streamed to the node, a bitset of the matching documents in the local index is built up. This avoids keeping the full set of join keys in memory at any given time. This bitset is then inserted into the filter cache upon successful execution as with the normal behavior of the Solr filter cache.

If the local index is sharded according to the join key field, the cross collection join can leverage a secondary query parser called the "hash_range" query parser. The hash_range query parser is responsible for returning only the documents that hash to a given range of values. This allows the CrossCollectionQuery to query the remote Solr collection and return only the join keys that would match a specific shard in the local Solr collection. This has the benefit of making sure that network traffic doesn’t increase as the number of shards increases and allows for much greater scalability.

The CrossCollection join query works with both String and Point types of fields. The fields that are being used for the join key must be single-valued and have docValues enabled.

It’s advised to shard the local collection by the join key as this allows for the optimization mentioned above to be utilized.

The cross collection join queries should not be generally used as part of the q parameter, but rather it is designed to be used as a filter query (fq parameter) to ensure proper caching.

The remote Solr collection that is being queried should have a single-valued field for the join key with docValues enabled.

The remote Solr collection does not have any specific sharding requirements.

  <queryParser name="join" class="org.apache.solr.search.JoinQParserPlugin">
    <str name="routerField">product_id_s</str>
    <arr name="allowSolrUrls">
      <str>http://othersolr.example.com:8983/solr</str>
    </arr>
  </queryParser>

http://localhost:8983/solr/localCollection/query?fl=id&q={!join method="crossCollection" fromIndex="otherCollection" from="fromField" to="toField" v="*:*"}

Example with Analysed Fields

Typical analysis:

 <fieldType name="text_min_hash" class="solr.TextField" positionIncrementGap="100">
    <analyzer>
      <tokenizer class="solr.ICUTokenizerFactory"/>
      <filter class="solr.ICUFoldingFilterFactory"/>
      <filter class="solr.ShingleFilterFactory" minShingleSize="5" outputUnigrams="false" outputUnigramsIfNoShingles="false" maxShingleSize="5" tokenSeparator=" "/>
      <filter class="org.apache.lucene.analysis.minhash.MinHashFilterFactory" bucketCount="512" hashSetSize="1" hashCount="1"/>
    </analyzer>
  </fieldType>
...

  <field name="min_hash_analysed" type="text_min_hash" multiValued="false" indexed="true" stored="false" />

Here, the input text is split on whitespace, the tokens normalised, the resulting token stream assembled into a stream of all the 5 word shingles which are then hashed. The lowest hashes from each of 512 buckets are kept and produced as the output tokens.

Queries to this field would need to generate at least one shingle so would require 5 distinct tokens.

Example queries:

 {!min_hash field="min_hash_analysed"}At least five or more tokens

 {!min_hash field="min_hash_analysed" sim="0.5"}At least five or more tokens

 {!min_hash field="min_hash_analysed" sim="0.5" tp="0.5"}At least five or more tokens

Example with Pre-Analysed Fields

Here, the MinHash is pre-computed, most likely using Lucene analysis inline as shown below. It would be more prudent to get the analyser from the schema.

    ICUTokenizerFactory factory = new ICUTokenizerFactory(Collections.EMPTY_MAP);
    factory.inform(null);
    Tokenizer tokenizer = factory.create();
    tokenizer.setReader(new StringReader(text));
    ICUFoldingFilterFactory filter = new ICUFoldingFilterFactory(Collections.EMPTY_MAP);
    TokenStream ts = filter.create(tokenizer);
    HashMap<String, String> args = new HashMap<>();
    args.put("minShingleSize", "5");
    args.put("outputUnigrams", "false");
    args.put("outputUnigramsIfNoShingles", "false");
    args.put("maxShingleSize", "5");
    args.put("tokenSeparator", " ");
    ShingleFilterFactory sff = new ShingleFilterFactory(args);
    ts = sff.create(ts);
    HashMap<String, String> args2 = new HashMap<>();
    args2.put("bucketCount", "512");
    args2.put("hashSetSize", "1");
    args2.put("hashCount", "1");
    MinHashFilterFactory mhff = new MinHashFilterFactory(args2);
    ts = mhff.create(ts);

    CharTermAttribute termAttribute = ts.getAttribute(CharTermAttribute.class);

    ts.reset();
    while (ts.incrementToken())
    {
        char[] buff = termAttribute.buffer();
        ...
     }
     ts.end();

The schema will just define a multi-valued string value and an optional field to use at anlysis time - similar to above.

 <field name="min_hash_string" type="strings" multiValued="true" indexed="true" stored="true"/>

 <!-- Optional -->
 <field name="min_hash_analysed" type="text_min_hash" multiValued="false" indexed="true" stored="false"/>

 <fieldType name="strings" class="solr.StrField" sortMissingLast="true" multiValued="true"/>

 <!-- Optional -->
 <fieldType name="text_min_hash" class="solr.TextField" positionIncrementGap="100">
    <analyzer>
      <tokenizer class="solr.ICUTokenizerFactory"/>
      <filter class="solr.ICUFoldingFilterFactory"/>
      <filter class="solr.ShingleFilterFactory" minShingleSize="5" outputUnigrams="false" outputUnigramsIfNoShingles="false" maxShingleSize="5" tokenSeparator=" "/>
      <filter class="org.apache.lucene.analysis.minhash.MinHashFilterFactory" bucketCount="512" hashSetSize="1" hashCount="1"/>
    </analyzer>
  </fieldType>

Example queries:

{!min_hash field="min_hash_string" sep=","}HASH1,HASH2,HASH3

{!min_hash field="min_hash_string" sim="0.9" analyzer_field="min_hash_analysed"}Lets hope the config and code for analysis are in sync

It is also possible to query analysed fields using known hashes (the reverse of the above)

{!min_hash field="min_hash_analysed" analyzer_field="min_hash_string" sep=","}HASH1,HASH2,HASH3

Pre-analysed fields mean hash values can be recovered per document rather than re-hashed. An initial query stage that returns the minhash stored field could be followed by a min_hash query to find similar documents.

Banded Queries

The default behaviour of the query parser, given the configuration above is to generate a boolean query and OR 512 constant score term queries together: one for each hash. In this case, generating a score of 1 if one hash matches and a score of 512 if they all match.

A banded query mixes conjunctions and disjunctions. We could have 256 bands each of two queries ANDed together, 128 with 4 hashes ANDed together etc. With fewer bands query performance increases but we may miss some matches. There is a trade off between speed and accuracy. With 64 bands the score will range from 0 to 64 (the number of bands ORed together)

Given the required similarity and an acceptable true positive rate, the query parser computes the appropriate band size^[1]. It finds the minimum number of bands subject to

\(tp \leq 1 - (1 - sim^{rows})^{bands}\)

If there are not enough hashes to fill the final band of the query it wraps to the start.

A Note on Similarity

Low similarities can be meaningful. The number of 5 word hashes is large. Even a single match may indicate some kind of similarity either in meaning, style or structure.

Further Reading

For a general introduction see "Mining of Massive Datasets"^[1].

For documents of ~1500 words expect an index size overhead of ~10%; your milage will vary. 512 hashes would be expected to represent ~2500 words well.

Using a set of MinHash values was proposed in the initial paper^[2] but provides a biased estimate of Jaccard similarity. There may be cases where that bias is a good thing. Likewise with rotation and short documents. The implementation is derived from an unbiased method proposed in later work^[3].

^[1] Leskovec, Jure; Rajaraman, Anand & Ullman, Jeffrey D. "Mining of Massive Datasets", Cambridge University Press; 2 edition (December 29, 2014), Chapter 3, ISBN: 9781107077232.

^[2] Broder, Andrei Z. (1997), "On the resemblance and containment of documents", Compression and Complexity of Sequences: Proceedings, Positano, Amalfitan Coast, Salerno, Italy, June 11-13, 1997 (PDF), IEEE, pp. 21–29, doi:10.1109/SEQUEN.1997.666900.

^[3] Shrivastava, Anshumali & Li, Ping (2014), "Improved Densification of One Permutation Hashing", 30th Conference on Uncertainty in Artificial Intelligence (UAI), Quebec City, Quebec, Canada, July 23-27, 2014, AUAI, pp. 225-234, http://www.auai.org/uai2014/proceedings/individuals/225.pdf

Payload Score Parser

PayloadScoreQParser incorporates each matching term’s numeric (integer or float) payloads into the scores.

This parser accepts the following parameters:

Examples

{!payload_score f=my_field_dpf v=some_term func=max}

{!payload_score f=payload_field func=sum operator=or}A B C

Payload Check Parser

PayloadCheckQParser only matches when the matching terms also have the specified payloads.

This parser accepts the following parameters:

Example

{!payload_check f=words_dps payloads="VERB NOUN"}searching stuff

Customizing XML Query Parser

You can configure your own custom query builders for additional XML elements. The custom builders need to extend the SolrQueryBuilder or the SolrSpanQueryBuilder class. Example solrconfig.xml snippet:

<queryParser name="xmlparser" class="XmlQParserPlugin">
  <str name="MyCustomQuery">com.mycompany.solr.search.MyCustomQueryBuilder</str>
</queryParser>