Indexing Nested Child Documents | Apache Solr Reference Guide 8.7

Solr supports indexing nested documents, described here, and ways to search and retrieve them very efficiently.

By way of examples: nested documents in Solr can be used to bind a blog post (parent document) with comments (child documents) — or as a way to model major product lines as parent documents, with multiple types of child documents representing individual SKUs (with unique sizes / colors) and supporting documention (either directly nested under the products, or under individual SKUs.

The "top most" parent with all children is referred to as a "root level" document or "block document" and it explains some of the nomenclature of related features.

At query time, the Block Join Query Parsers can search these relationships, and the [child] Document Transformer can attach child (or other "descendent") documents to the result documents. In terms of performance, indexing the relationships between documents usually yields much faster queries than an equivalent "query time join", since the relationships are already stored in the index and do not need to be computed.

However, nested documents are less flexible than query time joins as it imposes rules that some applications may not be able to accept. Nested documents may be indexed via either the XML or JSON data syntax, and is also supported by SolrJ with javabin.

Re-Indexing Considerations

With the exception of in-place updates, blocks of nested documents must be updated/deleted together. Modifying or replacing individual child documents requires reindexing of the entire block (either explicitly/externally, or under the covers inside of Solr). For some applications this may result in a lot of extra indexing overhead and may not be worth the performance gains at query time.

Example Indexing Syntax: Psuedo-Fields

This example shows what it looks like to index two root level "product" documents, each containing two different types of child documents specified in "psuedo-fields": "skus" and "manuals". Two of the "sku" type documents have their own nested child "manuals" documents…

Even though the child documents in these examples are provided syntactically as field values syntactically, this is simply a matter of syntax and as such skus and manuals are not actual fields in the documents. Consequently, these field names need not be defined in the schema and probably shouldn’t be as it would be confusing. There is no "child document" field type.

JSON

[{ "id": "P11!prod",
   "name_s": "Swingline Stapler",
   "description_t": "The Cadillac of office staplers ...",
   "skus": [ { "id": "P11!S21",
               "color_s": "RED",
               "price_i": 42,
               "manuals": [ { "id": "P11!D41",
                              "name_s": "Red Swingline Brochure",
                              "pages_i":1,
                              "content_t": "..."
                            } ]
             },
             { "id": "P11!S31",
               "color_s": "BLACK",
               "price_i": 3
             } ],
   "manuals": [ { "id": "P11!D51",
                  "name_s": "Quick Reference Guide",
                  "pages_i":1,
                  "content_t": "How to use your stapler ..."
                },
                { "id": "P11!D61",
                  "name_s": "Warranty Details",
                  "pages_i":42,
                  "content_t": "... lifetime guarantee ..."
                } ]
 },
 { "id": "P22!prod",
   "name_s": "Mont Blanc Fountain Pen",
   "description_t": "A Premium Writing Instrument ...",
   "skus": [ { "id": "P22!S22",
               "color_s": "RED",
               "price_i": 89,
               "manuals": [ { "id": "P22!D42",
                              "name_s": "Red Mont Blanc Brochure",
                              "pages_i":1,
                              "content_t": "..."
                            } ]
             },
             { "id": "P22!S32",
               "color_s": "BLACK",
               "price_i": 67
             } ],
   "manuals": [ { "id": "P22!D52",
                  "name_s": "How To Use A Pen",
                  "pages_i":42,
                  "content_t": "Start by removing the cap ..."
                } ]
 } ]

The /update/json/docs convenience path will automatically flatten complex JSON documents by default — so to index nested JSON documents make sure to use /update.

XML

<add>
  <doc>
    <field name="id">P11!prod</field>
    <field name="name_s">Swingline Stapler</field>
    <field name="description_t">The Cadillac of office staplers ...</field>
    <field name="skus">
      <doc>
        <field name="id">P11!S21</field>
        <field name="color_s">RED</field>
        <field name="price_i">42</field>
        <field name="manuals">
          <doc>
            <field name="id">P11!D41</field>
            <field name="name_s">Red Swingline Brochure</field>
            <field name="pages_i">1</field>
            <field name="content_t">...</field>
          </doc>
        </field>
      </doc>
      <doc>
        <field name="id">P11!S31</field>
        <field name="color_s">BLACK</field>
        <field name="price_i">3</field>
      </doc>
    </field>
    <field name="manuals">
      <doc>
        <field name="id">P11!D51</field>
        <field name="name_s">Quick Reference Guide</field>
        <field name="pages_i">1</field>
        <field name="content_t">How to use your stapler ...</field>
      </doc>
      <doc>
        <field name="id">P11!D61</field>
        <field name="name_s">Warranty Details</field>
        <field name="pages_i">42</field>
        <field name="content_t">... lifetime guarantee ...</field>
      </doc>
    </field>
  </doc>
  <doc>
    <field name="id">P22!prod</field>
    <field name="name_s">Mont Blanc Fountain Pen</field>
    <field name="description_t">A Premium Writing Instrument ...</field>
    <field name="skus">
      <doc>
        <field name="id">P22!S22</field>
        <field name="color_s">RED</field>
        <field name="price_i">89</field>
        <field name="manuals">
          <doc>
            <field name="id">P22!D42</field>
            <field name="name_s">Red Mont Blanc Brochure</field>
            <field name="pages_i">1</field>
            <field name="content_t">...</field>
          </doc>
        </field>
      </doc>
      <doc>
        <field name="id">P22!S32</field>
        <field name="color_s">BLACK</field>
        <field name="price_i">67</field>
      </doc>
    </field>
    <field name="manuals">
      <doc>
        <field name="id">P22!D52</field>
        <field name="name_s">How To Use A Pen</field>
        <field name="pages_i">42</field>
        <field name="content_t">Start by removing the cap ...</field>
      </doc>
    </field>
  </doc>
</add>

SolrJ

final SolrClient client = getSolrClient();

final SolrInputDocument p1 = new SolrInputDocument();
p1.setField("id", "P11!prod");
p1.setField("name_s", "Swingline Stapler");
p1.setField("description_t", "The Cadillac of office staplers ...");
{
  final SolrInputDocument s1 = new SolrInputDocument();
  s1.setField("id", "P11!S21");
  s1.setField("color_s", "RED");
  s1.setField("price_i", 42);
  {
    final SolrInputDocument m1 = new SolrInputDocument();
    m1.setField("id", "P11!D41");
    m1.setField("name_s", "Red Swingline Brochure");
    m1.setField("pages_i", 1);
    m1.setField("content_t", "...");

    s1.setField("manuals", m1);
  }

  final SolrInputDocument s2 = new SolrInputDocument();
  s2.setField("id", "P11!S31");
  s2.setField("color_s", "BLACK");
  s2.setField("price_i", 3);

  p1.setField("skus", Arrays.asList(s1, s2));
}
{
  final SolrInputDocument m1 = new SolrInputDocument();
  m1.setField("id", "P11!D51");
  m1.setField("name_s", "Quick Reference Guide");
  m1.setField("pages_i", 1);
  m1.setField("content_t", "How to use your stapler ...");

  final SolrInputDocument m2 = new SolrInputDocument();
  m2.setField("id", "P11!D61");
  m2.setField("name_s", "Warranty Details");
  m2.setField("pages_i", 42);
  m2.setField("content_t", "... lifetime guarantee ...");

  p1.setField("manuals", Arrays.asList(m1, m2));
}

final SolrInputDocument p2 = new SolrInputDocument();
p2.setField("id", "P22!prod");
p2.setField("name_s", "Mont Blanc Fountain Pen");
p2.setField("description_t", "A Premium Writing Instrument ...");
{
  final SolrInputDocument s1 = new SolrInputDocument();
  s1.setField("id", "P22!S22");
  s1.setField("color_s", "RED");
  s1.setField("price_i", 89);
  {
    final SolrInputDocument m1 = new SolrInputDocument();
    m1.setField("id", "P22!D42");
    m1.setField("name_s", "Red Mont Blanc Brochure");
    m1.setField("pages_i", 1);
    m1.setField("content_t", "...");

    s1.setField("manuals", m1);
  }

  final SolrInputDocument s2 = new SolrInputDocument();
  s2.setField("id", "P22!S32");
  s2.setField("color_s", "BLACK");
  s2.setField("price_i", 67);

  p2.setField("skus", Arrays.asList(s1, s2));
}
{
  final SolrInputDocument m1 = new SolrInputDocument();
  m1.setField("id", "P22!D52");
  m1.setField("name_s", "How To Use A Pen");
  m1.setField("pages_i", 42);
  m1.setField("content_t", "Start by removing the cap ...");

  p2.setField("manuals", m1);
}

client.add(Arrays.asList(p1, p2));

Schema Configuration

Indexing nested documents requires an indexed field named _root_:

<field name="_root_" type="string" indexed="true" />

Solr automatically populates this field in every nested document with the id value of the top most parent document in the block.

There are several additional schema considerations that should be considered for people who wish to use nested documents:

Nested child documents are very much documents in their own right even if certain nested documents hold different information from the parent, Therefore:
- All field names in the schema can only be configured in one — different types of child documents can not have the same field name configured in different ways.
- It may be infeasible to use required for any field names that aren’t reqiured for all types of documents.
- Even child documents need a globally unique id.
_root_ must be configured to either be stored (stored="true") or use doc values (docValues="true") to enable atomic updates of nested documents.
- Also, beware of uniqueBlock(_root_) field type limitation, if you plan to use one.
_nest_path_ is an optional field that (if definied) will be populated by Solr automatically with the ancestor path of each non-root document.
```
<fieldType name="_nest_path_" class="solr.NestPathField" />
<field name="_nest_path_" type="_nest_path_" />`
```
- This field is neccessary if you wish to use atomic updates of nested documents
- This field is neccessary in order for Solr to properly record & reconstruct the nested relationship of documents when using the [child] doc transformer.
  - If this field does not exist, the [child] transformer will return all descendent child documents as a flattened list — just as if they had been indexed as anonymous children.
- If you do not use _nest_path_ it is strongly recomended that every document have some field that differentiates root documents from their nested children — and differentiates different "types" of child documents. This is not strictly neccessary, so long as it’s possible to write a "filter" query that can be used to isolate and select only parent documents for use in the block join query parsers and [child] doc transformer

_nest_parent_ is an optional field that (if defined) will be populated by Solr automatically to store the id of each document’s immediate parent document (if there is one).

<field name="_nest_parent_" type="string" indexed="true" stored="true" />

When using SolrCloud it is a VERY good idea to use prefix based compositeIds with a common prefix for all documents in the block. This makes it much easier to apply atomic updates to individual child documents

Maintaining Integrity with Updates and Deletes

Blocks of nested documents can be modified simply by adding/replacing the root document with more or fewer child/descendent documents as an application desires. This can either be done explicitly/externaly by an indexing client completely reindexing the root level document, or internally by Solr when a client uses atomic updates to modify child documents. This aspect isn’t different than updating any normal document except that Solr takes care to ensure that all related child documents of the existing version get deleted.

Clients should however be very careful to never add a root document that has the same id of a child document — or vice-versa. Solr does not prevent clients from attempting this, but it will violate integrity assumptions that Solr expects.

To delete an entire block of documents, you can simply delete-by-ID using the id of the root document. Delete-by-ID will not work with the id of a child document, since only root document IDs are considered. (Instead, use atomic updates to remove the child document from it’s parent)

If you use Solr’s delete-by-query APIs, you MUST be careful to ensure that any deletion query is strutured to ensure no descendent children remain of any documents that are being deleted. Doing otherwise will violate integrity assumptions that Solr expects.

Indexing Anonymous Children

Although not recommended, it is also possible to index child documents "anonymously":

JSON

[{ "id": "P11!prod",
   "name_s": "Swingline Stapler",
   "type_s": "PRODUCT",
   "description_t": "The Cadillac of office staplers ...",
   "_childDocuments_": [
       { "id": "P11!S21",
         "type_s": "SKU",
         "color_s": "RED",
         "price_i": 42,
         "_childDocuments_": [
             { "id": "P11!D41",
               "type_s": "MANUAL",
               "name_s": "Red Swingline Brochure",
               "pages_i":1,
               "content_t": "..."
             } ]
       },
       { "id": "P11!S31",
         "type_s": "SKU",
         "color_s": "BLACK",
         "price_i": 3
       },
       { "id": "P11!D51",
         "type_s": "MANUAL",
         "name_s": "Quick Reference Guide",
         "pages_i":1,
         "content_t": "How to use your stapler ..."
       },
       { "id": "P11!D61",
         "type_s": "MANUAL",
         "name_s": "Warranty Details",
         "pages_i":42,
         "content_t": "... lifetime guarantee ..."
       }
    ]
} ]

XML

<add>
  <doc>
    <field name="id">P11!prod</field>
    <field name="type_s">PRODUCT</field>
    <field name="name_s">Swingline Stapler</field>
    <field name="description_t">The Cadillac of office staplers ...</field>
    <doc>
      <field name="id">P11!S21</field>
      <field name="type_s">SKU</field>
      <field name="color_s">RED</field>
      <field name="price_i">42</field>
      <doc>
        <field name="id">P11!D41</field>
        <field name="type_s">MANUAL</field>
        <field name="name_s">Red Swingline Brochure</field>
        <field name="pages_i">1</field>
        <field name="content_t">...</field>
      </doc>
    </doc>
    <doc>
      <field name="id">P11!S31</field>
      <field name="type_s">SKU</field>
      <field name="color_s">BLACK</field>
      <field name="price_i">3</field>
    </doc>
    <doc>
      <field name="id">P11!D51</field>
      <field name="type_s">MANUAL</field>
      <field name="name_s">Quick Reference Guide</field>
      <field name="pages_i">1</field>
      <field name="content_t">How to use your stapler ...</field>
    </doc>
    <doc>
      <field name="id">P11!D61</field>
      <field name="type_s">MANUAL</field>
      <field name="name_s">Warranty Details</field>
      <field name="pages_i">42</field>
      <field name="content_t">... lifetime guarantee ...</field>
    </doc>
  </doc>
</add>

SolrJ

final SolrClient client = getSolrClient();

final SolrInputDocument p1 = new SolrInputDocument();
p1.setField("id", "P11!prod");
p1.setField("type_s", "PRODUCT");
p1.setField("name_s", "Swingline Stapler");
p1.setField("description_t", "The Cadillac of office staplers ...");
{
  final SolrInputDocument s1 = new SolrInputDocument();
  s1.setField("id", "P11!S21");
  s1.setField("type_s", "SKU");
  s1.setField("color_s", "RED");
  s1.setField("price_i", 42);
  {
    final SolrInputDocument m1 = new SolrInputDocument();
    m1.setField("id", "P11!D41");
    m1.setField("type_s", "MANUAL");
    m1.setField("name_s", "Red Swingline Brochure");
    m1.setField("pages_i", 1);
    m1.setField("content_t", "...");

    s1.addChildDocument(m1);
  }

  final SolrInputDocument s2 = new SolrInputDocument();
  s2.setField("id", "P11!S31");
  s2.setField("type_s", "SKU");
  s2.setField("color_s", "BLACK");
  s2.setField("price_i", 3);

  final SolrInputDocument m1 = new SolrInputDocument();
  m1.setField("id", "P11!D51");
  m1.setField("type_s", "MANUAL");
  m1.setField("name_s", "Quick Reference Guide");
  m1.setField("pages_i", 1);
  m1.setField("content_t", "How to use your stapler ...");

  final SolrInputDocument m2 = new SolrInputDocument();
  m2.setField("id", "P11!D61");
  m2.setField("type_s", "MANUAL");
  m2.setField("name_s", "Warranty Details");
  m2.setField("pages_i", 42);
  m2.setField("content_t", "... lifetime guarantee ...");

  p1.addChildDocuments(Arrays.asList(s1, s2, m1, m2));
}

client.add(p1);

This simplified approach was common in older versions of Solr, and can still be used with "Root-Only" schemas that do not contain any other nested related fields apart from _root_. (Many schemas in existence are this way simply because default configsets are this way, even if the application isn’t using nested documents.)

This approach should NOT be used when schemas include a _nest_path_ field, as the existence of that field triggers assumptions and changes in behavior in various query time functionality, such as the [child], that will not work when nested documents do not have any intrinsic "nested path" information.

The results of indexing anonymous nested children with a "Root-Only" schema are similar to what happens if you attempt to index "psuedo field" nested documents using a "Root-Only" schema. Notably: since there is no nested path information for the [child] transformer to use to reconstruct the structured of a block of documents, it returns all matching children as a flat list, similar in structure to how they were originally indexed:

JSON

$ curl --globoff 'http://localhost:8983/solr/gettingstarted/select?omitHeader=true&q=id:P11!prod&fl=*,[child%20parentFilter=%22type_s:PRODUCT%22]'
{
  "response":{"numFound":1,"start":0,"maxScore":0.7002023,"numFoundExact":true,"docs":[
      {
        "id":"P11!prod",
        "name_s":"Swingline Stapler",
        "type_s":"PRODUCT",
        "description_t":"The Cadillac of office staplers ...",
        "_version_":1673055562829398016,
        "_childDocuments_":[
        {
          "id":"P11!D41",
          "type_s":"MANUAL",
          "name_s":"Red Swingline Brochure",
          "pages_i":1,
          "content_t":"...",
          "_version_":1673055562829398016},
        {
          "id":"P11!S21",
          "type_s":"SKU",
          "color_s":"RED",
          "price_i":42,
          "_version_":1673055562829398016},
        {
          "id":"P11!S31",
          "type_s":"SKU",
          "color_s":"BLACK",
          "price_i":3,
          "_version_":1673055562829398016},
        {
          "id":"P11!D51",
          "type_s":"MANUAL",
          "name_s":"Quick Reference Guide",
          "pages_i":1,
          "content_t":"How to use your stapler ...",
          "_version_":1673055562829398016},
        {
          "id":"P11!D61",
          "type_s":"MANUAL",
          "name_s":"Warranty Details",
          "pages_i":42,
          "content_t":"... lifetime guarantee ...",
          "_version_":1673055562829398016}]}]
  }}

XML

$ curl --globoff 'http://localhost:8983/solr/gettingstarted/select?omitHeader=true&q=id:P11!prod&fl=*,[child%20parentFilter=%22type_s:PRODUCT%22]&wt=xml'
<?xml version="1.0" encoding="UTF-8"?>
<response>

<result name="response" numFound="1" start="0" maxScore="0.7002023" numFoundExact="true">
  <doc>
    <str name="id">P11!prod</str>
    <str name="name_s">Swingline Stapler</str>
    <str name="type_s">PRODUCT</str>
    <str name="description_t">The Cadillac of office staplers ...</str>
    <long name="_version_">1673055562829398016</long>
    <doc>
      <str name="id">P11!D41</str>
      <str name="type_s">MANUAL</str>
      <str name="name_s">Red Swingline Brochure</str>
      <int name="pages_i">1</int>
      <str name="content_t">...</str>
      <long name="_version_">1673055562829398016</long></doc>
    <doc>
      <str name="id">P11!S21</str>
      <str name="type_s">SKU</str>
      <str name="color_s">RED</str>
      <int name="price_i">42</int>
      <long name="_version_">1673055562829398016</long></doc>
    <doc>
      <str name="id">P11!S31</str>
      <str name="type_s">SKU</str>
      <str name="color_s">BLACK</str>
      <int name="price_i">3</int>
      <long name="_version_">1673055562829398016</long></doc>
    <doc>
      <str name="id">P11!D51</str>
      <str name="type_s">MANUAL</str>
      <str name="name_s">Quick Reference Guide</str>
      <int name="pages_i">1</int>
      <str name="content_t">How to use your stapler ...</str>
      <long name="_version_">1673055562829398016</long></doc>
    <doc>
      <str name="id">P11!D61</str>
      <str name="type_s">MANUAL</str>
      <str name="name_s">Warranty Details</str>
      <int name="pages_i">42</int>
      <str name="content_t">... lifetime guarantee ...</str>
      <long name="_version_">1673055562829398016</long></doc></doc>
</result>
</response>