Updating Parts of Documents

Atomic Updates

Solr supports several modifiers that atomically update values of a document. This allows updating only specific fields, which can help speed indexing processes in an environment where speed of index additions is critical to the application.

To use atomic updates, add a modifier to the field that needs to be updated. The content can be updated, added to, or incrementally increased if the field has a numeric type.

{"id":"mydoc",
 "price":10,
 "popularity":42,
 "categories":["kids"],
 "sub_categories":["under_5","under_10"],
 "promo_ids":["a123x"],
 "tags":["free_to_try","buy_now","clearance","on_sale"]
}

{"id":"mydoc",
 "price":{"set":99},
 "popularity":{"inc":20},
 "categories":{"add":["toys","games"]},
 "sub_categories":{"add-distinct":"under_10"},
 "promo_ids":{"remove":"a123x"},
 "tags":{"remove":["free_to_try","on_sale"]}
}

{"id":"mydoc",
 "price":99,
 "popularity":62,
 "categories":["kids","toys","games"],
 "sub_categories":["under_5","under_10"],
 "tags":["buy_now","clearance"]
}

{
 "id":"mydoc",
 "product":"T-Shirt",
 "stock": {
           "id":"mydoc2",
           "color":"red",
           "size": ["L"]
           }
}

{
 "id":"mydoc",
 "stock": {
           "add":
                 {
                  "id":"mydoc3",
                  "color":"blue",
                  "size": ["M"]
                 }
          }
}

{
 "id":"mydoc",
 "product":"T-Shirt",
 "stock": [{
            "id":"mydoc2",
            "color":"red",
            "size": ["L"]
           },
           {
            "id":"mydoc3",
            "color":"blue",
            "size": ["M"]
           }]
}

{
 "id":"mydoc2",
 "size": {"add": ["S"]}
}

{
 "id":"mydoc",
 "product":"T-Shirt",
 "stock": [{
            "id":"mydoc2",
            "color":"red",
            "size": ["L", "S"]
           },
           {
            "id":"mydoc3",
            "color":"blue",
            "size": ["M"]
           }]
}

In-Place Updates

In-place updates are very similar to atomic updates; in some sense, this is a subset of atomic updates. In regular atomic updates, the entire document is reindexed internally during the application of the update. However, in this approach, only the fields to be updated are affected and the rest of the documents are not reindexed internally. Hence, the efficiency of updating in-place is unaffected by the size of the documents that are updated (i.e., number of fields, size of fields, etc.). Apart from these internal differences, there is no functional difference between atomic updates and in-place updates.

An atomic update operation is performed using this approach only when the fields to be updated meet these three conditions:

To use in-place updates, add a modifier to the field that needs to be updated. The content can be updated or incrementally increased.

{
 "id":"mydoc",
 "price":10,
 "popularity":42,
 "categories":["kids"],
 "promo_ids":["a123x"],
 "tags":["free_to_try","buy_now","clearance","on_sale"]
}

{
 "id":"mydoc",
 "price":{"set":99},
 "popularity":{"inc":20}
}

{
 "id":"mydoc",
 "price":99,
 "popularity":62,
 "categories":["kids"],
 "promo_ids":["a123x"],
 "tags":["free_to_try","buy_now","clearance","on_sale"]
}

Optimistic Concurrency

Optimistic Concurrency is a feature of Solr that can be used by client applications which update/replace documents to ensure that the document they are replacing/updating has not been concurrently modified by another client application. This feature works by requiring a _version_ field on all documents in the index, and comparing that to a _version_ specified as part of the update command. By default, Solr’s Schema includes a _version_ field, and this field is automatically added to each new document.

In general, using optimistic concurrency involves the following work flow:

When the client resubmits a changed document to Solr, the _version_ can be included with the update to invoke optimistic concurrency control. Specific semantics are used to define when the document should be updated or when to report a conflict.

If the document being updated does not include the _version_ field, and atomic updates are not being used, the document will be treated by normal Solr rules, which is usually to discard the previous version.

When using Optimistic Concurrency, clients can include an optional versions=true request parameter to indicate that the new versions of the documents being added should be included in the response. This allows clients to immediately know what the _version_ is of every document added without needing to make a redundant /get request.

Following are some examples using versions=true in queries:

$ curl -X POST -H 'Content-Type: application/json' 'http://localhost:8983/solr/techproducts/update?versions=true' --data-binary '
[ { "id" : "aaa" },
  { "id" : "bbb" } ]'

{"responseHeader":{"status":0,"QTime":6},
 "adds":["aaa",1498562471222312960,
         "bbb",1498562471225458688]}

In this example, we have added 2 documents "aaa" and "bbb". Because we added versions=true to the request, the response shows the document version for each document.

$ curl -X POST -H 'Content-Type: application/json' 'http://localhost:8983/solr/techproducts/update?_version_=999999&versions=true' --data-binary '
[{ "id" : "aaa",
   "foo_s" : "update attempt with wrong existing version" }]'

{"responseHeader":{"status":409,"QTime":3},
 "error":{"msg":"version conflict for aaa expected=999999 actual=1498562471222312960",
          "code":409}}

In this example, we’ve attempted to update document "aaa" but specified the wrong version in the request: version=999999 doesn’t match the document version we just got when we added the document. We get an error in response.

$ curl -X POST -H 'Content-Type: application/json' 'http://localhost:8983/solr/techproducts/update?_version_=1498562471222312960&versions=true&commit=true' --data-binary '
[{ "id" : "aaa",
   "foo_s" : "update attempt with correct existing version" }]'

{"responseHeader":{"status":0,"QTime":5},
 "adds":["aaa",1498562624496861184]}

Now we’ve sent an update with a value for _version_ that matches the value in the index, and it succeeds. Because we included versions=true to the update request, the response includes a different value for the _version_ field.

$ curl 'http://localhost:8983/solr/techproducts/query?q=*:*&fl=id,_version_'

{
  "responseHeader":{
    "status":0,
    "QTime":5,
    "params":{
      "fl":"id,_version_",
      "q":"*:*"}},
  "response":{"numFound":2,"start":0,"docs":[
      {
        "id":"bbb",
        "_version_":1498562471225458688},
      {
        "id":"aaa",
        "_version_":1498562624496861184}]
  }}

Finally, we can issue a query that requests the _version_ field be included in the response, and we can see that for the two documents in our example index.

For more information, please also see Yonik Seeley’s presentation on NoSQL features in Solr 4 from Apache Lucene EuroCon 2012.

Document Centric Versioning Constraints

Optimistic Concurrency is extremely powerful, and works very efficiently because it uses an internally assigned, globally unique values for the _version_ field. However, in some situations users may want to configure their own document specific version field, where the version values are assigned on a per-document basis by an external system, and have Solr reject updates that attempt to replace a document with an "older" version. In situations like this the DocBasedVersionConstraintsProcessorFactory can be useful.

The basic usage of DocBasedVersionConstraintsProcessorFactory is to configure it in solrconfig.xml as part of the UpdateRequestProcessorChain and specify the name of your custom versionField in your schema that should be checked when validating updates:

<processor class="solr.DocBasedVersionConstraintsProcessorFactory">
  <str name="versionField">my_version_l</str>
</processor>

Note that versionField is a comma delimited list of fields to check for version numbers. Once configured, this update processor will reject (HTTP error code 409) any attempt to update an existing document where the value of the my_version_l field in the "new" document is not greater then the value of that field in the existing document.

DocBasedVersionConstraintsProcessorFactory supports the following additional configuration parameters, which are all optional:

Please consult the DocBasedVersionConstraintsProcessorFactory javadocs and test solrconfig.xml file for additional information and example usages.