Package Manager Internals

The package manager (CLI) internally uses various Solr APIs to install, deploy and update packages. This document contains an overview of those APIs.

Salient Features

  • Zero disruption deployment (hot deployment): Should be possible to install and update packages without node restarts or core reloads, and hence deployments should be quick and without failed requests or stale caches.

  • Easy packaging:

    • Standard plugin concepts, e.g., query parser, search component, request handler, URP etc., should be supported without any special code/packaging changes.

    • Artifacts (jars containing custom plugins) that users have already deployed (and are using in production) should be compatible, without needing to recompile or re-package, for greater adoption.

    • Should support single jar packages as well as multiple jar packages.

    • Use familiar / standard naming

    • Use industry standard concepts and terminology around package manager, similar to those like apt, dnf, homebrew etc.s

Classloaders

At the heart of the system, we have classloader isolation. To achieve this, the system is simplified into two layered classloaders:

  • The root classloader which has all the jars from Solr classpath. This requires Solr node restart to change anything.

  • A set of named classloaders that inherit from the root classloader. The life cycles of the named classloaders are tied to the package configuration in ZooKeeper. As soon as the configuration is modified, the corresponding classloaders are reloaded and components are asked to reload.

Package Loading Security

Packages are disabled by default. Start all your nodes with the system property -Denable.packages=true to use this feature.

Example

bin/solr start -c -Denable.packages=true

Upload Your Keys

Package binaries must be signed with your private keys and ensure your public keys are published in package store’s trusted store.

Example

openssl genrsa -out my_key.pem 512
# create the public key in .der format
openssl rsa -in my_key.pem -pubout -outform DER -out my_key.der
# upload key to package store
bin/solr package add-key my_key.der

Package Store

Package store is a distributed file store which can store arbitrary files in the file system.

  • This is a fully replicated file system based repository.

  • It lives at <solr.home>/filestore on each Solr node.

  • Every entry is a file + metadata. The metadata is named .<filename>.json.

  • The metadata file contains the sha256, signatures of the file.

  • Users can’t create files starting with period (.).

  • It is agnostic of content type of files. You may store jars as well as other files.

How Does the Package Store Work?

When a file is uploaded to the Package Store, the following is true:

  • It’s saved to the local file system.

  • It’s saved along with the metadata. The metadata file stores the sha512 and signatures of the jar files too.

  • Every live node in the cluster is asked to download it as well.

Package Store APIs

The end points are:

  • PUT /api/cluster/files/{full/path/to/file} in each node

  • GET /api/node/files/{full/path/to/file} to download the file

  • GET /api/node/files/{full/path/to/file}?meta=true get the metadata of the file

  • GET /api/node/files/{full/path/to/} get the list of files in /full/path/to

Signing Your Artifacts

Use the following steps to upload a jar signed with your public key:

  1. If you don’t have a jar file with plugins, download a sample from GitHub:

    curl -o runtimelibs.jar   -LO https://github.com/apache/solr/blob/releases/solr/9.6.1/solr/core/src/test-files/runtimecode/runtimelibs.jar.bin?raw=true

  2. Sign the jar with your private key:

    openssl dgst -sha1 -sign my_key.pem runtimelibs.jar | openssl enc -base64 | sed 's/+/%2B/g' | tr -d \\n | sed

  3. Upload your jar with signature, replacing the sig parameter with the output from the previous command:

    curl --data-binary @runtimelibs.jar -X PUT  http://localhost:8983/api/cluster/files/mypkg/1.0/myplugins.jar?sig=<signature-of-jar>

  4. Verify your jar upload:

    curl http://localhost:8983/api/node/files/mypkg/1.0?omitHeader=true
    {
  "files":{"/mypkg/1.0":[{
  "name":"myplugins.jar",
  "timestamp":"2019-11-11T07:36:17.354Z",
  "sha512":"d01b51de67ae1680a84a813983b1de3b592fc32f1a22b662fc9057da5953abd1b72476388ba342cad21671cd0b805503c78ab9075ff2f3951fdf75fa16981420",
  "sig":["elNjhmWIOgTgbAzeZ+OcwR42N7vqL6Ig9eAqn4YoP2thT7FJuhiaZuCPivjMkD682EBo9gveSCTyXIsZKjOCbQ=="]}]}}

Packages

A Package has the following attributes:

  • A unique name

  • One or more versions with the following attributes:

    • version: The version string

    • files: An array of files from the package store

For every package/version in the packages definition, there is a unique SolrResourceLoader instance. This is a child of the CoreContainer resource loader.

Solr does not require that the version string follows any particular format - it can be an arbitrary string or even an empty string.

packages.json

The package configurations live in a file called packages.json in ZooKeeper. At any given moment we can have multiple versions of a given package in the package configuration. The system will always use the latest version. Versions are sorted by their values in lexicographic order, and the largest string is considered to be the latest.

Lexicographic order for version strings means that for a package with versions 1.2.0, 1.9.0, 1.11.0, Solr would pick 1.9.0 as the latest version.

For example:

{
 "packages" : {
   "mypkg" : {
     "name": "mypkg",
     "versions": [
       {"version" : "0.1",
       "files" : ["/path/to/myplugin/1.1/plugin.jar"]
       },
       {"version" :  "0.2",
       "files" : ["/path/to/myplugin/1.0/plugin.jar"]
       }]}}}

API Endpoints

  • GET /api/cluster/package Get the list of packages

  • POST /api/cluster/package edit packages

    • add command: add a version of a package

    • delete command: delete a version of a package

How to Upgrade?

Use the add command to add a version that is higher than the current version.

How to Downgrade?

Use the delete command to delete the highest version and choose the next highest version.

Using Multiple Versions in Parallel

We use params.json in the collection config to store a version of a package it uses. By default, it is the $LATEST.

{"params":{
 "PKG_VERSIONS": {
   "mypkg": "0.1", (1)
   "pkg2" : "$LATEST", (2)
 }}}
1 For mypkg, use the version 0.1 irrespective of whether there is a newer version available or not.
2 For pkg2, use the latest. This is optional. The default is $LATEST.

The package version in params.json actually instructs Solr to pick the package with the largest version that is not greater than the provided value.

So in the example above, if the only available versions for mypkg are 0.01 and 0.2, the version 0.01 will be used.

Workflow

  • A new version of a package is added.

  • The package loader loads the classes and notifies every plugin holder of the availability of the new version.

  • It checks if it is supposed to use a specific version, Ignore the update.

  • If not, reload the plugin.

Using Packages in Plugins

Any class name can be prefixed with the package name, e.g., mypkg:fully.qualified.ClassName and Solr would use the latest version of the package to load the classes from. The plugins loaded from packages cannot depend on core level classes.

Plugin declaration in solrconfig.xml
<requestHandler name="/myhandler" class="mypkg:full.path.to.MyClass">
</requestHandler>

Full Working Example

  1. Create a package:

    curl  http://localhost:8983/api/cluster/package -H 'Content-type:application/json' -d  '
{"add": {
         "package" : "mypkg",
         "version":"1.0",
         "files" :["/mypkg/1.0/myplugins.jar"]}}'

  2. Verify the created package:

    curl http://localhost:8983/api/cluster/package?omitHeader=true
      {"result":{
    "znodeVersion":0,
    "packages":{"mypkg":[{
          "version":"1.0",
          "files":["/mypkg/1.0/myplugins.jar"]}]}}}

  3. The package should be ready to use at this point. Next, register a plugin in your collection from the package. Note the mypkg: prefix applied to the class attribute. The same result can be achieved by editing your solrconfig.xml as well:

    curl http://localhost:8983/solr/gettingstarted/config -H 'Content-type:application/json' -d  '{
          "create-requesthandler": { "name": "/test",
          "class": "mypkg:org.apache.solr.core.RuntimeLibReqHandler" }}'

  4. Verify that the component is created, and it is using the correct version of the package to load classes from:

    curl http://localhost:8983/solr/gettingstarted/config/requestHandler?componentName=/test&meta=true&omitHeader=true
    {
  "config":{"requestHandler":{"/test":{
        "name":"/test",
        "class":"mypkg:org.apache.solr.core.RuntimeLibReqHandler",
        "_packageinfo_":{
          "package":"mypkg",
          "version":"1.0",
          "files":["/mypkg/1.0/myplugins.jar"]}}}}}

  5. Test the request handler:

    curl http://localhost:8983/solr/gettingstarted/test?omitHeader=true
    {
  "params":{
    "omitHeader":"true"},
  "context":{
    "webapp":"/solr",
    "path":"/test",
    "httpMethod":"GET"},
  "class":"org.apache.solr.core.RuntimeLibReqHandler",
  "loader":"java.net.FactoryURLClassLoader"}

  6. Update the version of our component. Get a new version of the jar, sign and upload it:

    curl -o runtimelibs3.jar   -LO https://github.com/apache/solr/blob/releases/solr/9.6.1/solr/core/src/test-files/runtimecode/runtimelibs_v3.jar.bin?raw=true

openssl dgst -sha1 -sign my_key.pem runtimelibs.jar | openssl enc -base64 | sed 's/+/%2B/g' | tr -d \\n | sed

curl --data-binary @runtimelibs3.jar -X PUT  http://localhost:8983/api/cluster/files/mypkg/2.0/myplugins.jar?sig=

  7. Verify it:

    curl http://localhost:8983/api/node/files/mypkg/2.0?omitHeader=true
    {
  "files":{"/mypkg/2.0":[{
        "name":"myplugins.jar",
        "timestamp":"2019-11-11T11:46:14.771Z",
        "sha512":"60ec88c2a2e9b409f7afc309273383810a0d07a078b482434eda9674f7e25b8adafa8a67c9913c996cbfb78a7f6ad2b9db26dbd4fe0ca4068f248d5db563f922",
        "sig":["ICkC+nGE+AqiANM0ajhVPNCQsbPbHLSWlIe5ETV5835e5HqndWrFHiV2R6nLVjDCxov/wLPo1uK0VzvAPIioUQ=="]}]}}

  8. Add a new version of the package:

    curl  http://localhost:8983/api/cluster/package -H 'Content-type:application/json' -d  '
{"add": {
         "package" : "mypkg",
         "version":"2.0",
         "files" :["/mypkg/2.0/myplugins.jar"]}}'

  9. Verify the plugin to see if the correct version of the package is being used:

    curl http://localhost:8983/solr/gettingstarted/config/requestHandler?componentName=/test&meta=true&omitHeader=true
    {
  "config": {
    "requestHandler": {
      "/test": {
        "name": "/test",
        "class": "mypkg:org.apache.solr.core.RuntimeLibReqHandler",
        "_packageinfo_": {
          "package": "mypkg",
          "version": "2.0",
          "files": [
            "/mypkg/2.0/myplugins.jar"
          ]
        }}}}}

  10. Test the plugin:

    curl http://localhost:8983/solr/gettingstarted/test?omitHeader=true
    {
  "params": {
    "omitHeader": "true"
  },
  "context": {
    "webapp": "/solr",
    "path": "/test",
    "httpMethod": "GET"
  },
  "class": "org.apache.solr.core.RuntimeLibReqHandler",
  "loader": "java.net.FactoryURLClassLoader",
  "Version": "2"
}

    Note that the Version value is "2", which means the plugin is updated.

How to Avoid Automatic Upgrade

The default version used in any collection is always the latest. However, setting a per-collection property in params.json ensures that the collection uses the same package version (i.e., version 2.0), irrespective of any versions later than 2.0 that may be added to Solr at a later point.

curl http://localhost:8983/solr/gettingstarted/config/params -H 'Content-type:application/json'  -d '{
  "set":{
    "PKG_VERSIONS":{
      "mypkg":"2.0"
      }
  }}'