Authentication and Authorization

All well-configured Solr clusters should enforce users to authenticate, even for read-only operations. Even if you want to allow anonymous query requests from unknown users, you should make this explicit using Solr’s rule-based authorization plugin. In other words, always enforce security and then relax constraints as needed for specific endpoints based on your use case. The Solr operator can bootstrap a default security configuration for your SolrCloud during initialization. As such, there is no reason to deploy an unsecured SolrCloud cluster when using the Solr operator. In most cases, you’ll want to combine basic authentication with TLS to ensure credentials are never passed in clear text.

For background on Solr security, please refer to the Reference Guide for your version of Solr.

The Solr operator only supports the Basic authentication scheme. In general, you have two primary options for configuring authentication with the Solr operator:

  1. Let the Solr operator bootstrap the security.json to configure basic authentication for Solr.

  2. Supply your own security.json to Solr, which must define a user account that the operator can use to make API requests to secured Solr pods.

If you choose option 2, then you need to provide the credentials the Solr operator should use to make requests to Solr via a Kubernetes secret. With option 1, the operator creates a Basic Authentication Secret for you, which contains the username and password for the k8s-oper user.

Option 1: Bootstrap Security

The easiest way to get started with Solr security is to have the operator bootstrap a security.json (stored in ZK) as part of the initial deployment process. To activate this feature, add the following configuration to your SolrCloud CRD definition YAML:

spec:
  ...
  solrSecurity:
    authenticationType: Basic

Once the cluster is up, you’ll need the admin user password to login to the Solr Admin UI. The admin user will have a random password generated by the operator during security.json bootstrapping. Use the following command to retrieve the password from the bootstrap secret created by the operator:

kubectl get secret <CLOUD>-solrcloud-security-bootstrap -o jsonpath='{.data.admin}' | base64 --decode

where <CLOUD> is the name of your SolrCloud

Once security.json is bootstrapped, the operator will not update it! You’re expected to use the admin user to access the Security API to make further changes. In addition to the admin user, the operator defines a solr user, which has basic read access to Solr resources. You can retrieve the solr user password using:

kubectl get secret <CLOUD>-solrcloud-security-bootstrap -o jsonpath='{.data.solr}' | base64 --decode

You can safely delete the bootstrap secret, provided you’ve captured the admin password, after your SolrCloud deploys with the bootstrapped security.json. However, this will trigger a rolling restart across all pods as the setup-zk initContainer definition changes.

k8s-oper user

The operator makes requests to secured Solr endpoints as the k8s-oper user; credentials for the k8s-oper user are stored in a separate secret of type kubernetes.io/basic-auth with name <CLOUD>-solrcloud-basic-auth. The k8s-oper user is configured with read-only access to a minimal set of endpoints, see details in the Authorization sub-section below. Remember, if you change the k8s-oper password using the Solr security API, then you must update the secret with the new password or the operator will be locked out. Also, changing the password for the k8s-oper user in the K8s secret after bootstrapping will not update Solr! You’re responsible for changing the password in both places.

Liveness and Readiness Probes

We recommend configuring Solr to allow un-authenticated access over HTTP to the probe endpoint(s) and the bootstrapped security.json does this for you automatically (see next sub-section). However, if you want to secure the probe endpoints, then you need to set probesRequireAuth: true as shown below:

spec:
  ...
  solrSecurity:
    authenticationType: Basic
    probesRequireAuth: true

When probesRequireAuth is set to true, the liveness and readiness probes execute a command instead of using HTTP. The operator configures a command instead of setting the Authorization header for the HTTP probes, as that would require a restart of all pods if the password changes. With a command, we can load the username and password from a secret; Kubernetes will update the mounted secret files when the secret changes automatically.

By default, the operator will use Solr’s /admin/info/system endpoint for startup and liveness probes, and the /admin/info/health endpoint for readiness probes. This default can be customized by changing the HTTP path for any probes (under spec.customSolrKubeOptions.podOptions), however this also requires users to set probesRequireAuth=false as the operator does not reconfigure custom HTTP probes to use the command needed to support probesRequireAuth=true.

Custom readiness and liveness probes can be specified using configuration like the following:

spec:
  ...
  customSolrKubeOptions:
    podOptions:
      livenessProbe:
        httpGet:
          scheme: HTTP
          path: /solr/admin/customliveness
          port: 8983
      readinessProbe:
        httpGet:
          scheme: HTTP
          path: /solr/admin/customreadiness
          port: 8983

Consequently, the bootstrapped security.json will include additional rules to allow access to the endpoints used by the startup, liveness, and readiness probes:

      {
        "name": "k8s-probe-1",
        "role": null,
        "collection": null,
        "path": "/admin/customliveness"
      },
      {
        "name": "k8s-probe-2",
        "role": null,
        "collection": null,
        "path": "/admin/customreadiness"
      }

Note, if you change the probes after creating your solrcloud, then the new probe paths will not be added to the security.json. The security file is bootstrapped just once, so if your probes need to change you must add it to the allowed paths via the Solr Security API using the admin credentials.

Authorization

The default security.json created by the operator during initialization is shown below; the passwords for each user are randomized for every SolrCloud you create. In addition to configuring the solr.BasicAuthPlugin, the operator initializes a set of authorization rules for the default user accounts: admin, solr, and k8s-oper. Take a moment to review these authorization rules so that you’re aware of the roles and access granted to each user in your cluster.

{
  "authentication": {
    "blockUnknown": false,
    "class": "solr.BasicAuthPlugin",
    "credentials": {
      "admin": "...",
      "k8s-oper": "...",
      "solr": "..."
    },
    "realm": "Solr Basic Auth",
    "forwardCredentials": false
  },
  "authorization": {
    "class": "solr.RuleBasedAuthorizationPlugin",
    "user-role": {
      "admin": [ "admin", "k8s" ],
      "k8s-oper": [ "k8s" ],
      "solr": [ "users", "k8s" ]
    },
    "permissions": [
      {
        "name": "k8s-probe-0",
        "role": null,
        "collection": null,
        "path": "/admin/info/system"
      },
      {
        "name": "k8s-probe-1",
        "role": null,
        "collection": null,
        "path": "/admin/info/health"
      },
      {
        "name": "k8s-status",
        "role": "k8s",
        "collection": null,
        "path": "/admin/collections"
      },
      {
        "name": "k8s-metrics",
        "role": "k8s",
        "collection": null,
        "path": "/admin/metrics"
      },
      {
         "name": "k8s-zk",
         "role":"k8s",
         "collection": null,
         "path":"/admin/zookeeper/status"
      },
      {
        "name": "k8s-ping",
        "role": "k8s",
        "collection": "*",
        "path": "/admin/ping"
      },
      {
        "name": "read",
        "role": [ "admin", "users" ]
      },
      {
        "name": "update",
        "role": [ "admin" ]
      },
      {
        "name": "security-read",
        "role": [ "admin" ]
      },
      {
        "name": "security-edit",
        "role": [ "admin" ]
      },
      {
        "name": "all",
        "role": [ "admin" ]
      }
    ]
  }
}

A few aspects of the default security.json configuration warrant a closer look. First, the probesRequireAuth setting (defaults to false) governs the value for blockUnknown (under authentication) and whether the probe endpoint(s) require authentication:

      {
        "name": "k8s-probe-0",
        "role": null,
        "collection": null,
        "path": "/admin/info/system"
      }

In this case, the "role":null indicates this endpoint allows anonymous access by unknown users. The "collection":null value indicates the path is not associated with any collection, i.e. it is a top-level system path.

The operator sends GET requests to the /admin/collections endpoint to get cluster status to determine the rolling restart order:

      {
        "name": "k8s-status",
        "role": "k8s",
        "collection": null,
        "path": "/admin/collections"
      },

In this case, the "role":"k8s" indicates the requesting user must be in the k8s role; notice that all default users have the k8s role.

The Prometheus exporter sends GET requests to the /admin/metrics endpoint to collect metrics from each pod. The exporter also hits the /admin/ping endpoint for every collection, which requires the following authorization rules:

      {
        "name": "k8s-metrics",
        "role": "k8s",
        "collection": null,
        "path": "/admin/metrics"
      },
      {
        "name": "k8s-ping",
        "role": "k8s",
        "collection": "*",
        "path": "/admin/ping"
      },
      {
         "name": "k8s-zk",
         "role":"k8s",
         "collection": null,
         "path":"/admin/zookeeper/status"
      },

The "collection":"*" setting indicates this path applies to all collections, which maps to endpoint /collections/<COLL>/admin/ping at runtime.

The initial authorization config grants the read permission to the users role, which allows users to send query requests but cannot add / update / delete documents. For instance, the solr user is mapped to the users role, so the solr user can send query requests only. In general, please verify the initial authorization rules for each role before sharing user credentials.

Option 2: User-provided security.json and credentials secret

If users want full control over their cluster’s security config, then they can provide the Solr security.json via a Secret and the credentials the operator should use to make requests to Solr in a Secret.

Custom security.json Secret

For full control over the Solr security configuration, supply a security.json in a Secret. The following example illustrates how to point the operator to a Secret containing a custom security.json:

spec:
  ...
  solrSecurity:
    authenticationType: Basic
    bootstrapSecurityJson:
      name: my-custom-security-json
      key: security.json

For Basic authentication, if you don’t supply a security.json Secret, then the operator assumes you are bootstrapping the security configuration via some other means.

Refer to the example security.json shown in the Authorization section above to help you get started crafting your own custom configuration.

Basic Authentication

For Basic authentication, the supplied secret must be of type Basic Authentication Secret and define both a username and password.

spec:
  ...
  solrSecurity:
    authenticationType: Basic
    basicAuthSecret: user-provided-secret

Here is an example of how to define a basic auth secret using YAML:

apiVersion: v1
kind: Secret
metadata:
  name: my-basic-auth-secret
type: kubernetes.io/basic-auth
stringData:
  username: k8s-oper
  password: Test1234

With this config, the operator will make API requests to secured Solr pods as the k8s-oper user. Note: be sure to use a stronger password for real deployments

Users need to ensure their security.json contains the user supplied in the basicAuthSecret has read access to the following endpoints:

/admin/info/system
/admin/info/health
/admin/collections
/admin/metrics
/admin/ping (for collection="*")
/admin/zookeeper/status

Tip: see the authorization rules defined by the default security.json as a guide for configuring access for the operator user

Changing the Password

If you change the password for the user configured in your basicAuthSecret using the Solr security API, then you must update the secret with the new password or the operator will be locked out. Also, changing the password for this user in the K8s secret will not update Solr! You’re responsible for changing the password in both places.

Prometheus Exporter with Basic Auth

If you enable basic auth for your SolrCloud cluster, then you need to point the Prometheus exporter at the basic auth secret; refer to Prometheus Exporter with Basic Auth for more details.