Authentication and Authorization

All well-configured Solr clusters should enforce users to authenticate, even for read-only operations. Even if you want to allow anonymous query requests from unknown users, you should make this explicit using Solr’s rule-based authorization plugin. In other words, always enforce security and then relax constraints as needed for specific endpoints based on your use case. The Solr operator can bootstrap a default security configuration for your SolrCloud during initialization. As such, there is no reason to deploy an unsecured SolrCloud cluster when using the Solr operator. In most cases, you’ll want to combine basic authentication with TLS to ensure credentials are never passed in clear text.

For background on Solr security, please refer to the Reference Guide for your version of Solr.

Basic authentication is the only authentication scheme supported by the Solr operator at this time. In general, you have two basic options for configuring basic authentication with the Solr operator:

  1. Let the Solr operator bootstrap the security.json to configure basic authentication for Solr.

  2. Supply your own security.json to Solr, which must define a user account that the operator can use to make API requests to secured Solr pods.

If you choose option 2, then you need to provide the credentials to the Solr operator using a Kubernetes Basic Authentication Secret. With option 1, the operator creates the Basic Authentication Secret for you.

Option 1: Bootstrap Security

The easiest way to get started with Solr security is to have the operator bootstrap a security.json (stored in ZK) as part of the initial deployment process. To activate this feature, add the following configuration to your SolrCloud CRD definition YAML:

spec:
  ...
  solrSecurity:
    authenticationType: Basic

Once the cluster is up, you’ll need the admin user password to login to the Solr Admin UI. The admin user will have a random password generated by the operator during security.json bootstrapping. Use the following command to retrieve the password from the bootstrap secret created by the operator:

kubectl get secret <CLOUD>-solrcloud-security-bootstrap -o jsonpath='{.data.admin}' | base64 --decode

where <CLOUD> is the name of your SolrCloud

Once security.json is bootstrapped, the operator will not update it! You’re expected to use the admin user to access the Security API to make further changes. In addition to the admin user, the operator defines a solr user, which has basic read access to Solr resources. You can retrieve the solr user password using:

kubectl get secret <CLOUD>-solrcloud-security-bootstrap -o jsonpath='{.data.solr}' | base64 --decode

You can safely delete the bootstrap secret, provided you’ve captured the admin password, after your SolrCloud deploys with the bootstrapped security.json. However, this will trigger a rolling restart across all pods as the setup-zk initContainer definition changes.

k8s-oper user

The operator makes requests to secured Solr endpoints as the k8s-oper user; credentials for the k8s-oper user are stored in a separate secret of type kubernetes.io/basic-auth with name <CLOUD>-solrcloud-basic-auth. The k8s-oper user is configured with read-only access to a minimal set of endpoints, see details in the Authorization sub-section below. Remember, if you change the k8s-oper password using the Solr security API, then you must update the secret with the new password or the operator will be locked out. Also, changing the password for the k8s-oper user in the K8s secret after bootstrapping will not update Solr! You’re responsible for changing the password in both places.

Liveness and Readiness Probes

We recommend configuring Solr to allow un-authenticated access over HTTP to the probe endpoint(s) and the bootstrapped security.json does this for you automatically (see next sub-section). However, if you want to secure the probe endpoints, then you need to set probesRequireAuth: true as shown below:

spec:
  ...
  solrSecurity:
    authenticationType: Basic
    probesRequireAuth: true

When probesRequireAuth is set to true, the liveness and readiness probes execute a command instead of using HTTP. The operator configures a command instead of setting the Authorization header for the HTTP probes, as that would require a restart of all pods if the password changes. With a command, we can load the username and password from a secret; Kubernetes will update the mounted secret files when the secret changes automatically.

If you customize the HTTP path for any probes (under spec.customSolrKubeOptions.podOptions), then you must use probesRequireAuth=false as the operator does not reconfigure custom HTTP probes to use the command needed to support probesRequireAuth=true.

If you’re running Solr 8+, then we recommend using the /admin/info/health endpoint for your probes using the following config:

spec:
  ...
  customSolrKubeOptions:
    podOptions:
      livenessProbe:
        httpGet:
          scheme: HTTP
          path: /solr/admin/info/health
          port: 8983
      readinessProbe:
        httpGet:
          scheme: HTTP
          path: /solr/admin/info/health
          port: 8983

Consequently, the bootstrapped security.json will include an additional rule to allow access to the /admin/info/health endpoint:

      {
        "name": "k8s-probe-1",
        "role": null,
        "collection": null,
        "path": "/admin/info/health"
      }

Note, if you change the probes after creating your solrcloud, then the new probe paths will not be added to the security.json. The security file is bootstrapped just once, so if your probes need to change you must add it to the allowed paths via the Solr Security API using the admin credentials.

Authorization

The default security.json created by the operator during initialization is shown below; the passwords for each user are randomized for every SolrCloud you create. In addition to configuring the solr.BasicAuthPlugin, the operator initializes a set of authorization rules for the default user accounts: admin, solr, and k8s-oper. Take a moment to review these authorization rules so that you’re aware of the roles and access granted to each user in your cluster.

{
  "authentication": {
    "blockUnknown": false,
    "class": "solr.BasicAuthPlugin",
    "credentials": {
      "admin": "...",
      "k8s-oper": "...",
      "solr": "..."
    },
    "realm": "Solr Basic Auth",
    "forwardCredentials": false
  },
  "authorization": {
    "class": "solr.RuleBasedAuthorizationPlugin",
    "user-role": {
      "admin": [ "admin", "k8s" ],
      "k8s-oper": [ "k8s" ],
      "solr": [ "users", "k8s" ]
    },
    "permissions": [
      {
        "name": "k8s-probe-0",
        "role": null,
        "collection": null,
        "path": "/admin/info/system"
      },
      {
        "name": "k8s-status",
        "role": "k8s",
        "collection": null,
        "path": "/admin/collections"
      },
      {
        "name": "k8s-metrics",
        "role": "k8s",
        "collection": null,
        "path": "/admin/metrics"
      },
      {
         "name": "k8s-zk",
         "role":"k8s",
         "collection": null,
         "path":"/admin/zookeeper/status"
      },
      {
        "name": "k8s-ping",
        "role": "k8s",
        "collection": "*",
        "path": "/admin/ping"
      },
      {
        "name": "read",
        "role": [ "admin", "users" ]
      },
      {
        "name": "update",
        "role": [ "admin" ]
      },
      {
        "name": "security-read",
        "role": [ "admin" ]
      },
      {
        "name": "security-edit",
        "role": [ "admin" ]
      },
      {
        "name": "all",
        "role": [ "admin" ]
      }
    ]
  }
}

A few aspects of the default security.json configuration warrant a closer look. First, the probesRequireAuth setting (defaults to false) governs the value for blockUnknown (under authentication) and whether the probe endpoint(s) require authentication:

      {
        "name": "k8s-probe-0",
        "role": null,
        "collection": null,
        "path": "/admin/info/system"
      }

In this case, the "role":null indicates this endpoint allows anonymous access by unknown users. The "collection":null value indicates the path is not associated with any collection, i.e. it is a top-level system path.

The operator sends GET requests to the /admin/collections endpoint to get cluster status to determine the rolling restart order:

      {
        "name": "k8s-status",
        "role": "k8s",
        "collection": null,
        "path": "/admin/collections"
      },

In this case, the "role":"k8s" indicates the requesting user must be in the k8s role; notice that all default users have the k8s role.

The Prometheus exporter sends GET requests to the /admin/metrics endpoint to collect metrics from each pod. The exporter also hits the /admin/ping endpoint for every collection, which requires the following authorization rules:

      {
        "name": "k8s-metrics",
        "role": "k8s",
        "collection": null,
        "path": "/admin/metrics"
      },
      {
        "name": "k8s-ping",
        "role": "k8s",
        "collection": "*",
        "path": "/admin/ping"
      },

The "collection":"*" setting indicates this path applies to all collections, which maps to endpoint /collections/<COLL>/admin/ping at runtime.

The initial authorization config grants the read permission to the users role, which allows users to send query requests but cannot add / update / delete documents. For instance, the solr user is mapped to the users role, so the solr user can send query requests only. In general, please verify the initial authorization rules for each role before sharing user credentials.

Option 2: User-provided Basic Auth Secret

Alternatively, if users want full control over their cluster’s security config, then they can provide a kubernetes.io/basic-auth secret containing the credentials for the user they want the operator to make API requests as:

spec:
  ...
  solrSecurity:
    authenticationType: Basic
    basicAuthSecret: user-provided-secret

The supplied secret must be of type Basic Authentication Secret and define both a username and password. Here is an example of how to define a basic auth secret using YAML:

apiVersion: v1
kind: Secret
metadata:
  name: my-basic-auth-secret
type: kubernetes.io/basic-auth
stringData:
  username: k8s-oper
  password: Test1234

With this config, the operator will make API requests to secured Solr pods as the k8s-oper user.

be sure to use a stronger password for real deployments

If users supply their own basic auth secret, then the operator does not bootstrap the security.json; the reasoning is that if you’re supplying your own basic auth credentials then you’re also assuming the responsibility for configuring the desired access for this user.

Users need to ensure their security.json contains the user supplied in the basicAuthSecret with read access to:

/admin/info/system
/admin/info/health
/admin/collections
/admin/metrics
/admin/ping (for collection="*")
see the authorization rules defined by the default security.json as a guide for configuring access for the operator user

Changing the Password

If you change the password for the user configured in your basicAuthSecret using the Solr security API, then you must update the secret with the new password or the operator will be locked out. Also, changing the password for this user in the K8s secret will not update Solr! You’re responsible for changing the password in both places.

Prometheus Exporter with Basic Auth

If you enable basic auth for your SolrCloud cluster, then you need to point the Prometheus exporter at the basic auth secret; refer to Prometheus Exporter with Basic Auth for more details.