Encrypting Kubernetes Secrets with Sealed Secrets

We’ve talked about GitOps practices before, and we know that GitOps advocates managing all of your configuration through Git and versioning your environment configuration and infrastructure through declarative code.

In Kubernetes, we know that you can use resource manifest files to manage a cluster’s resource objects, but it’s not a good idea to store Kubernetes Secrets data in a Git repository, which is also very insecure.

Kubernetes Secrets are resource objects used to help us store sensitive information, such as passwords, keys, certificates, OAuth Token, SSH KEY, etc. Administrators can create Secrets objects, and then developers can very easily reference Secrets objects in resource manifest files without having to directly hardcode this sensitive information.

While this may seem convenient, the problem with Secrets is that they simply encode the sensitive information once in base64, and anyone can easily decrypt it to get the original data. So we say that Secrets manifest files can’t be stored directly in a Git repository, but if we had to create them manually every time, it would make our GitOps a lot less fluid.

Bitnami Labs has created an open source tool called Sealed Secrets to solve this problem.

Sealed Secrets

Sealed Secrets consists of two main components.

• One is the Kubernetes Operator within the cluster
• A client tool called kubeseal

kubeseal allows us to encrypt Kubernetes Secrets objects using asymmetric encryption algorithms, and SealedSecret is a Kubernetes CRD resource object containing an encrypted Secret that only the controller can decrypt, so it is very secure even if SealedSecret is stored in a public code repository.

When we create a SealedSecret resource object in a Kubernetes cluster, the corresponding Operator will read it and generate the corresponding Secret object, which we can then use directly in the Pod. The following is an example of a SealedSecret resource object.

1
2
3
4
5
6
7

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: mysecret
spec:
  encryptedData:
    secret: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq.....

When the Operator decrypts the above object, it generates the Secret object as follows.

1
2
3
4
5
6
7

apiVersion: v1
data:
  secret: VGhpcyBpcyBhIHNlY3JldCE=
kind: Secret
metadata:
  creationTimestamp: null
  name: mysecret

SealedSecret scope

Only the Operator can decrypt SealedSecret, and it is generally a better practice to not allow users to read Secret directly. We can prohibit low privilege users from reading Secret by creating RBAC rules, or we can restrict users to read Secret objects from their namespace only. Although SealedSecret is designed not to read them directly, users can still bypass this process and gain access to Secret objects that they are not allowed to see.

The SealedSecret resource provides several ways to prevent this behavior; it is namespace scoped by default, and once a SealedSecret is restricted to a namespace, the object cannot be used in other namespaces.

For example, if a Secret object named foo with the value bar is created under the web namespace, we cannot use this Secret in other namespaces, even if the same Secret is required. although SealedSecret’s controller does not use a separate decryption password for each namespace, it does Although SealedSecret’s controller does not use a separate decryption password for each namespace, it takes namespaces and names into account during encryption, so the effect is similar to each namespace having its own separate decryption key.

Another situation is that we may have a user on the web namespace above who can only view some Secrets but not all, which SealedSecret also allows. When we generate a SealedSecret object for a Secret named foo in the web namespace, a user on the web namespace who only has read access to the Secret object named bar cannot change the name of the Secret to bar in the SealedSecret resource object and use it to view the Secret.

Although these methods can help us prevent Secrets from being abused, they are still a bit tricky to manage. In the default configuration, there is no way to define a generic Secret to be used in multiple namespaces. And it’s likely that we have a very small team and the Kubernetes cluster is only accessed and maintained by operations staff, so we may not need this RBAC approach to permission control.

If we want to define SealedSecrets objects across namespaces, we can use scopes to achieve this functionality.

There are 3 scopes we can use to create SealedSecrets.

• strict (default): In this case, we need to consider the name and namespace of the Secret to encrypt it, and once we create the corresponding SealedSecret, we cannot change its name and namespace.
• namespace-wide: This scope allows us to rename the SealedSecret object within the namespace of the encrypted Secret.
• cluster-wide: This scope allows us to freely rename SealedSecret within the namespace of the encrypted Secret, allowing us to move the Secret to any namespace at will and name it at will.

We can use the --scope argument when using kubeseal to specify the scope.

1

$ kubeseal --scope cluster-wide --format yaml <secret.yaml >sealed-secret.yaml

You can also use annotation in Secret to use scopes before passing the configuration to kubeseal: * sealedsecrets.bitnami.com/namespace-wide: "true" means namespace-wide.

• sealedsecrets.bitnami.com/namespace-wide: "true" indicates namespace-wide
• sealedsecrets.bitnami.com/cluster-wide: "true" means cluster-wide

If no annotation is specified, then kubeseal uses the strict scope by default, and if two annotations are set, then the scope with the larger scope takes precedence.

SealedSecrets use

Installation

We mentioned earlier that SealedSecrets consists of a client-side kubeseal and a cluster-side Operator. Let’s start by installing the client-side kubeseal tool.

Select the latest release from the GitHub Release page and download the latest binaries at.

1
2
3
4

$ wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.12.4/kubeseal-linux-amd64 -O kubeseal
$ sudo install -m 755 kubeseal /usr/local/bin/kubeseal
$ kubeseal --version
kubeseal version: v0.12.4+dirty

Then install the Operator controller on the Kubernetes cluster side at

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

$ kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.12.4/controller.yaml
service/sealed-secrets-controller created
rolebinding.rbac.authorization.k8s.io/sealed-secrets-service-proxier created
rolebinding.rbac.authorization.k8s.io/sealed-secrets-controller created
clusterrole.rbac.authorization.k8s.io/secrets-unsealer created
serviceaccount/sealed-secrets-controller created
customresourcedefinition.apiextensions.k8s.io/sealedsecrets.bitnami.com created
role.rbac.authorization.k8s.io/sealed-secrets-service-proxier created
role.rbac.authorization.k8s.io/sealed-secrets-key-admin created
clusterrolebinding.rbac.authorization.k8s.io/sealed-secrets-controller created
deployment.apps/sealed-secrets-controller created

The controller will be installed under the kube-system namespace by default:

1
2
3

$ kubectl get pods -n kube-system -l name=sealed-secrets-controller
NAME                                         READY   STATUS    RESTARTS   AGE
sealed-secrets-controller-6bf8c44ff9-fqhgt   1/1     Running   0          3m36s

Once the controller is running successfully, the installation is proven to be successful. Next we can use SealedSecret to encrypt our Secret object.

Testing

In order to create the SealedSecret object, we first need to create a Secret file.

1
2
3
4
5
6
7
8
9

$ echo -n "This is a secret" | kubectl create secret generic mysecret --dry-run --from-file=secret=/dev/stdin -o yaml > secret.yaml
$ cat secret.yaml
apiVersion: v1
data:
  secret: VGhpcyBpcyBhIHNlY3JldA==
kind: Secret
metadata:
  creationTimestamp: null
  name: mysecret

Note that we used the -dry-run argument above, so it’s not really created, but the Secret object is just a base64-encoded string, so it’s not suitable for direct placement in the source repository.

Next, use the kubeseal tool to encrypt the object.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

$ kubeseal --format yaml <secret.yaml >sealedsecret.yaml
$ cat sealedsecret.yaml
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: mysecret
  namespace: default
spec:
  encryptedData:
    secret: AgActQcD5PS8BkGE9bgB0gCtJrL0HQBA7IWoKYFcQAuOBxxAL5r+i6shnJQByxyo4Wv5y62MrxXlKZkliXuHygrCaxZaKUyvuPkmAcpcgg/P5kY2KI2yQd0ENqGzgPCBiqbBiEFYATykeAUSEe3uUrYeE0EeIDWpiX758UzukaP30Z9nr5m1Ce+rvUUrIaVwQvlHH3pCENGcnb+iCOd2N1zO4YUua1GIm8TFB9IaINCyJR1Djv5zoiu9auNEeVrTWW9gqr1Wj9UaHA7uYqMpdUvupRAdUxBL5HSjZKtcesOKvVtxLNPBmIzolMf42FrxBH42WEoXHOPsRxuKw6UIdsiigVwnTEJYIZyQg/iIdcuWHfOUkm4YcxVdnAuXGxqu8mUhlVNfHjX4SR7MvC+dRPWQNoiL2+uxweHNl0rZCddbzM0ELYdtn1bktaoFLiNeq0bhYYIXhdIzIZypqruuP7ZoNg6zz7ySf7OxhsevSTAD6x1wwKCcjr2kWvNj+zSu1D3zcKT8LTNqHlk35cxbjMDaGPjZ4VdvUGS0d/fBuEtiK6js1vMCfrMdPLiQFNOibro3yKNE8ES3rASIOj3XBxD3FUVT9lGNLsCaRQDQlx/7Fqdjg4o/iAY0qVz8EET8rFWRG/GX3miZdgI9WyHTOY6oUd10VjdEvVPI6JTISI36/HUXOybP+Tc/j6B9FlGmt1CkfzANvXGojjZVjm0yzPtH
  template:
    metadata:
      creationTimestamp: null
      name: mysecret
      namespace: default

We can see that the Secret information has been encrypted, so we can now store it in the source repository with confidence.

Next, let’s use this Secret object in a sample Pod to see if we can get its data correctly.

Create the SealedSecret object directly from above.

1
2

$ kubectl apply -f sealedsecret.yaml
sealedsecret.bitnami.com/mysecret created

Once created, we can also find a Secret object named mysecret under the default namespace.

1
2
3

$ kubectl get secret mysecret
NAME       TYPE     DATA   AGE
mysecret   Opaque   1      7s

Use the resource list shown below to create a Pod for testing.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

$ kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  labels:
    app: busybox
spec:
  containers:
  - name: test
    image: busybox
    imagePullPolicy: IfNotPresent
    command:
      - sleep
      - "3600"
    volumeMounts:
    - name: mysecretvol
      mountPath: "/tmp/mysecret"
      readOnly: true
  volumes:
  - name: mysecretvol
    secret:
      secretName: mysecret
EOF
pod/busybox created
$ kubectl get pods busybox
NAME      READY   STATUS    RESTARTS   AGE
busybox   1/1     Running   0          19s

After a successful run we can look at the password data in the container to verify that it is correct:

1
2

$ kubectl exec -it busybox cat /tmp/mysecret/secret
This is a secret

You can see that the password message we defined This is a secret is printed correctly.

Modify namespace

Since there is no scope information specified in our Secret above, this Secret can only be used under the specified default (default) namespace. For example, here we modify the above SealedSecret object to a namespace named test.

1
2
3
4
5
6

$ cp -a sealedsecret.yaml sealedsecret-test.yaml
$ sed -i 's/default/test/g' sealedsecret-test.yaml
$ kubectl create ns test
namespace/test created
$ kubectl apply -f sealedsecret-test.yaml
sealedsecret.bitnami.com/mysecret created

Once created, we check to see if the corresponding Secret object has been generated.

1
2
3

$ kubectl get secret -n test
NAME                  TYPE                                  DATA   AGE
default-token-4gwfx   kubernetes.io/service-account-token   3      31s

We can see that the corresponding Secret object is not generated, check whether the SealedSecret object is created successfully.

1
2
3

$ kubectl get sealedsecret -n test
NAME       AGE
mysecret   104s

We can see that the SealedSecret object exists, but the corresponding Secret is not generated, so let’s check the controller’s logs.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

$ kubectl get pods -n kube-system -l name=sealed-secrets-controller
NAME                                         READY   STATUS    RESTARTS   AGE
sealed-secrets-controller-6bf8c44ff9-fqhgt   1/1     Running   0          28m
$ kubectl logs -f sealed-secrets-controller-6bf8c44ff9-fqhgt -n kube-system
......
2020/06/20 02:59:50 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"test", Name:"mysecret", UID:"78caa14b-c496-405b-91f8-652b2e4cef15", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"81242729", FieldPath:""}): type: 'Warning' reason: 'ErrUnsealFailed' Failed to unseal: no key could decrypt secret (secret)
2020/06/20 02:59:50 Updating test/mysecret
2020/06/20 02:59:50 Error updating test/mysecret, giving up: no key could decrypt secret (secret)
E0620 02:59:50.151844       1 controller.go:196] no key could decrypt secret (secret)
2020/06/20 02:59:50 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"test", Name:"mysecret", UID:"78caa14b-c496-405b-91f8-652b2e4cef15", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"81242729", FieldPath:""}): type: 'Warning' reason: 'ErrUnsealFailed' Failed to unseal: no key could decrypt secret (secret)

You can see the error message no key could decrypt secret (secret), this is because we use the default strict scope, so changing the namespace of SealedSecret is not effective.

Changing the Secret name

Let’s also test the effect of changing the name of the Secret under the Secret namespace.

1
2
3
4

$ cp -a sealedsecret.yaml sealedsecret-anothersecret.yaml
$ sed -i 's/mysecret/anothersecret/g' sealedsecret-anothersecret.yaml
$ kubectl apply -f sealedsecret-anothersecret.yaml
sealedsecret.bitnami.com/anothersecret created

Once created, check to see if the Secret generates a new name.

1
2
3
4

$ kubectl get secret
NAME                                    TYPE                                  DATA   AGE
default-token-5tsh4                     kubernetes.io/service-account-token   3      224d
mysecret                                Opaque                                1      18m

You can see that no modified Secret object named anothersecret has been generated either, and you can also view the Controller’s log message.

1
2
3
4
5

$ kubectl logs -f sealed-secrets-controller-6bf8c44ff9-fqhgt -n kube-system
......
2020/06/20 03:07:52 Updating default/anothersecret
2020/06/20 03:07:53 Error updating default/anothersecret, will retry: no key could decrypt secret (secret)
2020/06/20 03:07:53 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"default", Name:"anothersecret", UID:"b3467b24-b8d3-4c49-8de4-d8399875f8d5", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"81244607", FieldPath:""}): type: 'Warning' reason: 'ErrUnsealFailed' Failed to unseal: no key could decrypt secret (secret)

The error is basically the same as the one above for modifying the namespace, which is what we expected because we are creating a Secret in strict mode, so we can’t cross namespaces or modify names. We can create a cluster-wide Secret object to see what the effect is.

Creating Cluster-Wide SealedSecrets

Generate a list of resources for the Secret object using the command shown below.

1

$ echo -n "This is a secret" | kubectl create secret generic mycwsecret --dry-run --from-file=secret=/dev/stdin -o yaml > secret-cw.yaml

Then use the cluster-wide scope to encrypt the Secret object under the test namespace.

1
2
3
4
5
6
7

$ kubeseal --format yaml --scope cluster-wide <secret-cw.yaml >sealedsecret-cw.yaml
$ kubectl apply -n test -f sealedsecret-cw.yaml
sealedsecret.bitnami.com/mycwsecret created
$ kubectl get secret -n test
NAME                  TYPE                                  DATA   AGE
default-token-4gwfx   kubernetes.io/service-account-token   3      16m
mycwsecret            Opaque                                1      30s

We can see that the corresponding Secret object has been created successfully.

Now let’s rename the Secret and see if it still works after re-creation.

1
2
3
4

$ cp -a sealedsecret-cw.yaml sealedsecret-anothercwsecret.yaml
$ sed -i 's/mycwsecret/anothercwsecret/g' sealedsecret-anothercwsecret.yaml
$ kubectl apply -n test -f sealedsecret-anothercwsecret.yaml
sealedsecret.bitnami.com/anothercwsecret created

Once created, we can see that the Secret behind the rename is also automatically generated:

1
2
3

$ kubectl get secret anothercwsecret -n test
NAME              TYPE     DATA   AGE
anothercwsecret   Opaque   1      36s

What about modifying the namespace? For example, if we create the above encrypted Secret object under the default namespace.

1
2

$ kubectl apply -n default -f sealedsecret-cw.yaml
sealedsecret.bitnami.com/mycwsecret created

After successful creation, we can also see that the corresponding Secret object is created under the default namespace.

1
2
3

$ kubectl get secret mycwsecret
NAME         TYPE     DATA   AGE
mycwsecret   Opaque   1      17s

This is also as expected, since we are creating SealedSecrets with a cluster-wide scope, so we can modify the namespace and name as we wish.

More information on the use of SealedSecrets can be found in the official documentation for more information.