K8s provides a Secret resource to store and set sensitive information such as API endpoint addresses, various user passwords or tokens, and so on. When you are not using K8s, this information may be set at deployment time through a configuration file or environment variable.

However, Secret is not really secure, as anyone who has looked at Secret with kubectl knows, we can easily see the original text of Secret, as long as we have the relevant permissions, even though its contents are base64 encoded, which is basically equivalent to plaintext.

Therefore, K8s native Secret is very simple, not particularly suitable for direct use in large companies, and is a challenge for RBAC, as many people who should not see plaintext information may be able to see it.

This is especially true now that many companies have adopted the so-called GitOps philosophy, where many things need to be put into a VCS, such as git, and this problem becomes more pronounced because the VCS also needs to set the necessary permissions.

Problems

In short, there are probably several places where the Secret contents can be made available to people who should not see them: the

  • etcd storage
  • via the API server
  • Directly viewing the file on the node

Let’s take this example to see how Secret is used in K8s.

Secret definition, username and password are admin and hello-secret respectively.

1
2
3
4
5
6
7
8

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
data:
  username: YWRtaW4=
  password: aGVsbG8tc2VjcmV0Cg==

Pod definition, here we mount Secret as volume to the container.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - name: mypod
    image: docker.io/containerstack/alpine-stress
    command:
      - top
    volumeMounts:
    - name: foo
      mountPath: "/etc/foo"
      readOnly: true
  volumes:
  - name: foo
    secret:
      secretName: mysecret

After the pod is started, we can go to the container to see the contents of the files after Secret is mounted to the container as a volume.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

$ kubectl exec -it mypod sh
/ # cd /etc/foo/
/etc/foo # ls -tal
total 4
drwxr-xr-x    1 root     root          4096 Apr 14 08:55 ..
drwxrwxrwt    3 root     root           120 Apr 14 08:55 .
drwxr-xr-x    2 root     root            80 Apr 14 08:55 ..2021_04_14_08_55_54.401661151
lrwxrwxrwx    1 root     root            31 Apr 14 08:55 ..data -> ..2021_04_14_08_55_54.401661151
lrwxrwxrwx    1 root     root            15 Apr 14 08:55 password -> ..data/password
lrwxrwxrwx    1 root     root            15 Apr 14 08:55 username -> ..data/username
/etc/foo # ls -tal ..2021_04_14_08_55_54.401661151
total 8
drwxr-xr-x    2 root     root            80 Apr 14 08:55 .
drwxrwxrwt    3 root     root           120 Apr 14 08:55 ..
-rw-r--r--    1 root     root            13 Apr 14 08:55 password
-rw-r--r--    1 root     root             5 Apr 14 08:55 username
/etc/foo # cat password
hello-secret
/etc/foo #

etcd storage

The resources in the API server are stored in etcd, and we can see the contents directly from the file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# hexdump -C /var/lib/etcd/member/snap/db | grep -A 5 -B 5 hello
00043640  12 00 1a 07 64 65 66 61  75 6c 74 22 00 2a 24 32  |....default".*$2|
00043650  35 66 37 35 38 30 38 2d  37 33 31 33 2d 34 38 64  |5f75808-7313-48d|
00043660  39 2d 39 61 38 65 2d 38  61 35 66 66 32 32 63 64  |9-9a8e-8a5ff22cd|
00043670  64 35 39 32 00 38 00 42  08 08 98 dc da 83 06 10  |d592.8.B........|
00043680  00 7a 00 12 19 0a 08 70  61 73 73 77 6f 72 64 12  |.z.....password.|
00043690  0d 68 65 6c 6c 6f 2d 73  65 63 72 65 74 0a 12 11  |.hello-secret...|
000436a0  0a 08 75 73 65 72 6e 61  6d 65 12 05 61 64 6d 69  |..username..admi|
000436b0  6e 1a 06 4f 70 61 71 75  65 1a 00 22 00 00 00 00  |n..Opaque.."....|
000436c0  00 00 00 08 95 5f 00 00  00 00 00 00 00 00 0a 37  |....._.........7|
000436d0  2f 72 65 67 69 73 74 72  79 2f 73 65 72 76 69 63  |/registry/servic|
000436e0  65 73 2f 65 6e 64 70 6f  69 6e 74 73 2f 6b 75 62  |es/endpoints/kub|

As you can see, the contents of the basic yaml are stored in plaintext, and are decoded in base64.

A similar result can be obtained with the following command.

1

$ ETCDCTL_API=3 etcdctl get --prefix /registry/secrets/default/mysecret | hexdump -C

etcd originally stored plaintext data, but it seems that encrypted storage has been supported since 1.7. And direct access to etcd is not that easy physically.

API server

The API server is much simpler, as long as you have access to the API server from any node, you can get the plaintext contents of the secret.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

$ kubectl get secret mysecret -o yaml

apiVersion: v1
data:
  password: aGVsbG8tc2VjcmV0Cg==
  username: YWRtaW4=
kind: Secret
metadata:
  creationTimestamp: "2021-04-14T08:55:52Z"
  name: mysecret
  namespace: default
  resourceVersion: "2196"
  selfLink: /api/v1/namespaces/default/secrets/mysecret
  uid: 25f75808-7313-48d9-9a8e-8a5ff22cdd59
type: Opaque

On the node

You can also see the contents of the Secret file on the node.

To find the mount point of the foo volume.

1
2

# mount | grep foo
tmpfs on /var/lib/kubelet/pods/280451e8-512b-489c-b5dd-df2b1a3c9b29/volumes/kubernetes.io~secret/foo type tmpfs (rw,relatime)

View the contents of the file under this volume.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# ls -tal /var/lib/kubelet/pods/280451e8-512b-489c-b5dd-df2b1a3c9b29/volumes/kubernetes.io~secret/foo
total 4
drwxrwxrwt 3 root root  120 4月  14 16:55 .
drwxr-xr-x 2 root root   80 4月  14 16:55 ..2021_04_14_08_55_54.401661151
lrwxrwxrwx 1 root root   31 4月  14 16:55 ..data -> ..2021_04_14_08_55_54.401661151
lrwxrwxrwx 1 root root   15 4月  14 16:55 password -> ..data/password
lrwxrwxrwx 1 root root   15 4月  14 16:55 username -> ..data/username
drwxr-xr-x 4 root root 4096 4月  14 16:55 ..

# cat /var/lib/kubelet/pods/280451e8-512b-489c-b5dd-df2b1a3c9b29/volumes/kubernetes.io~secret/foo/password
hello-secret

Third-party solutions

For the points mentioned above that could compromise the Secret, it is easy to think of several solutions.

  • etcd encryption
  • Strict permission design for API server
  • Strengthen node node user rights management and system security

However, to ensure the absolute security of the Secret, all of the above solutions are necessary, and one of them is indispensable.

The community and public cloud providers have a number of products and solutions that we can refer to.

shyiko/kubesec

kubesec only encrypts/decrypts data in Secret and supports the following key management services or software.

  • AWS Key Management Service
  • Google Cloud KMS
  • GnuPG

bitnami-labs/sealed-secrets

Bitnami is also a well known company in the K8s space, exporting many technologies and best practices.

Bitnami

SealeSecret encrypts and stores the entire secret resource as a SealedSecret resource, and decryption can only be done by the controller in the cluster.

SealeSecret provides a kubeseal tool to encrypt the secret resource, which requires a public key, which is obtained from the SealeSecret controller.

However, just from the documentation, the key that SealeSecret controller relies on for encryption and decryption is also saved by a common Secret, isn’t that a problem? It also increases the operation and maintenance cost of SealeSecret controller.

mozilla/sops

Technically speaking, sops is not necessarily related to K8s, it is just an encrypted file editor supporting YAML/JSON/ENV/INI and other file formats, it supports AWS KMS, GCP KMS, Azure Key Vault, age, and PGP and other services and applications.

If interested, you can see its home page.

Kubernetes External Secrets

Kubernetes External Secrets is an open source software developed by godaddy, a well-known domain name service provider, which can pass confidential information stored in external KMS directly to K8s. Currently supported KSMs include.

  • AWS Secrets Manager
  • AWS System Manager
  • Hashicorp Vault
  • Azure Key Vault
  • GCP Secret Manager
  • Alibaba Cloud KMS Secret Manager

It is implemented through a custom controller and CRD with the following architecture diagram.

k8s

Specifically the user needs to create a resource of type ExternalSecret to map the external KMS data to the K8s Secret.

However, there are probably only two advantages to this approach.

  • Unified key management, or using existing key assets
  • key information does not want to be placed on VCS, etc.

For preventing the leakage of Sercet information, it is not very useful because the plaintext resources can still be seen on the API server/etcd.

Or, what External Secrets really does is to map the keys from external KMS to Secret resources in K8s, which is of little use to ensure the security of data in K8s clusters.

Kamus

Kamus also provides a way to encrypt the key (a command line tool) and decrypt it only through the controller in K8s. However, the Secret stored in K8s is in an encrypted state, and the user cannot directly access the plaintext content of the Secret, as with External Secrets.

Kamus consists of 3 components, which are

  • Encrypt API
  • Decrypt API
  • Key Management System (KMS)

KMS is a wrapper for external encryption services and currently supports the following services: - AES - AWS KMS - Azure KeyVault - Google Cloud KMS

Kamus encrypts the secret with a service account, and the Pod requests Kamus’ decryption service through the service account to decrypt the secret.

For K8s, decrypting the secret can be done by the init container: define a memory-based emptyDir and use the same volume for the service container and the init container, and store the data under the volume after the init container is decrypted, and then the service container can use the decrypted secret data.

k8s

Vault by HashiCorp

HashiCorp is one of the top companies in the cloud/DevOps space.

Vault itself is a KMS-like service for managing confidential data. For K8s’ native secret, support is provided in roughly two ways.

Agent Sidecar Injector

This approach is similar to Kamus above, and also requires two components.

  • Mutation webhook: responsible for modifying pod definition and injecting init/sidecar
  • agent-sidecar: responsible for fetching and decrypting data and saving it to the specified volume/path

Vault agent sidecar injector not only provides init container to initialize the secret, but also updates the secret periodically via sidecar, which is very close to the native secret implementation.

The application only needs to read the specified file on the filesystem and does not have to worry about how to get the encrypted information from outside.

Here is an example from the official blog.

Pod information.

1
2
3
4
5
6
7

spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-secret-helloworld: "secrets/helloworld"
        vault.hashicorp.com/role: "myapp"

In this definition, vault-k8s injects vault agent into the pod and initializes it with secrets/helloworld. After the pod is run, a file named helloworld can be found under /vault/secrets.

1
2
3

$ kubectl exec -ti app-XXXXXXXXX -c app -- cat /vault/secrets/helloworld
data: map[password:foobarbazpass username:foobaruser]
metadata: map[created_time:2019-12-16T01:01:58.869828167Z deletion_time: destroyed:false version:1]

Of course this data is raw data and is not formatted. If you want to specify the format of the output to the file, you can use vault’s template function.

Vault CSI Provider

This section can be found in the Community Solutions section below.

Community solutions

Of course, there is no reason why the community can’t be aware of the problems with native secrets, so the community also has the Kubernetes Secrets Store CSI Driver, a solution for integrating secrets into K8s via the CSI interface to K8s.

The Secrets Store CSI driver (secrets-store.csi.k8s.io) allows K8s to mount multiple secrets as a volume from an external KMS into the Pod.

To use the Secrets Store CSI Driver, the general process is as follows:

  • Define SecretProviderClass

    1
2
3
4
5
6
7

    apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
name: my-provider
spec:
provider: vault   # accepted provider options: azure or vault or gcp
parameters:       # provider-specific parameters

  • Configuring Volume for Pods

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

    kind: Pod
apiVersion: v1
metadata:
name: secrets-store-inline
spec:
containers:
- image: k8s.gcr.io/e2e-test-images/busybox:1.29
    name: busybox
    command:
    - "/bin/sleep"
    - "10000"
    volumeMounts:
    - name: secrets-store-inline
    mountPath: "/mnt/secrets-store"
    readOnly: true
volumes:
    - name: secrets-store-inline
    csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
        secretProviderClass: "my-provider"

Once the Pod is started, you can confirm the decrypted data.

1
2

$ kubectl exec secrets-store-inline -- ls /mnt/secrets-store/
foo

Summary

The above summary is based on publicly available information on the Internet, and has not been personally experienced, so some parts may be misunderstood, and you need to confirm the best for a deeper understanding by yourself.

But overall, the community solution is probably the easiest and not very troublesome to deploy, except that it has little to do with the native secret.

Vault solution is also very mature, worthy of attention.