Security is a top priority for cloud-native applications, and while security is a very broad topic, Linkerd still has an important role to play: its two-way TLS (mTLS) feature is designed to enable a zero-trust approach to security in Kubernetes. Zero-trust security is an IT security model that requires strict authentication for every person and every device that attempts to access resources on a private network, whether located inside or outside the network boundary.

mTLS

What is mTLS

An increasingly common approach to communications security in cloud environments is the Zero Trust approach, and while a comprehensive treatment of zero trust security is beyond the scope of this section, the core goal is to reduce the application’s security perimeter to the smallest possible level. For example, instead of putting firewalls around the data center to secure incoming traffic, leaving a “soft interior” without further authentication, each application in the data center can enforce security at its own boundary. This zero-trust approach is a natural fit for cloud environments where the underlying hardware and network infrastructure is not under your control.

The Linkerd security model enables zero-trust security by providing transparent, two-way TLS communication between services. Two-way TLS (mTLS) is a form of transport security that provides confidentiality and authentication of communications. In other words, not only are communications encrypted, but identities are also authenticated on both ends of the connection (the bidirectional component of mTLS differs from the TLS used by browsers in that it only authenticates the server side of the connection. mTLS authenticates both the client and the server). Linkerd’s goal is to authenticate and encryption, allowing you to adopt a zero-trust approach to Kubernetes clusters.

Authentication is especially important, and while most people think the value of TLS is in encryption, it is equally important to verify the identity of the entities on both sides of the connection. After all, encryption is only useful if you can trust that the entity on the other side of the communication with you is who they say they are - an encrypted message from a bad actor is still a message from a bad actor.

In Linkerd, identities authenticated by mTLS are associated with Kubernetes ServiceAccounts. This means that Linkerd’s mTLS identity system uses the exact same model that Kubernetes uses to establish identity and access control for workloads on a cluster, rather than inventing a new framework.

mTLS with Linkerd

One of Linkerd’s design principles is that complexity is the enemy of security. The harder it is to configure something, the less likely you are to use it; the more options and settings you have, the more likely you are to accidentally configure it in an insecure way.

By default, Linkerd enables mTLS for all pod-to-pod communication in the grid, and as long as both sides inject a data plane proxy, congratulations: you’ve already authenticated encrypted mTLS between services. but we didn’t realize it.

There are a few things to keep in mind about Linkerd’s ability to automatically add mTLS.

  • Both endpoints must be in the grid. linkerd needs to handle both client-side and server-side connections to work its mTLS magic.

  • In Linkerd 2.8.1 and earlier, Linkerd could only add mTLS to HTTP and gRPC traffic, and even then it could not perform this operation for certain types of permissions, hosts, or Headers; these restrictions have been removed in Linkerd 2.9, which adds mTLS to all TCP traffic, regardless of the protocol.

  • Connections where the client initiates TLS cannot be mTLSed by Linkerd. instead, Linkerd will treat these connections as TCP traffic. Note that this also means that Linkerd can only provide TCP-level metrics for these connections.

Next let’s understand how mTLS works and how to verify that our connection does indeed have mTLS.

Linkerd Identity component

The Identity component of the Linkerd control plane was discussed earlier. It plays the role of a CA or Certificate Authority. A Certificate Authority is the entity that issues digital certificates and makes the Identity component the issuer of digital certificates.

Linkerd uses the same “type” of certificate as the TLS certificate used by the website to authenticate its identity. Unlike websites, these certificates are not validated by third-party entities such as Verisign, as they do not require authentication, and they are only used by Linkerd agents within a cluster.

Linkerd’s CA (Identity service) is deployed to the cluster as part of the Linkerd control plane. During this deployment, the Linkerd CLI generates a certificate and stores it in a Kubernetes Secret named linkerd-identity-token-XXXXX in the Linkerd namespace.

1
2
3
4
5

$ kubectl get secret -n linkerd
NAME                                 TYPE                                  DATA   AGE
linkerd-identity-issuer              Opaque                                2      11d
linkerd-identity-token-nqhbk         kubernetes.io/service-account-token   3      11d
# ......

By looking at the Secret, we can see a Secret object prefixed with linkerd-identity-token-, which we can export for viewing.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

$ kubectl get secret -n linkerd linkerd-identity-token-nqhbk -o yaml
apiVersion: v1
data:
  ca.crt: <ca.crt>
  namespace: bGlua2VyZA==
  token: <token>
kind: Secret
metadata:
  annotations:
    kubernetes.io/service-account.name: linkerd-identity
    kubernetes.io/service-account.uid: cdc3d8fd-0e02-4b17-88ce-d2cc0a9e4907
  name: linkerd-identity-token-nqhbk
  namespace: linkerd
type: kubernetes.io/service-account-token

The field named ca.crt in the data section of the output is the UTF-8 encoded root certificate generated during the Linkerd installation. This certificate is called the “trust anchor” because it is used as the basis for all certificates issued to the proxy.

The trust anchor is also used to create another certificate and key pair at installation time: the issuer credentials, which are stored in a separate Kubernetes Secret called linkerd-identity-issuer. The issuer credentials are used to issue certificates to the Linkerd proxy, and we can also view the data for that Secret.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

$ kubectl get secret -n linkerd linkerd-identity-issuer -o yaml
apiVersion: v1
data:
  crt.pem: <crt.pem>
  key.pem: <key.pem>
kind: Secret
metadata:
  labels:
    linkerd.io/control-plane-component: identity
    linkerd.io/control-plane-ns: linkerd
  name: linkerd-identity-issuer
  namespace: linkerd
type: Opaque

Next we will learn how to use these keys to issue certificates to the proxy to enable mTLS.

How the Linkerd proxy obtains a certificate

First, when a Pod is injected into the Linkerd agent, the agent sends a certificate signing request (CSR) to Linkerd’s identity service. The identity service issues a signed certificate to the agent using the issuer credentials (the scope of the CSR is the Kubernetes ServiceAccount where the Pod is running, so the generated certificate is associated with that ServiceAccount), and the certificate expires after 24 hours. Before the certificate expires, the agent sends a new certificate signing request to the identity service to obtain a new certificate; this process continues throughout the life of the Linkerd agent and is called Certificate Rotation, an automated way to minimize the damage caused by compromised certificates: in the worst case, any compromised certificate can only be used for 24 hours.

The -linkerd check command has a simple way to ensure that the proxies all have certificates issued by the identity service, and we can check the status of the proxies ourselves by passing the --proxy flag.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

$ linkerd check --proxy
# ......
linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor

linkerd-identity-data-plane
---------------------------
√ data plane proxies certificate match CA

# ......

Status check results are √

The above output includes a linkerd-identity-data-plane section to indicate whether the proxy is using a certificate issued by a trust anchor.

1
2
3

linkerd-identity-data-plane
---------------------------
√ data plane proxies certificate match CA

We can also enable debug mode in the Linkerd agent and identity service to see the agent send the CSR to the identity service and retrieve the certificate.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

$ kubectl get deploy -n linkerd
NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
linkerd-destination      1/1     1            1           11d
linkerd-identity         1/1     1            1           11d
linkerd-proxy-injector   1/1     1            1           11d
$ kubectl edit deploy linkerd-identity -n linkerd
# ......
  spec:
    containers:
    - args:
      - identity
      - -log-level=debug  # debug 
      - -log-format=plain
      - -controller-namespace=linkerd
      - -identity-trust-domain=cluster.local
      - -identity-issuance-lifetime=24h0m0s
      - -identity-clock-skew-allowance=20s
      - -identity-scheme=linkerd.io/tls
# ......

When the identity service is re-updated, we can observe the log information of the corresponding Pod.

1

$ kubectl logs -f -n linkerd deploy/linkerd-identity -c identity

The above command outputs a lot of logs to the console. In the Linkerd identity log output, we can see the output related to Request Body and Response Body, which includes a long UTF-8 encoded data, i.e. CSR and the certificate issued to the proxy.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# ......
time="2022-08-30T07:39:17Z" level=debug msg="Issuer has been updated"
time="2022-08-30T07:39:17Z" level=info msg="starting admin server on :9990"
time="2022-08-30T07:39:17Z" level=info msg="starting gRPC server on :8080"
time="2022-08-30T07:39:17Z" level=debug msg="Validating token for linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local"
I0830 07:39:17.291787       1 request.go:1123] Request Body: {"kind":"TokenReview","apiVersion":"authentication.k8s.io/v1","metadata":{"creationTimestamp":null},"spec":{"token":"<......>"},"status":{"user":{}}}
# ......
I0830 07:39:17.291969       1 round_trippers.go:435] curl -k -v -XPOST  -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "User-Agent: controller/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer <masked>" 'https://10.96.0.1:443/apis/authentication.k8s.io/v1/tokenreviews'
I0830 07:39:17.293883       1 round_trippers.go:454] POST https://10.96.0.1:443/apis/authentication.k8s.io/v1/tokenreviews 201 Created in 1 milliseconds
# ......
I0830 07:39:17.294241       1 request.go:1123] Response Body: {"kind":"TokenReview","apiVersion":"authentication.k8s.io/v1","metadata":{"creationTimestamp":null,"managedFields":[{"manager":"controller","operation":"Update","apiVersion":"authentication.k8s.io/v1","time":"2022-08-30T07:39:17Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:token":{}}}}]},"spec":{"token":"<token>"},"status":{"authenticated":true,"user":{"username":"system:serviceaccount:linkerd:linkerd-identity","uid":"cdc3d8fd-0e02-4b17-88ce-d2cc0a9e4907","groups":["system:serviceaccounts","system:serviceaccounts:linkerd","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["linkerd-identity-5d9b874d66-m77ps"],"authentication.kubernetes.io/pod-uid":["2f147d37-1616-487a-b7aa-1805b84c026a"]}},"audiences":["https://kubernetes.default.svc.cluster.local"]}}
# ......

Next, let’s take a look at the security between the Emojivoto application services. First, we can use the linkerd viz edges command to see how Pods are connected to each other.

1
2
3
4
5
6
7
8
9

$ linkerd viz edges po -n emojivoto
SRC                           DST                         SRC_NS        DST_NS      SECURED
vote-bot-6d7677bb68-jvxsg     web-5f86686c4d-58p7k        emojivoto     emojivoto   √
web-5f86686c4d-58p7k          emoji-696d9d8f95-5vn9w      emojivoto     emojivoto   √
web-5f86686c4d-58p7k          voting-ff4c54b8d-xhjv7      emojivoto     emojivoto   √
prometheus-7bbc4d8c5b-5rc8r   emoji-696d9d8f95-5vn9w      linkerd-viz   emojivoto   √
prometheus-7bbc4d8c5b-5rc8r   vote-bot-6d7677bb68-jvxsg   linkerd-viz   emojivoto   √
prometheus-7bbc4d8c5b-5rc8r   voting-ff4c54b8d-xhjv7      linkerd-viz   emojivoto   √
prometheus-7bbc4d8c5b-5rc8r   web-5f86686c4d-58p7k        linkerd-viz   emojivoto   √

We can see that the output above contains a column SECURED at the end, indicating whether it is a secure connection or not, and the following values are all √, indicating a secure connection.

We then use the linkerd viz tap command again to capture the live traffic, and the output also contains a tag value of tls=true, as shown below.

1
2
3
4
5

$ linkerd viz tap deploy web -n emojivoto
req id=0:0 proxy=in  src=10.244.1.165:47130 dst=10.244.1.176:8080 tls=true :method=GET :authority=web-svc.emojivoto:80 :path=/api/list
req id=0:1 proxy=out src=10.244.1.176:42096 dst=10.244.1.188:8080 tls=true :method=POST :authority=emoji-svc.emojivoto:8080 :path=/emojivoto.v1.EmojiService/ListAll
rsp id=0:1 proxy=out src=10.244.1.176:42096 dst=10.244.1.188:8080 tls=true :status=200 latency=2731µs
# ......

At this point we understand how Linkerd’s Identity component issues certificates to Linkerd proxies in the data plane, and how Linkerd’s mTLS implementation in the proxy uses these certificates to encrypt communication and authenticate both parties.

Automatic Rotation of Controller Plane TLS Credentials

Linkerd’s automatic mTLS feature uses a set of TLS credentials to generate TLS certificates for the agent: a trust anchor, issuer certificate, and private key. While Linkerd automatically rotates TLS certificates for data plane agents every 24 hours, it does not rotate the TLS credentials used to issue these certificates. Next, let’s learn how to use Cert-manager to automatically rotate issuer certificates and private keys.

Cert-manager

Cert-manager is a very popular cloud-native certificate management tool. Cert-manager adds certificates and certificate authorities to Kubernetes clusters as CRD resource types, simplifying the process of obtaining, renewing, and using these certificates. It can issue certificates from a variety of supported sources, including Let's Encrypt, HashiCorp Vault, and Venafi, as well as private PKI’s. It will ensure that certificates are valid and up-to-date, and attempt to update them at the configured time before they expire.

Cert-manager

The installation of Cert-manager is also very simple and can be done in one click directly using the official resource list file provided as follows.

1

$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.yaml

By default, related resources are installed into a namespace named cert-manager.

1
2
3
4
5

$ kubectl get pods -n cert-manager
NAME                                      READY   STATUS    RESTARTS   AGE
cert-manager-55649d64b4-kdcfk             1/1     Running   0          43s
cert-manager-cainjector-666db4777-cp2dn   1/1     Running   0          43s
cert-manager-webhook-6466bc8f4-5lvw5      1/1     Running   0          43s

Issuing a certificate

Next we use the step tool to create a signed key pair and store it in a Secret object in Kubernetes.

1
2
3
4

$ step certificate create root.linkerd.cluster.local ca.crt ca.key \
  --profile root-ca --no-password --insecure
Your certificate has been saved in ca.crt.
Your private key has been saved in ca.key.

Then save the generated ca.crt and ca.key to the Secret object.

1
2

$ kubectl create secret tls linkerd-trust-anchor --cert=ca.crt --key=ca.key -n
secret/linkerd-trust-anchor created

With Secret, we can create an Issuer resource that references the key’s certificate authority.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

$ cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: linkerd-trust-anchor
  namespace: linkerd
spec:
  ca:
    secretName: linkerd-trust-anchor
EOF
$ kubectl get issuer -n linkerd
NAME                   READY   AGE
linkerd-trust-anchor   True    84s

Then we can issue certificates and write them to a Secret object. Finally, we can create a cert-manager “Certificate” resource that uses this Issuer to generate the required certificates.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

$ cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: linkerd-identity-issuer
  namespace: linkerd
spec:
  secretName: linkerd-identity-issuer
  duration: 48h
  renewBefore: 25h
  issuerRef:
    name: linkerd-trust-anchor
    kind: Issuer
  commonName: identity.linkerd.cluster.local
  dnsNames:
  - identity.linkerd.cluster.local
  isCA: true
  privateKey:
    algorithm: ECDSA
  usages:
  - cert sign
  - crl sign
  - server auth
  - client auth
EOF
$ kubectl get certificate -n linkerd
NAME                      READY   SECRET                    AGE
linkerd-identity-issuer   True    linkerd-identity-issuer   17s

In the resource manifest file above, duration instructs cert-manager to treat the certificate as valid for 48 hours, while renewBefore instructs cert-manager to attempt to issue a new certificate 25 hours before the current certificate expires.

At this point, cert-manager can now use this certificate resource to obtain TLS credentials, which will be stored in a Secret named linkerd-identity-issuer. To verify your newly issued certificate, we can run the following command.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

$ kubectl get secret linkerd-identity-issuer -o yaml -n linkerd
apiVersion: v1
data:
  ca.crt: <ca.crt>
  crt.pem: <crt.pem>
  key.pem: <key.pem>
  tls.crt: <tls.crt>
  tls.key: <tls.key>
kind: Secret
metadata:
  name: linkerd-identity-issuer
  namespace: linkerd
type: Opaque

Now we just need to inform Linkerd to use these credentials. Since we are installing via the linkerd command line tool, the Linkerd control plane will by default be installed via the --identity-external-issuer flag, which instructs Linkerd to read the credentials from the linkerd-identity-issuer Secret. Whenever the certificate and key stored in the Secret are updated, the identity service will automatically detect this change and reload the new credentials.

This sets up automatic rotation of Linkerd control plane TLS credentials, and if you want to monitor the update process, you can check the IssuerUpdated event issued by the service.

1
2
3
4

$ kubectl get events --field-selector reason=IssuerUpdated -n linkerd

LAST SEEN   TYPE     REASON          OBJECT                        MESSAGE
2m37s       Normal   IssuerUpdated   deployment/linkerd-identity   Updated identity issuer

You can see that the Updated identity issuer has been executed.