Managing admission webhooks certificates with cert-manager

Kubernetes provides ways to extend its built-in functionality, probably most commonly with custom resource types and custom controllers, but in addition, Kubernetes has some other very interesting features. For example, admission webhooks can be used to extend the API for modifying the basic behaviour of certain Kubernetes resources.

Admission Controllers are snippets of code used to intercept requests to the Kubernetes API Server before the object is persisted, and to let them through after they have been authenticated and authorized. Mutating controllers may modify the resource objects they process, Validating controllers do not. If any controller in either phase rejects the request, the entire request is immediately rejected and the error is returned to the end user.

This means that there are special controllers that can intercept Kubernetes API requests and modify or reject them according to custom logic.

Kubernetes has a list of controllers that it has implemented itself.

https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#what-does-each-admission-controller-do

Of course you can also write your own controllers, although these sound more powerful, they need to be compiled into kube-apiserver and can only be started when apiserver is started.

You can also use the kube-apiserver startup parameters directly to see what controllers are supported built-in.

`1`	`kube-apiserver --help \|grep enable-admission-plugins`

Due to the limitations of the controller above, we need to use the concept of dynamic rather than coupling it with the apiserver, and Admission webhooks solves this limitation with a dynamic configuration method.

What is an admission webhook?

There are two special Admission Controllers in the Kubernetes apiserver: MutatingAdmissionWebhook and ValidatingAdmissionWebhook, which will send an admission request to an external HTTP callback service and receive an admission response. If these two Admission Controllers are enabled, the Kubernetes administrator can create and configure an admission webhook in the cluster.

admission webhook

The overall steps are shown below.

Check that the admission webhook controller is enabled in the cluster and configure it as required.
Write an HTTP callback for handling the admission request, which can be a simple HTTP service deployed in the cluster, or even a serverless function.
Configure the admission webhook with the MutatingWebhookConfiguration and ValidatingWebhookConfiguration resources.

The difference between these two types of admission webhooks is very clear: validating webhooks can reject requests, but they cannot modify the objects fetched in an admission request, whereas mutating webhooks can modify objects before returning an admission response by creating patch to modify the object before returning the admission response, and if the webhook rejects a request, an error will be returned to the end user.

The very hot Service Mesh application istio, which automatically injects the sidecar container Envoy into the Pod by mutating webhooks.

https://istio.io/latest/docs/setup/additional-setup/sidecar-injection/

Creating and Configuring an Admission Webhook

We’ve covered the theory of the Admission Webhook, so let’s test it out in a real Kubernetes cluster. We will create a webhook webserver, deploy it to the cluster, and then create a webhook configuration to see if it works.

First make sure that MutatingAdmissionWebhook and ValidatingAdmissionWebhook are enabled in apiserver with the parameter --enable-admission-plugins, which is already enabled by default in v1.25+. If not, you need to add these two parameters and restart apiserver.

Then check if the Admission registration API is enabled in the cluster by running the following command.

1
2

☸ ➜ kubectl api-versions |grep admission
admissionregistration.k8s.io/v1

Once these prerequisites are met, we can write the admission webhook server, which is actually a webserver developed to handle an AdmissionReview request sent by the APIServer, in the following format.

{
  "apiVersion": "admission.k8s.io/v1",
  "kind": "AdmissionReview",
  ...
  "request": {
    # Random uid uniquely identifying this admission call
    "uid": "d595e125-9489-4d1c-877d-0a05984355c8",
    # object is the new object being admitted.
    "object": {"apiVersion":"v1","kind":"Pod", ...},
    ...
  }
}

Then just construct an AdmissionReview object and write it back.

{
  "apiVersion": "admission.k8s.io/v1",
  "kind": "AdmissionReview",
  "response": {
    "uid": "<value from request.uid>",
    "allowed": true
  }
}
// or
{
  "apiVersion": "admission.k8s.io/v1",
  "kind": "AdmissionReview",
  "response": {
    "uid": "<value from request.uid>",
    "allowed": false,
    "status": {
      "code": 402,
      "status": "Failure",
      "message": "xxx",
      "reason": "xxxx"
    }
  }
}

The determining fields are .response.uid, which uniquely identifies the request, and .response.allowed, which indicates whether the request has passed or failed, and the status field, which is mainly for error messages.

We can refer directly to the example implementation code in the Kubernetes e2e test.

https://github.com/kubernetes/kubernetes/blob/release-1.25/test/images/agnhost/webhook/main.go

Writing a webhook

Once the previous prerequisites have been met, let’s implement a webhook example that performs validating and mutating webhook authentication by listening to two different HTTP endpoints (validate and mutate).

The full code for this webhook is available on Github: https://github.com/cnych/admission-webhook-example(train4 branch). The webhook is a simple HTTP service with TLS authentication, deployed in our cluster using Deployment.

The main logic in the code is in two files: main.go and webhook.go. main.go contains the code to create the HTTP service, while webhook.go contains the logic for both the validates and mutates webhooks, most of the code is relatively simple. file to see how to use the standard golang package to start the HTTP service and how to read the TLS configured certificate from the command line flags.

1
2

flag.StringVar(&parameters.certFile, "tlsCertFile", "/etc/webhook/certs/cert.pem", "File containing the x509 Certificate for HTTPS.")
flag.StringVar(&parameters.keyFile, "tlsKeyFile", "/etc/webhook/certs/key.pem", "File containing the x509 private key to --tlsCertFile.")

Then there is the more important serve function, which handles incoming HTTP requests for the mutate and validating functions. This function deserializes the AdmissionReview object from the request, performs some basic content validation, calls the appropriate mutate and validating functions based on the URL path, and then serializes the AdmissionReview object.

// Serve method for webhook server
func (whsvr *WebhookServer) serve(w http.ResponseWriter, r *http.Request) {
 var body []byte
 if r.Body != nil {
  if data, err := ioutil.ReadAll(r.Body); err == nil {
   body = data
  }
 }
 if len(body) == 0 {
  glog.Error("empty body")
  http.Error(w, "empty body", http.StatusBadRequest)
  return
 }

 // verify the content type is accurate
 contentType := r.Header.Get("Content-Type")
 if contentType != "application/json" {
  glog.Errorf("Content-Type=%s, expect application/json", contentType)
  http.Error(w, "invalid Content-Type, expect `application/json`", http.StatusUnsupportedMediaType)
  return
 }

 var admissionResponse *admissionv1.AdmissionResponse
 ar := admissionv1.AdmissionReview{}
 if _, _, err := deserializer.Decode(body, nil, &ar); err != nil {
  glog.Errorf("Can't decode body: %v", err)
  admissionResponse = &admissionv1.AdmissionResponse{
   Result: &metav1.Status{
    Message: err.Error(),
   },
  }
 } else {
  fmt.Println(r.URL.Path)
  if r.URL.Path == "/mutate" {
   admissionResponse = whsvr.mutate(&ar)
  } else if r.URL.Path == "/validate" {
   admissionResponse = whsvr.validate(&ar)
  }
 }

 admissionReview := admissionv1.AdmissionReview{}
 admissionReview.TypeMeta = metav1.TypeMeta{
  Kind:       "AdmissionReview",
  APIVersion: "admission.k8s.io/v1",
 }
 if admissionResponse != nil {
  admissionReview.Response = admissionResponse
  if ar.Request != nil {
   admissionReview.Response.UID = ar.Request.UID
  }
 }

 resp, err := json.Marshal(admissionReview)
 if err != nil {
  glog.Errorf("Can't encode response: %v", err)
  http.Error(w, fmt.Sprintf("could not encode response: %v", err), http.StatusInternalServerError)
 }
 glog.Infof("Ready to write reponse ...")
 if _, err := w.Write(resp); err != nil {
  glog.Errorf("Can't write response: %v", err)
  http.Error(w, fmt.Sprintf("could not write response: %v", err), http.StatusInternalServerError)
 }
}

The main admission logic is the validate and mutate functions. The validate function checks if the resource object needs to be validated: resources in the kube-system namespace are not validated, and if you want to display a declaration that a resource is not validated, you can declare it by adding an admission-webhook-example.qikqiak.com/validate=false to the annotation. If validation is required, the resource type is compared to its counterpart based on the kind of the resource type and the tag. The service or deployment resource is deserialized from the request. If some label is missing, Allowed is set to false in the response. if validation fails, the reason for the failure is written in the response and the end user receives a failure message when trying to create the resource. the validate function is implemented as shown below.

// validate deployments and services
func (whsvr *WebhookServer) validate(ar *admissionv1.AdmissionReview) *admissionv1.AdmissionResponse {
 req := ar.Request
 var (
  availableLabels                 map[string]string
  objectMeta                      *metav1.ObjectMeta
  resourceNamespace, resourceName string
 )

 glog.Infof("AdmissionReview for Kind=%v, Namespace=%v Name=%v (%v) UID=%v patchOperation=%v UserInfo=%v",
  req.Kind, req.Namespace, req.Name, resourceName, req.UID, req.Operation, req.UserInfo)

 switch req.Kind.Kind {
 case "Deployment":
  var deployment appsv1.Deployment
  if err := json.Unmarshal(req.Object.Raw, &deployment); err != nil {
   glog.Errorf("Could not unmarshal raw object: %v", err)
   return &admissionv1.AdmissionResponse{
    Result: &metav1.Status{
     Message: err.Error(),
    },
   }
  }
  resourceName, resourceNamespace, objectMeta = deployment.Name, deployment.Namespace, &deployment.ObjectMeta
  availableLabels = deployment.Labels
 case "Service":
  var service corev1.Service
  if err := json.Unmarshal(req.Object.Raw, &service); err != nil {
   glog.Errorf("Could not unmarshal raw object: %v", err)
   return &admissionv1.AdmissionResponse{
    Result: &metav1.Status{
     Message: err.Error(),
    },
   }
  }
  resourceName, resourceNamespace, objectMeta = service.Name, service.Namespace, &service.ObjectMeta
  availableLabels = service.Labels
 }

 if !validationRequired(ignoredNamespaces, objectMeta) {
  glog.Infof("Skipping validation for %s/%s due to policy check", resourceNamespace, resourceName)
  return &admissionv1.AdmissionResponse{
   Allowed: true,
  }
 }

 allowed := true
 var result *metav1.Status
 glog.Info("available labels:", availableLabels)
 glog.Info("required labels", requiredLabels)
 for _, rl := range requiredLabels {
  if _, ok := availableLabels[rl]; !ok {
   allowed = false
   result = &metav1.Status{
    Reason: "required labels are not set",
   }
   break
  }
 }

 return &admissionv1.AdmissionResponse{
  Allowed: allowed,
  Result:  result,
 }
}

The method for determining whether checks are required is as follows, either by ignoring them through namespace or by configuring them through annotations settings.

func validationRequired(ignoredList []string, metadata *metav1.ObjectMeta) bool {
 required := admissionRequired(ignoredList, admissionWebhookAnnotationValidateKey, metadata)
 glog.Infof("Validation policy for %v/%v: required:%v", metadata.Namespace, metadata.Name, required)
 return required
}

func admissionRequired(ignoredList []string, admissionAnnotationKey string, metadata *metav1.ObjectMeta) bool {
 // skip special kubernetes system namespaces
 for _, namespace := range ignoredList {
  if metadata.Namespace == namespace {
   glog.Infof("Skip validation for %v for it's in special namespace:%v", metadata.Name, metadata.Namespace)
   return false
  }
 }

 annotations := metadata.GetAnnotations()
 if annotations == nil {
  annotations = map[string]string{}
 }

 var required bool
 switch strings.ToLower(annotations[admissionAnnotationKey]) {
 default:
  required = true
 case "n", "no", "false", "off":
  required = false
 }
 return required
}

The code for the mutate function is very similar, but instead of just comparing tags and setting Allowed in the response, it creates a patch that adds the missing tag to the resource and sets not_available to the value of the tag.

// main mutation process
func (whsvr *WebhookServer) mutate(ar *admissionv1.AdmissionReview) *admissionv1.AdmissionResponse {
 req := ar.Request
 var (
  availableLabels, availableAnnotations map[string]string
  objectMeta                            *metav1.ObjectMeta
  resourceNamespace, resourceName       string
 )

 glog.Infof("AdmissionReview for Kind=%v, Namespace=%v Name=%v (%v) UID=%v patchOperation=%v UserInfo=%v",
  req.Kind, req.Namespace, req.Name, resourceName, req.UID, req.Operation, req.UserInfo)

 switch req.Kind.Kind {
 case "Deployment":
  var deployment appsv1.Deployment
  if err := json.Unmarshal(req.Object.Raw, &deployment); err != nil {
   glog.Errorf("Could not unmarshal raw object: %v", err)
   return &admissionv1.AdmissionResponse{
    Result: &metav1.Status{
     Message: err.Error(),
    },
   }
  }
  resourceName, resourceNamespace, objectMeta = deployment.Name, deployment.Namespace, &deployment.ObjectMeta
  availableLabels = deployment.Labels
 case "Service":
  var service corev1.Service
  if err := json.Unmarshal(req.Object.Raw, &service); err != nil {
   glog.Errorf("Could not unmarshal raw object: %v", err)
   return &admissionv1.AdmissionResponse{
    Result: &metav1.Status{
     Message: err.Error(),
    },
   }
  }
  resourceName, resourceNamespace, objectMeta = service.Name, service.Namespace, &service.ObjectMeta
  availableLabels = service.Labels
 }

 if !mutationRequired(ignoredNamespaces, objectMeta) {
  glog.Infof("Skipping validation for %s/%s due to policy check", resourceNamespace, resourceName)
  return &admissionv1.AdmissionResponse{
   Allowed: true,
  }
 }

 annotations := map[string]string{admissionWebhookAnnotationStatusKey: "mutated"}
 patchBytes, err := createPatch(availableAnnotations, annotations, availableLabels, addLabels)
 if err != nil {
  return &admissionv1.AdmissionResponse{
   Result: &metav1.Status{
    Message: err.Error(),
   },
  }
 }

 glog.Infof("AdmissionResponse: patch=%v\n", string(patchBytes))
 return &admissionv1.AdmissionResponse{
  Allowed: true,
  Patch:   patchBytes,
  PatchType: func() *admissionv1.PatchType {
   pt := admissionv1.PatchTypeJSONPatch
   return &pt
  }(),
 }
}

Build

We’ve actually packaged the code into a docker image that you can use directly, the image repository address is: cnych/admission-webhook-example:v4. Of course, if you want to change some of the code, you’ll need to rebuild the project. Since this project is developed in go, the package management tool has been changed to go mod, so we need to make sure that the go environment is installed in the build environment, and of course docker is essential, since all we need is to package it as a docker image.

Clone Project.

1
2
3

☸ ➜ mkdir admission-webhook && cd admission-webhook
☸ ➜ git clone https://github.com/cnych/admission-webhook-example.git
☸ ➜ git checkout train4 # train4 branch

We can see that there is a build script under the root of the code, so we just need to provide our own docker image username and build it.

1
2
3


☸ ➜ export DOCKER_USER=cnych
☸ ➜ ./build

Deployment

Registering our webhook with the apiserver, how the apiserver knows the service exists and how to invoke the interface requires the use of the ValidatingWebhookConfiguration and MutatingWebhookConfiguration objects. By creating this resource object, apiserver will register our webhook in its ValidatingAdmissionWebhook controller module, with a few caveats when creating the object.

apiserver only supports HTTPS webhook, so you must prepare a TLS certificate, which you can usually get using Kubernetes CertificateSigningRequest or cert-manager.
clientConfig.caBundle is used to specify the CA certificate for issuing TLS certificates, if you use Kubernetes CertificateSigningRequest to issue certificates, you can get the cluster CA from the kube-public namespace clusterinfo, base64 formatted and written to clientConfig.caBundle; if you use cert-manager to issue the certificate, the component will automatically inject the certificate.
To prevent yourself from intercepting yourself, you can use objectSelector to exclude yourself.
For in-cluster deployments, use service ref to specify the service
For out-of-cluster deployments, use url to specify the HTTPS interface

Here we create a ValidatingWebhookConfiguration object as shown below.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: validation-webhook-example-cfg
  labels:
    app: admission-webhook-example
  annotations:
    cert-manager.io/inject-ca-from: default/admission-example-tls-secret # $(CERTIFICATE_NAMESPACE)/$(CERTIFICATE_NAME)
webhooks:
  - name: required-labels.qikqiak.com
    admissionReviewVersions:
      - v1
    clientConfig:
      caBundle: "" # "<Kubernetes CA> or <cert-manager CA>"
      service:
        name: admission-webhook-example-svc
        namespace: default
        port: 443
        path: /validate
    rules:
      - operations: ["CREATE"]
        apiGroups: ["apps", ""]
        apiVersions: ["v1"]
        resources: ["deployments", "services"]
    namespaceSelector:
      matchLabels:
        admission-webhook-example: enabled
    failurePolicy: Fail
    matchPolicy: Exact
    sideEffects: None

In this object we register a validating webhook and specify the address of the webhook via clientConfig.service, normally we would also specify the contents of the caBundle, but here we have configured a cert-manager.io/inject-ca-from: default/admission-example-tls-secret annotations, so that we can automatically inject CA content with cert-manager, and we also configure a namespaceSelector to indicate that we have The webhook will only be applied to namespaces with the admission-webhook-example: enabled tag.

So here we also need to deploy the cert-manager.

`1`	`☸ ➜ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.10.1/cert-manager.yaml`

The following Pods will be run in the cert-manager namespace when the installation is complete.

☸ ➜ kubectl get pods -n cert-manager
NAME                                      READY   STATUS    RESTARTS   AGE
cert-manager-74d949c895-4v8kj             1/1     Running   0          71s
cert-manager-cainjector-d9bc5979d-2tt4v   1/1     Running   0          71s
cert-manager-webhook-84b7ddd796-7wdgk     1/1     Running   0          71s

The cert-manager has a component called CA injector which is responsible for injecting the CA bundle into the Mutating | ValidatingWebhookConfiguration. This means that we need to use an annotation with key certmanager.k8s.io/inject-ca-from in the Mutating | ValidatingWebhookConfiguration object, and the value of the annotation should start with <certificate-namespace>/<certificate-name> to point to an existing certificate CR instance.

Here we create the Certificate object as shown below, requiring only the Issuer of selfSigned.

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: admission-example-issuer
  namespace: default
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: admission-example-tls-secret
spec:
  duration: 8760h
  renewBefore: 8000h
  subject:
    organizations:
      - qikqiak.com
  commonName: admission-webhook-example-svc.default
  isCA: false
  privateKey:
    algorithm: RSA
    encoding: PKCS1
    size: 2048
  usages:
    - digital signature
    - key encipherment
    - server auth
  dnsNames:
    - admission-webhook-example-svc
    - admission-webhook-example-svc.default
    - admission-webhook-example-svc.default.svc
  issuerRef:
    kind: Issuer
    name: admission-example-issuer
  secretName: admission-webhook-example-certs

Note that the Certificate object name corresponds to the annotations above, and then we write the final issued certificate to the Secret object called admission-webhook-example-certs.

Next we can deploy our webhook server. Deployment is very simple, we just need to configure the TLS configuration of the service. We can look at the configuration statement for the certificate in the deployment.yaml file under the deployment folder in the root of the code and see that the certificate and private key files read from the command line parameters are mounted via a secret object.

args:
 - -tlsCertFile=/etc/webhook/certs/tls.crt
 - -tlsKeyFile=/etc/webhook/certs/tls.key
[...]
 volumeMounts:
 - name: webhook-certs
  mountPath: /etc/webhook/certs
  readOnly: true
volumes:
- name: webhook-certs
  secret:
    secretName: admission-webhook-example-certs

The admission-webhook-example-certs object is automatically generated from the Certificate object created by the Cert-manager above, and once the secret object is created, we can create the deployment and service objects directly.

☸ ➜ kubectl apply -f deployment/rbac.yaml
☸ ➜ kubectl apply -f deployment/deployment.yaml
deployment.apps "admission-webhook-example-deployment" created
☸ ➜ kubectl apply -f deployment/service.yaml
service "admission-webhook-example-svc" created

We then add the following tag to the default namespace.

1
2

☸ ➜ kubectl label namespace default admission-webhook-example=enabled
namespace "default" labeled

Finally, just create the validatingwebhookconfigurations object above, and once it is created, it will intercept the request and call our webhook service.

`1`	`validatingwebhookconfigurations`

Testing

Now let’s create a deployment resource to verify that it works. There is a sleep.yaml resource manifest file under the code repository, just create it.

1
2

☸ ➜ kubectl apply -f deployment/sleep.yaml
Error from server (required labels are not set): error when creating "deployment/sleep.yaml": admission webhook "required-labels.qikqiak.com" denied the request: required labels are not set

Normally it would be created with the error message above because we didn’t bring any of the required tags and then deploy another sleep-with-labels.yaml list of resources.

1
2

☸ ➜ kubectl apply -f deployment/sleep-with-labels.yaml
deployment.apps "sleep" created

We can see that it deploys fine, then we remove the above deployment and deploy another sleep-no-validation.yaml resource manifest, which does not have the required tags, but is configured with an annotation like admission-webhook-example.qikqiak.com/validate=false, so that it can be created normally.

1
2
3

☸ ➜ kubectl delete deployment sleep
☸ ➜ kubectl apply -f deployment/sleep-no-validation.yaml
deployment.apps "sleep" created

Deploying mutating webhook

In the same way we can create a MutatingWebhookConfiguration object. First, we remove the validating webhook from above to prevent interference with mutating, and then deploy the new configuration. The mutating webhook configuration is basically the same as the validating webhook configuration, but the path to the webook server is /mutate.

1
2
3

### Deploying mutating webhook

In the same way we can create a `MutatingWebhookConfiguration` object. First, we remove the `validating webhook` from above to prevent interference with mutating, and then deploy the new configuration. The `mutating webhook` configuration is basically the same as the `validating webhook` configuration, but the path to the webook server is `/mutate`.

Now we can deploy the sleep application above again and see if the label tags are correctly added.

☸ ➜ kubectl get mutatingwebhookconfigurations
NAME                           WEBHOOKS   AGE
mutating-webhook-example-cfg   1          42s
☸ ➜ kubectl apply -f deployment/sleep.yaml
deployment.apps "sleep" created
☸ ➜ kubectl get deploy sleep -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    admission-webhook-example.qikqiak.com/status: mutated
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: "2023-01-05T08:43:59Z"
  generation: 1
  labels:
    app.kubernetes.io/component: not_available
    app.kubernetes.io/instance: not_available
    app.kubernetes.io/managed-by: not_available
    app.kubernetes.io/name: not_available
    app.kubernetes.io/part-of: not_available
    app.kubernetes.io/version: not_available
  name: sleep
  namespace: default
# ......

Finally, we recreate the validating webhook to test it together. Now, try to create the sleep application again. It works as expected.

Admission control is carried out in two stages, the first stage running the mutating admission controller and the second stage running the validating admission controller.

So mutating webhook adds the missing labels in the first stage, and then validating webhook doesn’t reject the deployment in the second stage, because the labels already exist, and sets their value with not_available.

☸ ➜ kubectl apply -f deployment/validatingwebhook.yaml
validatingwebhookconfiguration.admissionregistration.k8s.io "validation-webhook-example-cfg" created
☸ ➜ kubectl delete -f deployment/sleep.yaml
deployment.apps "sleep" deleted
☸ ➜ kubectl apply -f deployment/sleep.yaml
deployment.apps/sleep created

However, if we have such a need, it would be very cumbersome and inflexible to develop a separate webhook for the admission controller. To solve this problem we can use some policy management engines provided by Kubernetes to implement our needs without writing code, such as Kyverno, Gatekeeper, etc. Policy management is also officially supported in Kubernetes v1.26.

Table of Contents