Kubebuilder can build AdmissionWebhooks in addition to CRD APIs and their Controllers, and this article will analyze how Kubebuilder builds AdmissionWebhooks in detail.

AdmissionWebhooks for K8s

First of all, it is important to know what AdmissionWebhooks is in K8s and what its purpose is.

Let’s start with the scenario. If we need to make configuration changes or checks on a pod before it is created, this part of the work would require the administrator to compile it into a binary file in ApiServer, which would be very annoying if the configuration changes were made in a custom form. Admission controllers are tools for this scenario, and are attached to ApiServer in the form of plug-ins, of which AdmissionWebhooks is one.

K8s’ AdmissionWebhooks are of two kinds: MutatingAdmissionWebhook and ValidatingAdmissionWebhook, which together are a special type of admission controllers, one dealing with resource changes and the other with validation.

MutatingAdmissionWebhook does three main things.

  1. MutatingWebhookConfiguration: the configuration for MutatingAdmissionWebhook to register itself with the ApiServer.
  2. MutatingAdmissionWebhook itself: an admission controller in the form of a plugin that needs to register itself with the ApiServer.
  3. Webhook Admission Server: an http server attached to the k8s ApiServer that receives requests from the ApiServer.

If we use Kubebuilder to build AdmissionWebhooks, Kubebuilder will automatically generate the Webhook Server for us and leave a few functions for us to add our own logic.

Creating Custom AdmissionWebhooks

Here’s a demo using a simple scenario where we customize a resource called App, and when a user creates an App instance, we create a Deployment based on the user’s description.

Then we add a MutatingAdmissionWebhook that automatically adds a sidecar container to the Pod when the user creates a Deployment via the App (using nginx as the sidecar here).

Initialize API and Controller

The first step is to create the CRD and its Controller, which can be done with a few lines of command.

1
2
3
4
5
6
7

$ export GO111MODULE=on

$ mkdir $GOPATH/src/zww-app
$ cd $GOPATH/src/zww-app
$ kubebuilder init --domain o0w0o.cn --owner "zwwhdls"

$ kubebuilder create api --group app --version v1 --kind App

What I’ve done here is relatively simple. The AppSpec only defines a deploy property (which is appsv1.DeploymentSpec), and the Controller generates the corresponding Deployment based on the deploy property.

1
2
3
4
5

type AppSpec struct {
	// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
	// Important: Run "make" to regenerate code after modifying this file
	Deploy appsv1.DeploymentSpec `json:"deploy,omitempty"`
}

After refining the Reconcile functions of AppSpec and Controller, make Kubebuilder regenerate the code and apply the CRD yaml under config/crd to the current cluster.

1
2

make
make install

Creating a Webhook Server

The next step is to use Kubebuilder to generate Webhooks.

1

kubebuilder create webhook --group app --version v1 --kind App

A file named app_webhook.go is generated under the path api/v1. You can see that Kubebuilder has defined two variables for you.

1
2

var _ webhook.Defaulter = &App{}
var _ webhook.Validator = &App{}

These two variables represent MutatingWebhookServer and ValidatingWebhookServer respectively, which will run up when the program starts.

For MutatingWebhookServer, Kubebuilder reserves the Default() function for users to fill in their own logic.

1
2
3
4
5
6

// Default implements webhook.Defaulter so a webhook will be registered for the type
func (r *App) Default() {
	applog.Info("default", "name", r.Name)

	// TODO(user): fill in your defaulting logic.
}

For what kind of changes we want the Webhook to trigger when the resource changes, you can modify it with this comment.

1

// +kubebuilder:webhook:path=/mutate-app-o0w0o-cn-v1-app,mutating=true,failurePolicy=fail,groups=app.o0w0o.cn,resources=apps,verbs=create;update,versions=v1,name=mapp.kb.io

The corresponding parameters are.

  • failurePolicy: indicates the failure policy when the ApiServer cannot communicate with the webhook server, takes the value of “ignore” or “fail”.
  • groups: indicates the Api Group under which this webhook will receive requests.
  • mutating: this parameter is a bool type, indicating whether it is a mutating type.
  • name: the name of the webhook, which should correspond to the configuration.
  • path: the path of the webhook.
  • resources: indicates which resource the webhook will receive a request for when it changes.
  • verbs: indicates the webhook will receive requests for which resource changes, and takes the values “create”, “update”, “delete”, “connect”, or “*” (i.e., all).
  • versions: indicates at which version of the resource this webhook will receive a request when it changes.

For ValidatingWebhookServer, Kubebuilder handles it in the same way as MutatingWebhookServer, so I won’t go over it here.

For convenience, I only defined the Default function of MutatingWebhookServer to inject an nginx sidecar container for each pod of App-type resources.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

func (r *App) Default() {
	applog.Info("default", "name", r.Name)
	var cns []core.Container
	cns = r.Spec.Deploy.Template.Spec.Containers

	container := core.Container{
		Name:  "sidecar-nginx",
		Image: "nginx:1.12.2",
	}

	cns = append(cns, container)
	r.Spec.Deploy.Template.Spec.Containers = cns
}

Running Webhook Server

This article only shares the debugging solution for local development testing, please refer to the official documentation for the online deployment solution.

First, you need to modify MutatingWebhookConfiguration slightly to make ApiServer communicate with Webhook Server. The specific method is as follows.

Configuring Server Path

The first step is to configure Server Path; remove the service and replace it with url: https://<server_ip>:9443/mutate-app-o0w0o-cn-v1-app, where server_ip is the ip of the Webhook Server, or if running locally, the local ip. Note that the path in the url should be the same as the one defined in app_webhook.go.

Configure the certificate

The second step is to configure caBundle; since all components interacting with ApiServer in Kube require bi-directional TLS authentication with ApiServer, we need to manually issue a self-signed CA certificate here first.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

$ openssl genrsa -out ca.key 2048
$ openssl req -x509 -new -nodes -key ca.key -subj "/CN=<server_ip>" -days 10000 -out ca.crt
$ openssl genrsa -out server.key 2048
$ cat << EOF >csr.conf
> [ req ]
> default_bits = 2048
> prompt = no
> default_md = sha256
> req_extensions = req_ext
> distinguished_name = dn
> 
> [ dn ]
> C = <country>
> ST = <state>
> L = <city>
> O = <organization>
> OU = <organization unit>
> CN = <server_ip>
> 
> [ req_ext ]
> subjectAltName = @alt_names
> 
> [ alt_names ]
> IP.1 = <server_ip>
> 
> [ v3_ext ]
> authorityKeyIdentifier=keyid,issuer:always
> basicConstraints=CA:FALSE
> keyUsage=keyEncipherment,dataEncipherment
> extendedKeyUsage=serverAuth,clientAuth
> subjectAltName=@alt_names
> EOF
$ openssl req -new -key server.key -out server.csr -config csr.conf
$ openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 10000 -extensions v3_ext -extfile csr.conf

After certificate generation, copy server.key and server.crt to the private key and certificate path of the webhook server set by Kubebuilder: $(TMPDIR)/k8s-webhook-server/serving-certs/tls.key.

  • Path to the private key of the webhook server: $(TMPDIR)/k8s-webhook-server/serving-certs/tls.key
  • Path to webhook server’s certificate: $(TMPDIR)/k8s-webhook-server/serving-certs/tls.crt

Note: If $(TMPDIR) is empty, the default path is “/tmp/k8s-webhook-server/…” but the default path for android is “/data/local/tmp/k8s-webhook-server/…”.

And the caBundle in MutatingWebhookConfiguration is the base64 encoded result of ca.crt. The final yaml result is as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
  creationTimestamp: null
  name: mutating-webhook-configuration
webhooks:
- clientConfig:
  caBundle: LS0tLS1CRUdJTiBDRVJ...FLS0tLS0=
  url: https://<server_ip>:9443/mutate-app-o0w0o-cn-v1-app
  failurePolicy: Fail
  name: mapp.kb.io
  rules:
    ...

ValidatingWebhookConfiguration is similar to MutatingWebhookConfiguration, just note that the server path is the same as in app_webhook.go. After both configuration files are modified, apply them in the cluster.

runs

Finally, run the CRD Controller and Webhook Server directly locally.

1

make run

Verification

Simply run an app and try it.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

apiVersion: app.o0w0o.cn/v1
kind: App
metadata:
  name: app-sample
spec:
  deploy:
    selector:
      matchLabels:
        app: app-sample
    template:
      metadata:
        name: sample
        labels:
          app: app-sample
      spec:
        containers:
          - name: cn
            image: daocloud.io/library/redis:4.0.14-alpine

To see if the sidecar container has been injected.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

$ kubectl apply -f config/samples/app_v1_app.yaml
$ kubectl get app
NAME         AGE
app-sample   43s
$ kubectl get deploy
NAME                READY   UP-TO-DATE   AVAILABLE   AGE
app-sample-deploy   0/1     1            0           43s
$ kubectl get po
NAME                                 READY   STATUS              RESTARTS   AGE
app-sample-deploy-5b5cfb9c9b-z8jk5   0/2     ContainerCreating   0          43s