Overview

controller-runtime is a subproject of Kubebuilder, which provides a series of libraries for building controllers; Kubebuilder itself generates a lot of template code that uses controller-runtime. controller-runtime contains several The basic concepts are as follows.

  • Controller: literally, a controller.
  • Reconciler: provides the Reconcile function, the main part of the Controller and the entry function, which contains all the business logic of the controller (equivalent to the syncHandler function in a normal controller), and is used to make the actual state of the object we care about gradually approach the to the desired state. The Reconciler also has the following features.
    • Usually only one type of object is targeted, and different types of objects use separate controllers.
    • Usually does not care about the content and type of events that trigger the Reconcile function; for example, whether a ReplicaSet is created or updated, the ReplicaSet Reconciler always compares the number of Pods in the cluster with the number set in the object, and then takes the appropriate action.
  • Builder: Generate Controller for Reconciler based on some configuration.
  • Manager: manages and starts Controller, a Manager can contain multiple Controllers.

Use

The following is a step-by-step description of how to use the official simple example to build a controller using controller-runtime controller: first define the Reconciler which contains the main logic of the controller, then use the Builder to generate the Controller and add it to the Manager, and finally start the Manager.

Note: Take v0.5.0 version as an example, the definition of Reconcile function has changed in the latest version.

Each step is described in detail as follows.

Reconciler

Introduction

Reconciler is defined as follows and contains only one Reconcile function.

1
2
3
4
5
6

type Reconciler interface {
    // Reconciler performs a full reconciliation for the object referred to by the Request.
    // The Controller will requeue the Request to be processed again if an error is non-nil or
    // Result.Requeue is true, otherwise upon completion it will remove the work from the queue.
    Reconcile(Request) (Result, error)
}

Request and Result are defined as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

// Request contains the information necessary to reconcile a Kubernetes object.  This includes the
// information to uniquely identify the object - its Name and Namespace.  It does NOT contain information about
// any specific Event or the object contents itself.
type Request struct {
    // NamespacedName is the name and namespace of the object to reconcile.
    types.NamespacedName
}

// Result contains the result of a Reconciler invocation.
type Result struct {
    // Requeue tells the Controller to requeue the reconcile key.  Defaults to false.
    Requeue bool

    // RequeueAfter if greater than 0, tells the Controller to requeue the reconcile key after the Duration.
    // Implies that Requeue is true, there is no need to set Requeue to true at the same time as RequeueAfter.
    RequeueAfter time.Duration
}

Request contains the namespace and name of this reconcile object, the object type is configured when generating the controller, a Reconciler can only handle one type of object; Result is basically nothing to care about, if Reconcile returns an error it will automatically requeue.

Usage examples

Define a ReplicaSetReconciler that contains a generic client provided by controller-runtime that functions like a normal kubernetes client and has access to all resources of the cluster.

1
2
3
4

// ReplicaSetReconciler is a simple ControllerManagedBy example implementation.
type ReplicaSetReconciler struct {
    client.Client
}

But unlike the normal kubernetes client, this generic client is a single client that CRUD’s all types of resources, making it very convenient and easy to use.

Then there is the implementation of business logic.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

// Implement the business logic:
// This function will be called when there is a change to a ReplicaSet or a Pod with an OwnerReference
// to a ReplicaSet.
//
// * Read the ReplicaSet
// * Read the Pods
// * Set a Label on the ReplicaSet with the Pod count
func (a *ReplicaSetReconciler) Reconcile(req reconcile.Request) (reconcile.Result, error) {
    // Read the ReplicaSet
    rs := &appsv1.ReplicaSet{}
    err := a.Get(context.TODO(), req.NamespacedName, rs)
    if err != nil {
        return reconcile.Result{}, err
    }

    // List the Pods matching the PodTemplate Labels
    pods := &corev1.PodList{}
    err = a.List(context.TODO(), client.InNamespace(req.Namespace).MatchingLabels(rs.Spec.Template.Labels), pods)
    if err != nil {
        return reconcile.Result{}, err
    }

    // Update the ReplicaSet
    rs.Labels["pod-count"] = fmt.Sprintf("%v", len(pods.Items))
    err = a.Update(context.TODO(), rs)
    if err != nil {
        return reconcile.Result{}, err
    }

    return reconcile.Result{}, nil
}

InjectClient assigns the manager’s real client to ReplicaSetReconciler.

1
2
3
4

func (a *ReplicaSetReconciler) InjectClient(c client.Client) error {
    a.Client = c
    return nil
}

Builder and Manager

Builder is used to generate Controller for Reconciler and Manager is used to manage and launch Controller, which is introduced directly with an example, first generating a Manager.

1
2
3
4
5

mgr, err := manager.New(config, manager.Options{})
if err != nil {
    log.Error(err, "could not create manager")
    os.Exit(1)
}

where config is rest.Config for client-go.

Generate Controller for ReplicaSetReconciler and add it to Manager.

1
2
3
4
5
6
7
8
9

_, err = builder.
    ControllerManagedBy(mgr).  // Create the ControllerManagedBy
    For(&appsv1.ReplicaSet{}). // ReplicaSet is the Application API
    Owns(&corev1.Pod{}).       // ReplicaSet owns Pods created by it
    Build(&ReplicaSetReconciler{})
if err != nil {
    log.Error(err, "could not create controller")
    os.Exit(1)
}

The For function is used to specify the type of object we want to reconcile, and Owns is used to watch the object whose owner is the reconcile object type (Owns can specify multiple types), the add/delete/change events of both types of object will trigger the Reconcile function.

Finally the Manager is started and the whole component is started.

1
2
3
4

if err := mgr.Start(stopCh); err != nil {
    log.Error(err, "could not start manager")
    os.Exit(1)
}

Unit testing

If you want to write unit tests for a common controller, you should rely on the fake client provided by client-go, append various objects needed for testing to a fake client, and use the fake client to complete the tests. If you use the lister, you need to manually add the corresponding object to the corresponding informer indexer, for example.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

// 创建 fake client
f.client = fake.NewSimpleClientset(f.objects...)

// 创建基于 fake client 的 informer
informer := informers.NewSharedInformerFactory(f.client, 1)

// 往 informer indexer 中添加 object
for _, s := range f.storageClasses {
    informer.Native().Storage().V1().StorageClasses().Informer().GetIndexer().Add(s)
}

The controller-runtime uses its own envtest package to start a real apiserver and etcd locally, and then connects to this apiserver for testing. We still need fake objects, but unlike the fake client, these fake objects are created in the apiserver started by controller-runtime, and in the case of CR, CRD needs to be registered with the new apiserver first. a complete example.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63

func TestNodeLocalStorageReconciler(t *testing.T) {
    // 配置 testEnv，其中包含该项单测需要的 CRD 等
    testEnv := &envtest.Environment{
        CRDs: []runtime.Object{
            newNodeLocalStorageCRD(),
        },
    }

    // 启动测试环境（etcd 和 apiserver），返回该环境的 rest config
    config, err := testEnv.Start()
    if err != nil {
        t.Fatal(err)
    }
    defer testEnv.Stop()

    // 生成 controller-runtime 需要的 client
    c, err := client.New(config, client.Options{
        Scheme: scheme,
    })
    if err != nil {
        t.Fatal(err)
    }

    // 初始化 Reconciler
    nlsReconciler := &NodeLocalStorageReconciler{
        Client: c,
    }

    // 准备好测试数据
    nls1 := test.GetNodeLocalStorage()
    nls1.Name = "test"
    if err = c.Create(context.TODO(), nls1); err != nil {
        t.Fatal(err)
    }

    // test cases
    testCases := []struct {
        describe  string
        namespace string
        name      string
        isErr     bool
    }{
        {
            describe:  "normal test",
            namespace: "",
            name:      "test",
            isErr:     false,
        },
    }

    // 测试每个 case
    for _, testCase := range testCases {
        _, err := nlsReconciler.Reconcile(reconcile.Request{
            NamespacedName: types.NamespacedName{
                Namespace: testCase.namespace,
                Name:      testCase.name,
            },
        })
        if testCase.isErr != (err != nil) {
            t.Fatalf("%s unexpected error: %v", testCase.describe, err)
        }
    }
}

Summary

The controller-runtime framework itself provides a lot of libraries to help build the controller, making the whole process simple, shielding a lot of generic details to make the whole process of building the controller easier, interested students can try it in different scenarios and needs; even if we do not use controller-runtime we can also use its generic client and envtest and other generic libraries alone.

Separately, to use envtest you need to install a series of bin files (mainly etcd and kube-apiserver) provided by Kubebuilder in your runtime environment (your local and CI environment), you can download them yourself if you are local, or add them to the base image if you need them for your CI environment. An example.

1
2
3
4
5

RUN mkdir -p /usr/local && \
    wget https://go.kubebuilder.io/dl/2.3.1/linux/amd64 && \
    tar xvf amd64 && \
    mv kubebuilder_2.3.1_linux_amd64 /usr/local/kubebuilder && \
    rm amd64