I guess many people, like me, were confused by the various repositories when they first opened the kubernetes project source code on Github. kuberentes has many repositories under the organization, including kubernetes, client-go, api, apimachinery, etc., where should I start? The kubernetes repository should be the core repository of the kubernetes project, which contains the source code of the core components of the kubernetes control plane; client-go is also the go language client that operates the kubernetes API, as the name suggests; api and apimachinery should be the repositories related to the kubernetes API, but Why are they separated into two different repositories? How do these code repositories interact with each other? apimachinery repository also has api and apis packages that define various complex interfaces and implementations. It is beneficial to have a clear understanding of these complex interfaces to extend the kubernetes API. So, this article will focus on api and apimachinery repositories.

api

We know that kubernetes officially provides a variety of API resource types that are defined in the k8s.io/api repository, which serves as the canonical address for kubernetes API definitions. In fact, at first this repository was only part of the kubernetes core repository, but later the kubernetes API was used by more and more other repositories, such as k8s.io/client-go, k8s.io/apimachinery, k8s.io/apiserver, etc. To avoid cross-dependencies, the api as a separate repository to avoid cross-dependencies.

The k8s.io/api repository is a read-only repository and all code is synchronized from the https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/api core repository.

In the kubernetes API specification defined by the k8s.io/api repository, Pods are used as the most basic resource type. A typical serialized pod object in YAML form is shown below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
apiVersion: v1
kind: Pod
metadata:
  name: webserver
  labels:
    app: webserver
spec:
  containers:
  - name: webserver
    image: nginx
    ports:
    - containerPort: 80

From a programming perspective, the serialized pod object is eventually sent to the API-Server and decoded into a Go structure of type Pod, and the fields in the YAML are assigned to that Go structure. So, how is the Pod type defined in a Go structure?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
// source code from https://github.com/kubernetes/api/blob/master/core/v1/types.go
type Pod struct {
    // From the TypeMeta field name, you can see that this field defines the meta-information of the Pod type, similar to the object-oriented programming inside Class itself, similar to the API grouping, API version, etc. of the Pod type
    metav1.TypeMeta `json:",inline"`
    // The ObjectMeta field defines the meta-information for a single Pod object. Each kubernetes resource object has its own meta-information, such as name, namespace, tag, annotation, etc. kuberentes extracts these public properties as metav1.ObjectMeta, which becomes the parent class of the API object type
    metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`
     // PodSpec represents the object definition specification for the Pod type, most representative of CPU and memory resource usage. This field corresponds to the spec field in YAML
    Spec PodSpec `json:"spec,omitempty" protobuf:"bytes,2,opt,name=spec"`
    // PodStatus indicates the status of the pod, such as whether it is running or hanging, the IP of the pod, etc. Kubernetes updates the PodStatus field based on the actual status of the pod in the Kubernetes updates the PodStatus field based on the actual status of the pod in the cluster
    Status PodStatus `json:"status,omitempty" protobuf:"bytes,3,opt,name=status"`
}

As can be seen from the structure defined by Pod above, it inherits the metav1.TypeMeta and metav1.ObjectMeta types. metav1.TypeMeta corresponds to the kind and apiVersion segments in YAML, while metav1.ObjectMeta corresponds to the metadata field ObjectMeta corresponds to the metadata field. This can also be seen in the Go structure’s field json tag. In addition to the metav1.TypeMeta and metav1.ObjectMeta fields, the Pod structure also defines two member variables, Spec and Status. If you look at the structure definitions of other API resources in the k8s.io/api repository, you will see that most of the kubernetes API resource types are similar structures, meaning that the kubernetes API resource types all inherit from metav1.TypeMeta and metav1. The former is used to define the public properties of the resource type, and the latter is used to define the public properties of the resource object; Spec is used to define the private properties of the API resource type, and is what differentiates the different API resource types; and Status is used to describe the state of each resource object, which is closely related to each resource type.

Regarding the metav1.TypeMeta and metav1.ObjectMeta fields, they are also semantically well understood as the base class for all kubernetes API resource objects. Each API resource object needs the metav1.TypeMeta field to describe what type it is so that it can construct objects of the corresponding type, so the metav1.TypeMeta field is the same for all resource objects of the same type, but metav1.ObjectMeta is different in that it is the property that defines the resource object instance properties that all resource object instances should have. This part is just related to the object instance itself and has nothing to do with the type, so the metav1.ObjectMeta for resource object instances of the same type is generally different.

In addition to single objects in the kubernetes API resource objects, there is also the object list type, which is used to describe a list of objects of the same type. The typical application scenario for object lists is enumeration, where an object list can express a set of resource objects. Some people may ask why not use the object slice, such as []Pod , and as the discussion progresses, this will be understood. Here we take PodList as an example for analysis.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
// source code from https://github.com/kubernetes/api/blob/master/core/v1/types.go
type PodList struct {
    // PodList also needs to inherit from metav1.TypeMeta, after all, both object list and single object need type property.
    // PodList has more type description than []Pod type in yaml or json expression, when you need to build a list of objects based on YAML, you can deserialize it to PodList based on the type description, while []Pod can not, you must make sure that YAML is []Pod serialized result, otherwise an error will be reported. This does not allow for a generic object serialization/deserialization.
    metav1.TypeMeta `json:",inline"`
    // Unlike Pod, PodList inherits from metav1.ListMeta, which is the parent class of all resource object list types. ListMeta defines the common properties of all instances of the object list type.
    metav1.ListMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`
    // The Items field is the essence of the PodList definition, and represents a list of Pod resource objects, so the PodList is []Pod with a few additions on top meta information related to the type and object list
    Items []Pod `json:"items" protobuf:"bytes,2,rep,name=items"`
}

Before we start the next section, let’s make a short summary.

  1. metav1.TypeMeta and metav1.ObjectMeta are the parent classes of all API singleton resource objects.
  2. metav1.TypeMeta and metav1.ListMeta are the parents of all API resource object lists.
  3. metav1.TypeMeta is the parent class of all API resource objects, since all resource objects specify what type is represented.

metav1

Here metav1 is an alias for package k8s.io/apimachinery/pkg/apis/meta/v1, which will be referred to by metav1 for the rest of this article.

metav1.TypeMeta

metav1.TypeMeta is used to describe the meta information of the kubernetes API resource object type, including the name of the resource type and the schema of the corresponding API. The schema here mainly includes the resource grouping and version information.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
// source code from https://github.com/kubernetes/apimachinery/blob/master/pkg/apis/meta/v1/types.go
//
// TypeMeta describes an individual object in an API response or request
// with strings representing the type of the object and its API schema version.
// Structures that are versioned or persisted should inline TypeMeta.
type TypeMeta struct {
    // Kind is a string value representing the REST resource this object represents.
    // Servers may infer this from the endpoint the client submits requests to.
    // Cannot be updated.
    Kind string `json:"kind,omitempty" protobuf:"bytes,1,opt,name=kind"`

    // APIVersion defines the versioned schema of this representation of an object.
    // Servers should convert recognized schemas to the latest internal value, and
    // may reject unrecognized values.
    APIVersion string `json:"apiVersion,omitempty" protobuf:"bytes,2,opt,name=apiVersion"`
}

TypeMeta itself implements the schema.ObjectKind interface and contains the GetObjectKind() method to get the schema.ObjectKind type object.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
// source code from https://github.com/kubernetes/apimachinery/blob/master/pkg/apis/meta/v1/types.go

func (obj *TypeMeta) GetObjectKind() schema.ObjectKind { return obj }

// SetGroupVersionKind satisfies the ObjectKind interface for all objects that embed TypeMeta
func (obj *TypeMeta) SetGroupVersionKind(gvk schema.GroupVersionKind) {
    obj.APIVersion, obj.Kind = gvk.ToAPIVersionAndKind()
}

// GroupVersionKind satisfies the ObjectKind interface for all objects that embed TypeMeta
func (obj *TypeMeta) GroupVersionKind() schema.GroupVersionKind {
    return schema.FromAPIVersionAndKind(obj.APIVersion, obj.Kind)
}

This means that any object that inherits metav1.TypeMeta (that is, any kubernetes API resource object) knows how to decode and encode serialized object type information.

metav1.ObjectMeta

The metav1.ObjectMeta is used to define the properties of the resource object instance, i.e. the properties that all resource objects should have. This part is related to the object itself, not to the type, so the metav1.ObjectMeta may be different for resource objects of the same type.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
// source code from https://github.com/kubernetes/apimachinery/blob/master/pkg/apis/meta/v1/types.go
//
// ObjectMeta is metadata that all persisted resources must have, which includes all objects
// users must create.
type ObjectMeta struct {
    // Name must be unique within a namespace. Is required when creating resources, although
    // some resources may allow a client to request the generation of an appropriate name
    // automatically. Name is primarily intended for creation idempotence and configuration
    // definition.
    Name string `json:"name,omitempty" protobuf:"bytes,1,opt,name=name"`

    // GenerateName is an optional prefix, used by the server, to generate a unique
    // name ONLY IF the Name field has not been provided.
    // Populated by the system. Read-only.
    GenerateName string `json:"generateName,omitempty" protobuf:"bytes,2,opt,name=generateName"`

    // Namespace defines the space within which each name must be unique. An empty namespace is
    // equivalent to the "default" namespace, but "default" is the canonical representation.
    // Not all objects are required to be scoped to a namespace - the value of this field for
    // those objects will be empty.
    Namespace string `json:"namespace,omitempty" protobuf:"bytes,3,opt,name=namespace"`

    // SelfLink is a URL representing this object.
    // Populated by the system. Read-only.
    SelfLink string `json:"selfLink,omitempty" protobuf:"bytes,4,opt,name=selfLink"`

    // UID is the unique in time and space value for this object. It is typically generated by
    // the server on successful creation of a resource and is not allowed to change on PUT
    // operations.
    // Populated by the system. Read-only.
    UID types.UID `json:"uid,omitempty" protobuf:"bytes,5,opt,name=uid,casttype=k8s.io/kubernetes/pkg/types.UID"`

    // An opaque value that represents the internal version of this object that can
    // be used by clients to determine when objects have changed. May be used for optimistic
    // concurrency, change detection, and the watch operation on a resource or set of resources.
    // Clients must treat these values as opaque and passed unmodified back to the server.
    // They may only be valid for a particular resource or set of resources.
    // Populated by the system. Read-only.
    ResourceVersion string `json:"resourceVersion,omitempty" protobuf:"bytes,6,opt,name=resourceVersion"`

    // A sequence number representing a specific generation of the desired state.
    // Populated by the system. Read-only.
    Generation int64 `json:"generation,omitempty" protobuf:"varint,7,opt,name=generation"`

    // CreationTimestamp is a timestamp representing the server time when this object was
    // created. It is not guaranteed to be set in happens-before order across separate operations.
    // Clients may not set this value. It is represented in RFC3339 form and is in UTC.
    // Populated by the system. Read-only.
    CreationTimestamp Time `json:"creationTimestamp,omitempty" protobuf:"bytes,8,opt,name=creationTimestamp"`

    // DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This
    // field is set by the server when a graceful deletion is requested by the user, and is not
    // directly settable by a client. The resource is expected to be deleted (no longer visible
    // from resource lists, and not reachable by name) after the time in this field, once the
    // finalizers list is empty. As long as the finalizers list contains items, deletion is blocked.
    // Once the deletionTimestamp is set, this value may not be unset or be set further into the
    // future, although it may be shortened or the resource may be deleted prior to this time.
    // For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react
    // by sending a graceful termination signal to the containers in the pod. After that 30 seconds,
    // the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup,
    // remove the pod from the API. In the presence of network partitions, this object may still
    // exist after this timestamp, until an administrator or automated process can determine the
    // resource is fully terminated.
    // If not set, graceful deletion of the object has not been requested.
    //
    // Populated by the system when a graceful deletion is requested.
    // Read-only.
    DeletionTimestamp *Time `json:"deletionTimestamp,omitempty" protobuf:"bytes,9,opt,name=deletionTimestamp"`

    // Number of seconds allowed for this object to gracefully terminate before
    // it will be removed from the system. Only set when deletionTimestamp is also set.
    // May only be shortened.
    // Read-only.
    DeletionGracePeriodSeconds *int64 `json:"deletionGracePeriodSeconds,omitempty" protobuf:"varint,10,opt,name=deletionGracePeriodSeconds"`

    // Map of string keys and values that can be used to organize and categorize
    // (scope and select) objects. May match selectors of replication controllers
    // and services.
    Labels map[string]string `json:"labels,omitempty" protobuf:"bytes,11,rep,name=labels"`

    // Annotations is an unstructured key value map stored with a resource that may be
    // set by external tools to store and retrieve arbitrary metadata. They are not
    // queryable and should be preserved when modifying objects.
    Annotations map[string]string `json:"annotations,omitempty" protobuf:"bytes,12,rep,name=annotations"`

    // List of objects depended by this object. If ALL objects in the list have
    // been deleted, this object will be garbage collected. If this object is managed by a controller,
    // then an entry in this list will point to this controller, with the controller field set to true.
    OwnerReferences []OwnerReference `json:"ownerReferences,omitempty" patchStrategy:"merge" patchMergeKey:"uid" protobuf:"bytes,13,rep,name=ownerReferences"`

    // Must be empty before the object is deleted from the registry. Each entry
    // is an identifier for the responsible component that will remove the entry
    // from the list. If the deletionTimestamp of the object is non-nil, entries
    // in this list can only be removed.
    // Finalizers may be processed and removed in any order.  Order is NOT enforced
    // because it introduces significant risk of stuck finalizers.
    // finalizers is a shared field, any actor with permission can reorder it.
    // If the finalizer list is processed in order, then this can lead to a situation
    // in which the component responsible for the first finalizer in the list is
    // waiting for a signal (field value, external system, or other) produced by a
    // component responsible for a finalizer later in the list, resulting in a deadlock.
    // Without enforced ordering finalizers are free to order amongst themselves and
    // are not vulnerable to ordering changes in the list.
    Finalizers []string `json:"finalizers,omitempty" patchStrategy:"merge" protobuf:"bytes,14,rep,name=finalizers"`

    // The name of the cluster which the object belongs to.
    // This is used to distinguish resources with same name and namespace in different clusters.
    // This field is not set anywhere right now and apiserver is going to ignore it if set in create or update request.
    ClusterName string `json:"clusterName,omitempty" protobuf:"bytes,15,opt,name=clusterName"`

    // ManagedFields maps workflow-id and version to the set of fields
    // that are managed by that workflow. This is mostly for internal
    // housekeeping, and users typically shouldn't need to set or
    // understand this field. A workflow can be the user's name, a
    // controller's name, or the name of a specific apply path like
    // "ci-cd". The set of fields is always in the version that the
    // workflow used when modifying the object.
    ManagedFields []ManagedFieldsEntry `json:"managedFields,omitempty" protobuf:"bytes,17,rep,name=managedFields"`
}

metav1.ObjectMeta also implements metav1.Object and metav1. ObjectMetaAccessor, where the metav1.Object interface defines Get and Set methods for obtaining various meta-information about a single resource object.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
// source code from https://github.com/kubernetes/apimachinery/blob/master/pkg/apis/meta/v1/meta.go
//
// Object lets you work with object metadata from any of the versioned or
// internal API objects. Attempting to set or retrieve a field on an object that does
// not support that field (Name, UID, Namespace on lists) will be a no-op and return
// a default value.
type Object interface {
    GetNamespace() string
    SetNamespace(namespace string)
    GetName() string
    SetName(name string)
    GetGenerateName() string
    SetGenerateName(name string)
    GetUID() types.UID
    SetUID(uid types.UID)
    GetResourceVersion() string
    SetResourceVersion(version string)
    GetGeneration() int64
    SetGeneration(generation int64)
    GetSelfLink() string
    SetSelfLink(selfLink string)
    GetCreationTimestamp() Time
    SetCreationTimestamp(timestamp Time)
    GetDeletionTimestamp() *Time
    SetDeletionTimestamp(timestamp *Time)
    GetDeletionGracePeriodSeconds() *int64
    SetDeletionGracePeriodSeconds(*int64)
    GetLabels() map[string]string
    SetLabels(labels map[string]string)
    GetAnnotations() map[string]string
    SetAnnotations(annotations map[string]string)
    GetFinalizers() []string
    SetFinalizers(finalizers []string)
    GetOwnerReferences() []OwnerReference
    SetOwnerReferences([]OwnerReference)
    GetClusterName() string
    SetClusterName(clusterName string)
    GetManagedFields() []ManagedFieldsEntry
    SetManagedFields(managedFields []ManagedFieldsEntry)
}

The metav1.ObjectMetaAccessor interface defines methods to obtain resource object accessors.

1
2
3
4
5
// source code from https://github.com/kubernetes/apimachinery/blob/master/pkg/apis/meta/v1/meta.go
//
type ObjectMetaAccessor interface {
    GetObjectMeta() Object
}

Since all single resource objects in kubernetes inherit from metav1.ObjectMeta, all API resource objects implement the metav1.Object and metav1.ObjectMetaAccessor interfaces. kubernetes has many places to access API resources There are many places in kubernetes to access the meta-information of API resource objects and they are accessible regardless of object type, as long as they are of the metav1.Object interface type.

metav1.ListMeta

metav1.ListMeta defines the public properties of all instances of the object list type.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
// source code from https://github.com/kubernetes/apimachinery/blob/master/pkg/apis/meta/v1/types.go
//
// ListMeta describes metadata that synthetic resources must have, including lists and
// various status objects. A resource may have only one of {ObjectMeta, ListMeta}.
type ListMeta struct {
    // selfLink is a URL representing this object.
    // Populated by the system.
    // Read-only.
    SelfLink string `json:"selfLink,omitempty" protobuf:"bytes,1,opt,name=selfLink"`

    // String that identifies the server's internal version of this object that
    // can be used by clients to determine when objects have changed.
    // Value must be treated as opaque by clients and passed unmodified back to the server.
    // Populated by the system.
    // Read-only.
    ResourceVersion string `json:"resourceVersion,omitempty" protobuf:"bytes,2,opt,name=resourceVersion"`

    // continue may be set if the user set a limit on the number of items returned, and indicates that
    // the server has more data available. The value is opaque and may be used to issue another request
    // to the endpoint that served this list to retrieve the next set of available objects. Continuing a
    // consistent list may not be possible if the server configuration has changed or more than a few
    // minutes have passed. The resourceVersion field returned when using this continue value will be
    // identical to the value in the first response, unless you have received this token from an error
    // message.
    Continue string `json:"continue,omitempty" protobuf:"bytes,3,opt,name=continue"`

    // remainingItemCount is the number of subsequent items in the list which are not included in this
    // list response. If the list request contained label or field selectors, then the number of
    // remaining items is unknown and the field will be left unset and omitted during serialization.
    // If the list is complete (either because it is not chunking or because this is the last chunk),
    // then there are no more remaining items and this field will be left unset and omitted during
    // serialization.
    // Servers older than v1.15 do not set this field.
    // The intended use of the remainingItemCount is *estimating* the size of a collection. Clients
    // should not rely on the remainingItemCount to be set or to be exact.
    RemainingItemCount *int64 `json:"remainingItemCount,omitempty" protobuf:"bytes,4,opt,name=remainingItemCount"`
}

Similar to the metav1.ObjectMeta structure, metav1.ListMeta also implements the metav1.ListInterface and metav1.ListMetaAccessor interfaces, where the metav1.ListInterface interface defines Get and Set methods for getting various meta-information about the list of resource objects.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
// source code from https://github.com/kubernetes/apimachinery/blob/master/pkg/apis/meta/v1/meta.go
//
// ListInterface lets you work with list metadata from any of the versioned or
// internal API objects. Attempting to set or retrieve a field on an object that does
// not support that field will be a no-op and return a default value.
type ListInterface interface {
    GetResourceVersion() string
    SetResourceVersion(version string)
    GetSelfLink() string
    SetSelfLink(selfLink string)
    GetContinue() string
    SetContinue(c string)
    GetRemainingItemCount() *int64
    SetRemainingItemCount(c *int64)
}

The metav1.ListMetaAccessor interface defines methods to get access to a list of resource objects.

1
2
3
4
5
6
// source code from https://github.com/kubernetes/apimachinery/blob/master/pkg/apis/meta/v1/meta.go
//
// ListMetaAccessor retrieves the list interface from an object
type ListMetaAccessor interface {
    GetListMeta() ListInterface
}

runtime.Object

When we introduced metav1.TypeMeta and metav1.ObjectMeta earlier, we found that schema.ObjectKind is the abstraction of all API resource types and metav1.Object is the abstraction of all API single resource object properties, so wouldn’t a type that implements both of these interfaces objects that implement both interfaces can access the public properties of any API object? Yes, for each particular type, such as Pod, Deployment, etc., they do have access to the public properties of the current API object. Is there a unified parent class of all specific types that has both schema.ObjectKind and metav1.Object interfaces so that it can represent any specific type of object. That is what this section will discuss with the runtime.Object interface.

Let’s first look at the definition of the runtime.Object interface.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
// source code from: https://github.com/kubernetes/apimachinery/blob/master/pkg/runtime/interfaces.go
//
// Object interface must be supported by all API types registered with Scheme. Since objects in a scheme are
// expected to be serialized to the wire, the interface an Object must provide to the Scheme allows
// serializers to set the kind, version, and group the object is represented as. An Object may choose
// to return a no-op ObjectKindAccessor in cases where it is not expected to be serialized.
type Object interface {
    // used to access type metadata(GVK)
    GetObjectKind() schema.ObjectKind
    // DeepCopyObject needed to implemented by each kubernetes API type definition,
    // usually by automatically generated code.
    DeepCopyObject() Object
}

Why does the runtime.Object interface only have these two methods? Shouldn’t there be a GetObjectMeta() method to get the metav1.Object object? If you look further down, you will see that a different implementation is used, with the Accessor method defined in k8s.io/apimachinery/pkg/api/meta as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
// source code from: https://github.com/kubernetes/apimachinery/blob/master/pkg/api/meta/meta.go
//
// Accessor takes an arbitrary object pointer and returns meta.Interface.
// obj must be a pointer to an API type. An error is returned if the minimum
// required fields are missing. Fields that are not required return the default
// value and are a no-op if set.
func Accessor(obj interface{}) (metav1.Object, error) {
    switch t := obj.(type) {
    case metav1.Object:
        return t, nil
    case metav1.ObjectMetaAccessor:
        if m := t.GetObjectMeta(); m != nil {
            return m, nil
        }
        return nil, errNotObject
    default:
        return nil, errNotObject
    }
}

Accessor method can convert any type of Golang object to metav1.Object or return an error message, thus avoiding the need to implement the GetObjectMeta() method for each API resource type.

Another question is why I don’t see the runtime.Object.DeepCopyObject() method implemented for the API resource type? That’s because the deep copy method is overloaded by the specific API resource type and has type dependencies that cannot be uniformly implemented by the parent class of the API resource type. In general, deep copy methods are automatically generated by tools and defined in the zz_generated.deepcopy.go file, take configMap as an example.

1
2
3
4
5
6
7
8
9
// source code from https://github.com/kubernetes/api/blob/master/core/v1/zz_generated.deepcopy.go
//
// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
func (in *ConfigMap) DeepCopyObject() runtime.Object {
    if c := in.DeepCopy(); c != nil {
        return c
    }
    return nil
}

unstructured.Unstructured

The runtime.Object interface is implemented mainly for structured objects like Pods, Services, Deployment, etc. Unlike structured objects of type runtimeO.bject, Kubernetes also provides unstructured.Unstructured unstructured objects.

Before we look at the unstructured.Unstructured source code implementation, let’s understand what structured and unstructured objects are. A structured object, as the name implies, is data in which the field names and field values are fixed. The Go structure of a Pod object can be represented as follows.

1
2
3
4
5
6
type Pod struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`
    Spec PodSpec `json:"spec,omitempty" protobuf:"bytes,2,opt,name=spec"`
    Status PodStatus `json:"status,omitempty" protobuf:"bytes,3,opt,name=status"`
}

We can get the corresponding field values from each field of the Pod structure, such as the image of the Pod runtime container, etc. However, the actual situation is that a lot of information about Pod objects is unknown at compile time and can only be obtained at runtime. How to deal with this situation? That’s where unstructured objects come in.

As anyone familiar with reflection will quickly recall, the Go language relies on reflection to dynamically fetch fields at runtime, unifying these unknown types as interface{} at compile time. For JSON/YAML-compatible deserialized objects, we can use map[string]interface{} to represent the fields. It follows that a Pod or any other Kubernetes API object that is unknown at compile time can also be represented as a Go structure as follows.

1
2
3
type k8sUnstructuredObject struct {
    Object map[string]interface{}
}

It is on this basis that the data structure of unstructured.Unstructured is defined very similarly.

1
2
3
4
5
6
7
// soure code from: https://github.com/kubernetes/apimachinery/blob/master/pkg/apis/meta/v1/unstructured/unstructured.go
//
type Unstructured struct {
    // Object is a JSON compatible map with string, float, int, bool, []interface{}, or
    // map[string]interface{} children.
    Object map[string]interface{}
}

Only the definition of the unstructured.Unstructured interface does not do much; what really matters are the methods implemented, which allow flexible handling of unstructured objects. unstructured.Unstructured implements methods for accessing type meta-information and object meta-information, in addition to Unstructured implements all the methods of the runtime.Unstructured interface.

Based on these methods, using unstructured.Unstructured unstructured decodes directly as YAML/JSON object representations, we can easily manipulate various types of kubernetes resources. A typical example is when we need to add the same tags to the various types of resources we create. If we use structured object types, we have to deal with different types, but with unstructured.Unstructured unstructured objects, we can do all the work in a single loop.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
import (
    "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
    "k8s.io/apimachinery/pkg/runtime/serializer/yaml"
)

var manifest = `
apiVersion: v1
kind: pod
metadata:
  name: web
spec:
  ...
`
// convert yaml to unstructured
objs := &unstructured.Unstructured{}
dec := yaml.NewDecodingSerializer(unstructured.UnstructuredJSONScheme)
dec.Decode([]byte(manifest), nil, obj)

labels := obj.GetLabels()
labels["owner"]="xxx"
obj.SetLabels(labels)

dynamicClient.Resource().Namespace(namespace).Create(context.TODO(), obj, metav1.CreateOptions{})

The various types of objects that implement the runtime.Object interface are strictly type-checked, and can be easily added, deleted, and checked by various clientset API resource objects, It is mainly used for static clients. For example, use static clients with structured objects to create Pod objects.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
import "k8s.io/client-go/kubernetes"

clientset, err := kubernetes.NewForConfig(config)

pod := &Corev1.Pod{}
pod.Name = "web"
pod.Spec = pod.PodSpec{
    ...
}

clientset.CoreV1().Pods(apiv1.NamespaceDefault).Create(pod)

For the same example, for dynamic clients using the unstructure.Unstructured type, we don’t even need to know how the Pod fields are defined, we just need to convert from YAML/JSON to unstructured.Unstructured objects or simply define map[string] interface{} to represent the Kubernetes API object.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
import "k8s.io/client-go/dynamic"

client, _ := dynamic.NewForConfig(config)
podGVR := schema.GroupVersionResource{Group: "core", Version: "v1", Resource: "pods"}

pod := &unstructured.Unstructured{
    Object: map[string]interface{}{
        "apiVersion": "v1",
        "kind":       "Pod",
        "metadata": map[string]interface{}{
            "name": "web",
        },
        "spec": map[string]interface{}{
            "serviceAccount": "default",
            ...
        }
    }
}

client.Resource(podGVR).Namespace(namespace).Create(context.TODO(), pod, metav1.CreateOptions{})

It is important to mention that both structured objects + static clients and unstructured objects + dynamic clients will eventually call the Kubernetes RESTful API.

In addition, Go’s reflection mechanism enables the conversion of unstructured.Unstructured objects to and from concrete resource objects. The runtime.DefaultUnstructuredConverter interface defines methods for converting unstructured.Unstructured objects to concrete resource objects, and the built-in runtime.DefaultUnstructuredConverter implements the runtime.unstructuredConverter interface.

1
2
3
4
5
6
7
8
// source code from: https://github.com/kubernetes/apimachinery/blob/master/pkg/runtime/converter.go
//
// UnstructuredConverter is an interface for converting between interface{}
// and map[string]interface representation.
type UnstructuredConverter interface {
    ToUnstructured(obj interface{}) (map[string]interface{}, error)
    FromUnstructured(u map[string]interface{}, obj interface{}) error
}

Summary

To make it easier to remember, a short summary of the various interfaces and implementations introduced earlier is now available.

k8s api

  1. the runtime.Object interface is the root class of all API monolithic resource objects, and the encoding and decoding of each API object depends on this interface type.
  2. the schema.ObjectKind interface is an abstraction of the API resource object type that can be used to get or set the GVK.
  3. the metav1.Object interface is an abstraction of the API resource object properties and can be used to access the properties of the resource object.
  4. the metav1.ListInterface interface is an abstraction of the API object list properties and is used to access the properties of the resource object list.
  5. the metav1.TypeMeta structure implements the schema.ObjectKind interface, which is inherited by all API resource types.
  6. the metav1.ObjectMeta structure implements the metav1.Object interface, which all API resource types inherit from.
  7. the metav1.ListMeta structure implements the metav1.ListInterface interface, which is inherited by all API resource object list types.
  8. the unstructured.Unstructured structure implements the runtime.Unstructured interface and can be used in conjunction with dynamic clients to access resource objects. from instances of unstructured.Unstructured you can get resource type meta information and resource object meta information, as well as the generic content representation of the map[string]interface{} of the object.
  9. unstructured.Unstructured interconverts with concrete API types that implement the metav1.Object interface, and the conversion relies on the interface methods of runtime.UnstructuredConverter.