Crossplane is an open source Kubernetes plugin that solves the problem of provisioning cloud resources by extending the Kubernetes API. When using Crossplane, you can define declaratively what cloud resources need to be created for your application to work properly, without writing any code. The definition of these cloud resources can be done directly by creating the associated CRD objects, which can be thought of as a cloud-native version of Terraform.

Crossplane

VCluster is a tool that provides flexibility and cost savings through lightweight virtual Kubernetes clusters. Using VCluster, you can create an isolated virtual Kubernetes cluster within a Kubernetes cluster. This greatly reduces the complexity of creating and maintaining a Kubernetes cluster control plane.

VCluster

The following table compares the isolation levels and management complexity of using namespaces, vcluster, and Kubernetes clusters.

table

So what happens when we combine the two tools Crossplane and VCluster together? We will use an example to illustrate the use of these two tools together.

Example

In this example we want to implement some functionality as shown below.

  • Have a cluster that can receive requests to start a new cluster environment
  • These environments will be able to use Helm to install applications
  • The team requesting the new environment doesn’t care where the cluster was created, so using VCluster or creating a Kubernetes cluster in a cloud provider should provide a similar experience for end users.

Here I’m using KinD in my local environment for this demonstration, and a list of relevant resources can be found here https://github.com/salaboy/from-monolith-to-k8s/tree/main/platform/crossplane-vcluster (you need to install them yourself in advance) kubectl, helm, kind).

Example

As shown above, we just need to create a KinD cluster (or of course any other Kuberentes cluster) and install Crossplane and Crossplane Helm Provider on the cluster, since we are not creating any cloud resources here, we don’t need to configure any other Crossplane Provider (e.g. GCP, AWS, Azure, etc.).

Installing Crossplane

Next we can start by creating a Kubernetes cluster using KinD.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

$ kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.23.4) đŸ–ŧ
 ✓ Preparing nodes đŸ“Ļ
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹ī¸
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂
$ kubectl get nodes
NAME                 STATUS   ROLES                  AGE   VERSION
kind-control-plane   Ready    control-plane,master   56s   v1.23.4

Once the cluster is ready we can then install Crossplane and Crossplane Helm Provider into our KinD cluster as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

$ kubectl create ns crossplane-system
namespace/crossplane-system created
$ helm install crossplane --namespace crossplane-system crossplane-stable/crossplane

NAME: crossplane
LAST DEPLOYED: Tue Aug  9 15:20:22 2022
NAMESPACE: crossplane-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Release: crossplane

Chart Name: crossplane
Chart Description: Crossplane is an open source Kubernetes add-on that enables platform teams to assemble infrastructure from multiple vendors, and expose higher level self-service APIs for application teams to consume.
Chart Version: 1.9.0
Chart Application Version: 1.9.0

Kube Version: v1.23.4

After installation, two Pods will be run under the crossplane-system namespace as follows.

1
2
3
4

$ kubectl get pods -n crossplane-system
NAME                                       READY   STATUS    RESTARTS   AGE
crossplane-c9b9fc9f9-4hn47                 1/1     Running   0          11m
crossplane-rbac-manager-56c8ff5b65-8lgrp   1/1     Running   0          11m

Then you need to install Crossplane Helm Provider, which can be done directly using the kubectl plugin for crossplane.

1
2

$ kubectl crossplane install provider crossplane/provider-helm:v0.10.0
provider.pkg.crossplane.io/crossplane-provider-helm created

Also note that when installing the Crossplane Helm Provider, we need to provide an appropriate ServiceAccount for the Provider to create a new ClusterRoleBinding so that the Provider can install Helm Charts.

1
2
3
4
5

$ SA=$(kubectl -n crossplane-system get sa -o name | grep provider-helm | sed -e 's|serviceaccount\/|crossplane-system:|g')
$ echo $SA
crossplane-system:crossplane-provider-helm-3d2f09bcd965
$ kubectl create clusterrolebinding provider-helm-admin-binding --clusterrole cluster-admin --serviceaccount="${SA}"
clusterrolebinding.rbac.authorization.k8s.io/provider-helm-admin-binding created

Then create a ProviderConfig object, as shown below, to declare the installation of the Helm Provider.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# helm-provider-config.yaml
apiVersion: helm.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: InjectedIdentity

---
# SA=$(kubectl -n crossplane-system get sa -o name | grep provider-helm | sed -e 's|serviceaccount\/|crossplane-system:|g')
# kubectl create clusterrolebinding provider-helm-admin-binding --clusterrole cluster-admin --serviceaccount="${SA}"

Just apply the list of resources above directly.

1
2

$ kubectl apply -f helm-provider-config.yaml
providerconfig.helm.crossplane.io/default created

At this point we have completed the installation of Crossplane. Finally, we also recommend installing the vcluster command line tool to connect to the Kubernetes cluster, which can be installed by referring to the documentation at https://www.vcluster.com/docs/getting-started/setup.

Creating VClusters with Crossplane Composition

Now that Crossplane and Crossplane Helm Provider are ready, let’s take a look at Crossplane Composition. Crossplane provides a mechanism for composing managed resources in which users can create their own abstractions in a declarative way.

  • Composite Resource: A Composite Resource (XR) is a custom resource that consists of managed resources that allow you to abstract infrastructure details. The CompositeResourceDefinition (XRD) defines a new type of combined resource, XRDs are cluster-wide, and in order to create a namespace XR, the corresponding XRD can provide a combined resource declaration (XRC).
  • Composition: A composition specifies what resources an XR will consist of, that is, what happens when you create an XR, and an XR can have multiple compositions. For example, for CompositeDatabase XR, you could use a combination to create an AWS RDS instance, a security group, and a MySQL database. Another combination can define a GCP CloudSQL instance and a PostgreSQL database.
  • Configuration: A configuration is a package of XRDs and combinations that can then be published to the OCI image registry using the Crossplane CLI and installed into a Crossplane cluster by creating declarative configuration resources.

Our Crossplane Composition(XR) here defines the related operations that need to be performed when creating a new environment resource, this object will perform some of the following operations.

  • Install the VCluster Helm Chart using the Helm Provider Config we configured when we installed the Helm Provider, and when we install this Chart, we can create a new VCluster, isn’t it very simple.
  • The VCluster installation creates a Kubernetes Secret object containing tokens connected to the VCluster APIServer, which we can use to configure a second Helm Provider Config, which allows us to install Helm Charts into the newly created cluster in the newly created cluster.
  • We can use the second Helm Provider Config to install the application into the created VCluster.

VCluster

Let’s look at how this is done. First we need to apply Crossplane Composition and Environment CRD to our cluster so that we can create new Environment resources.

First define an Environment composition resource, corresponding to the list of resources shown below, which is the equivalent of a CRD in a Kubernetes cluster.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

# environment-resource-definition.yaml
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: environments.fmtok8s.salaboy.com
spec:
  group: fmtok8s.salaboy.com
  names:
    kind: Environment
    plural: environments
  claimNames:
    kind: Cluster
    plural: clusters
  versions:
  - name: v1alpha1
    served: true
    referenceable: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties: {}

With the XRD composition resource declaration in place, the next step is to define a composition object, as shown in the resource manifest file below, in which multiple resources are defined in the Composition composition object, which is associated with the Environment object declared in the XRD object above.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141

# composition.yaml
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: environment.fmtok8s.salaboy.com
spec:
  writeConnectionSecretsToNamespace: crossplane-system
  compositeTypeRef:
    apiVersion: fmtok8s.salaboy.com/v1alpha1
    kind: Environment
  resources:
    - name: vcluster-helm-release
      base:
        apiVersion: helm.crossplane.io/v1beta1
        kind: Release
        metadata:
          annotations:
            crossplane.io/external-name: # patched
        spec:
          rollbackLimit: 3
          forProvider:
            namespace: # patched
            chart:
              name: vcluster
              repository: https://charts.loft.sh
              version: "0.10.2"
            values: 
              syncer:
                extraArgs: [] # patched
              #     - --out-kube-config-server=https://cluster-1.cluster-1.svc
          providerConfigRef:
            name: default
      patches:
        - fromFieldPath: metadata.name
          toFieldPath: spec.forProvider.namespace
          policy:
            fromFieldPath: Required
        - fromFieldPath: metadata.name
          toFieldPath: metadata.annotations[crossplane.io/external-name]
          policy:
            fromFieldPath: Required
        - fromFieldPath: metadata.name
          toFieldPath: metadata.name
          transforms:
            - type: string
              string:
                fmt: "%s-vcluster"
        - type: CombineFromComposite
          combine:
            variables:
            - fromFieldPath: metadata.name
            strategy: string
            string:
              fmt: "--out-kube-config-secret=%s-secret"
          toFieldPath: spec.forProvider.values.syncer.extraArgs[0]
        - type: CombineFromComposite
          combine:
            variables:
            - fromFieldPath: metadata.name
            - fromFieldPath: metadata.name
            strategy: string
            string:
              fmt: "--out-kube-config-server=https://%s.%s.svc"
          toFieldPath: spec.forProvider.values.syncer.extraArgs[1]
        - type: CombineFromComposite
          combine:
            variables:
            - fromFieldPath: metadata.name
            - fromFieldPath: metadata.name
            strategy: string
            string:
              fmt: "--tls-san=%s.%s.svc"
          toFieldPath: spec.forProvider.values.syncer.extraArgs[2]    
    - name: helm-providerconfig
      base:
        apiVersion: helm.crossplane.io/v1alpha1
        kind: ProviderConfig
        spec:
          credentials:
            source: Secret
            secretRef:
              name: # patched
              namespace: # patched
              key: config
      patches:
        - fromFieldPath: metadata.name
          toFieldPath: spec.credentials.secretRef.name
          transforms:
            - type: string
              string:
                fmt: vc-%s
        - fromFieldPath: metadata.name
          toFieldPath: spec.credentials.secretRef.namespace
        - fromFieldPath: metadata.uid
          toFieldPath: metadata.name     
    - name: helm-provider-vcluster
      base:
        apiVersion: helm.crossplane.io/v1beta1
        kind: ProviderConfig
        spec:
          credentials:
            source: Secret
            secretRef:
              namespace: #patched
              key: config
      patches:
        - fromFieldPath: metadata.name
          toFieldPath: metadata.name
        - fromFieldPath: metadata.name
          toFieldPath: spec.credentials.secretRef.namespace
          policy:
            fromFieldPath: Required  
        # This ProviderConfig uses the above VCluster's connection secret as
        # its credentials secret.
        - fromFieldPath: "metadata.name"
          toFieldPath: spec.credentials.secretRef.name
          transforms:
            - type: string
              string:
                fmt: "%s-secret"
      readinessChecks:
        - type: None      
    - name: conference-chart-vcluster
      base:
        apiVersion: helm.crossplane.io/v1beta1
        kind: Release
        metadata:
          annotations: 
            crossplane.io/external-name: conference
        spec:
          forProvider:
            chart:
              name: fmtok8s-conference-chart
              repository: https://salaboy.github.io/helm/
              version: "v0.1.1"
            namespace: conference
          providerConfigRef: 
            name: #patched
      patches:
        - fromFieldPath: metadata.name
          toFieldPath: spec.providerConfigRef.name

We set 3 parameters for VCluster’s Chart package to be used with Crossplane.

  • Configured fmt:"--out-kube-config-secret=%s-secret" on line 53, because we need VCluster to create a Secret object to host kubeconfig in so we can get it to connect with the newly created APIServer.
  • Configure fmt:"--out-kube-config-server=https://%s.%s.svc" on line 62, because we need kubeconfig to point to the new APIServer URL from within the cluster, and by default, the generated kubeconfig points to https:/ /localhost:8443.
  • The fmt:"--tls-san=%s.%s.svc" is configured on line 71, indicating that the new service address needs to be added to the list of hosts to which the APIServer accepts connections.

Then just apply the two objects above directly.

1
2
3
4

$ kubectl apply -f composition.yaml
composition.apiextensions.crossplane.io/environment.fmtok8s.salaboy.com created
$ kubectl apply -f environment-resource-definition.yaml
compositeresourcedefinition.apiextensions.crossplane.io/environments.fmtok8s.salaboy.com created

Once the portfolio and CRD are available within the cluster, we can start creating new environment resources and check which VClusters are currently available before running them.

1
2
3
4

$ vcluster list

 NAME   NAMESPACE   STATUS   CONNECTED   CREATED   AGE
 No entries found

There shouldn’t be any VClusters yet, now we can create a new environment, for example now we define a dev environment by simply declaring an Environment object as shown below.

1
2
3
4
5
6

# environment-resource.yaml
apiVersion: fmtok8s.salaboy.com/v1alpha1
kind: Environment
metadata:
  name: dev-environment
spec: {}

Apply the object directly.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

$ kubectl apply -f environment-resource.yaml
environment.fmtok8s.salaboy.com/dev-environment created
$ kubectl get environments
NAME              READY   COMPOSITION                       AGE
dev-environment   False   environment.fmtok8s.salaboy.com   57s
$ kubectl describe environments dev-environment
Name:         dev-environment
Namespace:
Labels:       crossplane.io/composite=dev-environment
Annotations:  <none>
API Version:  fmtok8s.salaboy.com/v1alpha1
Kind:         Environment
......
Events:
  Type    Reason                   Age                From                                                             Message
  ----    ------                   ----               ----                                                             -------
  Normal  PublishConnectionSecret  76s                defined/compositeresourcedefinition.apiextensions.crossplane.io  Successfully published connection details
  Normal  SelectComposition        19s (x7 over 76s)  defined/compositeresourcedefinition.apiextensions.crossplane.io  Successfully selected composition
  Normal  ComposeResources         18s (x7 over 76s)  defined/compositeresourcedefinition.apiextensions.crossplane.io  Successfully composed resources

Now we can treat the environment resources you created like any other Kubernetes resource, you can list them directly using kubectl get environments and even describe them to see more details.

Now let’s go back and check the VCluster normal and we’ll find a new resource.

1
2
3
4

 vcluster list

 NAME              NAMESPACE         STATUS    CONNECTED   CREATED                         AGE
 dev-environment   dev-environment   Running               2022-08-09 17:44:07 +0800 CST   56m38s

VCluster will install an APIServer (using K3s by default), CoreDNS instances, and a Syncer in a new namespace, dev-environment, allowing users to interact with the VCluster API Server via kubectl, just as they would with a regular cluster. VCluster will synchronize these resources with the cluster of hosts responsible for scheduling the workload, thus enabling the functionality of a namespace as a Kubernetes cluster.

Once we have configured Crossplane and the Crossplane Helm Provider, we can create a new VCluster by creating a new Helm Release and installing a Helm Chart, which is very simple.

VCluster

Once we have created the VCluster using the correct kubeconfig created in secret, we can configure a second Helm Provider to install our application into the newly created VCluster, as defined in line 95 of the composition object above, helm-provider- vcluster defined in line 95 of the composition object above is that description.

Then inside the composition, we configure the Helm Chart package that uses one of our meeting applications.

VCluster

After configuring everything and creating a new environment, we can connect to VCluster and check if the application is installed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

$ vcluster connect dev-environment --server https://localhost:8443 -- bash

The default interactive shell is now zsh.
To update your account to use zsh, please run `chsh -s /bin/zsh`.
For more details, please visit https://support.apple.com/kb/HT208050.
bash-3.2$ kubectl get ns
NAME              STATUS   AGE
default           Active   32m
kube-system       Active   32m
kube-public       Active   32m
kube-node-lease   Active   32m
conference        Active   23m
bash-3.2$ kubectl get pods -n conference
NAME                                                 READY   STATUS    RESTARTS      AGE
conference-fmtok8s-frontend-7cd5db8669-pv944         1/1     Running   0             23m
conference-fmtok8s-email-service-768bc88cbb-sklrg    1/1     Running   0             23m
conference-postgresql-0                              1/1     Running   0             23m
conference-fmtok8s-c4p-service-7f56d7bd9d-2vjtx      1/1     Running   2 (19m ago)   23m
conference-redis-master-0                            1/1     Running   0             23m
conference-redis-replicas-0                          1/1     Running   0             23m
conference-fmtok8s-agenda-service-7db66c9568-xsh5m   1/1     Running   2 (16m ago)   23m

You can see that our applications have been successfully installed in that cluster, and they are actually deployed under the dev-environment namespace of the original KinD cluster.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

$ kubectl get pods -n dev-environment
NAME                                                              READY   STATUS    RESTARTS      AGE
conference-fmtok8s-agenda-service-7db66c9568-xsh5m-x-08f9332627   1/1     Running   2 (18m ago)   25m
conference-fmtok8s-c4p-service-7f56d7bd9d-2vjtx-x-co-fc2c58eaec   1/1     Running   2 (21m ago)   25m
conference-fmtok8s-email-service-768bc88cbb-sklrg-x--c5d9594434   1/1     Running   0             25m
conference-fmtok8s-frontend-7cd5db8669-pv944-x-confe-2832ac1bef   1/1     Running   0             25m
conference-postgresql-0-x-conference-x-dev-environment            1/1     Running   0             25m
conference-redis-master-0-x-conference-x-dev-environment          1/1     Running   0             25m
conference-redis-replicas-0-x-conference-x-dev-environment        1/1     Running   0             25m
coredns-76dd5485df-6cbl7-x-kube-system-x-dev-environment          1/1     Running   0             34m
dev-environment-0                                                 2/2     Running   0             63m

It’s just that VCluster isolates a namespace and is essentially the same experience as a Kubernetes cluster.

Summary

This is just a simple example of how to use Crossplane in combination with VCluster to quickly configure a Kubernetes clustered environment and install applications in it to make developers more productive. Of course there are many areas that can be optimized, such as.

  • Installing ArgoCD in VCluster and using the GitHub URL provided as an environment parameter to implement GitOps, which will avoid using kubectl for VCluster. using composition can be used to create ArgoCD resources to configure the repository and cluster without user intervention.
  • Install Knative in VCluster so that developers can rely on Knative Functions, Knative Serving, and Eventing features to design their applications.
  • Environment parameters to determine which cloud resources VCluster uses, such as GCP, AKS and EKS implementations, etc.
  • The same way Crossplane is used for testing in local KinD clusters, we can also interface to real cloud resources such as GCP, AWS, Azure, etc.