This article mainly deploys v1.23.6 version of k8s native cluster based on docker and flannel components on centos7 system, because the cluster is mainly used for own learning and testing, plus limited resources, not involved in high availability deployment for now.

1. Preparation

1.1 flannel-cluster node information

The machines are all 8C8G virtual machines with 100G hard disk.

IP Hostname
10.31.8.1 tiny-flannel-master-8-1.k8s.tcinternal
10.31.8.11 tiny-flannel-worker-8-11.k8s.tcinternal
10.31.8.12 tiny-flannel-worker-8-12.k8s.tcinternal
10.8.64.0/18 podSubnet
10.8.0.0/18 serviceSubnet

1.2 Checking mac and product_uuid

All nodes in the same k8s cluster need to make sure that both mac address and product_uuid are unique, so you need to check the relevant information before starting cluster initialization.

1
2
3
4
5
6

# Check mac address
ip link 
ifconfig -a

# Check product_uuid
sudo cat /sys/class/dmi/id/product_uuid

1.3 Configure ssh password-free login (optional)

If the nodes of the k8s cluster have multiple NICs, ensure that each node can be accessed through the correct NIC interconnect.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

# Generate a public key under the root user, and configure the key to be used for password-free login
su root
ssh-keygen
cd /root/.ssh/
cat id_rsa.pub >> authorized_keys
chmod 600 authorized_keys


cat >> ~/.ssh/config <<EOF
Host tiny-flannel-master-8-1.k8s.tcinternal
    HostName 10.31.8.1
    User root
    Port 22
    IdentityFile ~/.ssh/id_rsa

Host tiny-flannel-worker-8-11.k8s.tcinternal
    HostName 10.31.8.11
    User root
    Port 22
    IdentityFile ~/.ssh/id_rsa

Host tiny-flannel-worker-8-12.k8s.tcinternal
    HostName 10.31.8.12
    User root
    Port 22
    IdentityFile ~/.ssh/id_rsa
EOF

1.4 Modify the hosts file

1
2
3
4
5

cat >> /etc/hosts <<EOF
10.31.8.1  tiny-flannel-master-8-1 tiny-flannel-master-8-1.k8s.tcinternal
10.31.8.11 tiny-flannel-worker-8-11 tiny-flannel-worker-8-11.k8s.tcinternal
10.31.8.12 tiny-flannel-worker-8-12 tiny-flannel-worker-8-12.k8s.tcinternal
EOF

1.5 Turn off swap memory

1
2
3
4

# Use the command to turn off swap memory directly
swapoff -a
# Modify fstab file to disable automatic mounting of swap partition on boot
sed -i '/swap / s/^\(.*\)$/#\1/g' /etc/fstab

1.6 Configure time synchronization

Here you can choose either ntp or chrony synchronization according to your custom, and the synchronized time source server can choose Aliyun’s ntp1.aliyun.com or National Time Center’s ntp.ntsc.ac.cn.

Use ntp to synchronize

1
2
3
4
5
6
7
8

# Install ntpdate tool using yum
yum install ntpdate -y

# Synchronize time using the source of the national time center
ntpdate ntp.ntsc.ac.cn

# Check the time at last
hwclock

Synchronize with chrony

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

# Install chrony with yum
yum install chrony -y

# Set boot up and turn on chony and check running status
systemctl enable chronyd.service
systemctl start chronyd.service
systemctl status chronyd.service

# Of course you can also customize the time server
vim /etc/chrony.conf

# Before modification
$ grep server /etc/chrony.conf
# Use public servers from the pool.ntp.org project.
server 0.centos.pool.ntp.org iburst
server 1.centos.pool.ntp.org iburst
server 2.centos.pool.ntp.org iburst
server 3.centos.pool.ntp.org iburst

# After modification
$ grep server /etc/chrony.conf
# Use public servers from the pool.ntp.org project.
server ntp.ntsc.ac.cn iburst

# Restart the service to make the configuration file take effect
systemctl restart chronyd.service

# View chrony's ntp server status
chronyc sourcestats -v
chronyc sources -v

1.7 Shutting down selinux

1
2
3
4
5

# Close directly using the command
setenforce 0

# You can also modify the /etc/selinux/config file directly
sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config

1.8 Configuring Firewalls

Communication and service exposure between k8s clusters requires the use of more ports, so for convenience, disable the firewall directly.

1
2

# centos7 use systemctl to disable the default firewalld service
systemctl disable firewalld.service

1.9 Configuring netfilter parameters

The main thing here is to configure the kernel to load br_netfilter and iptables to release ipv6 and ipv4 traffic to ensure that the containers in the cluster can communicate properly.

1
2
3
4
5
6
7
8
9

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system

1.10 Turn off IPV6 (optional)

Although newer versions of k8s already support dual-stack networks, this cluster deployment process does not involve communication over IPv6 networks, so turn off IPv6 network support.

1
2

# Add the ipv6 disable parameter directly to the kernel
grubby --update-kernel=ALL --args=ipv6.disable=1

1.11 Configuring IPVS (optional)

IPVS is a component specifically designed to cope with load balancing scenarios. IPVS implementation in kube-proxy increases scalability by reducing the use of iptables. Instead of using PREROUTING in the iptables input chain, a dummy interface is created called kube-ipvs0, which enables IPVS to achieve more efficient forwarding performance than iptables when the load balancing configuration in a k8s cluster becomes more numerous.

Notes : use nf_conntrack instead of nf_conntrack_ipv4 for Linux kernel 4.19 and later.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

# Make sure ipset and ipvsadm are installed before using ipvs mode
sudo yum install ipset ipvsadm -y

# Manually load ipvs related modules
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4

# Configure boot up to automatically load ipvs related modules
cat <<EOF | sudo tee /etc/modules-load.d/ipvs.conf
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack_ipv4
EOF

sudo sysctl --system
# It's best to reboot the system to make sure it works

$ lsmod | grep -e ip_vs -e nf_conntrack_ipv4
ip_vs_sh               12688  0
ip_vs_wrr              12697  0
ip_vs_rr               12600  0
ip_vs                 145458  6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack_ipv4      15053  2
nf_defrag_ipv4         12729  1 nf_conntrack_ipv4
nf_conntrack          139264  7 ip_vs,nf_nat,nf_nat_ipv4,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_ipv4
libcrc32c              12644  4 xfs,ip_vs,nf_nat,nf_conntrack
$ cut -f1 -d " "  /proc/modules | grep -e ip_vs -e nf_conntrack_ipv4
ip_vs_sh
ip_vs_wrr
ip_vs_rr
ip_vs
nf_conntrack_ipv4

2. Install container runtime

2.1 Installing docker

Detailed official documentation can be found here, as docker-shim was removed in the just released version 1.24, so the installation of version ≥ 1.24 needs to pay attention to the container runtime selection. Here we have installed a version lower than 1.24, so we continue to use docker.

1
2
3
4
5
6

# Install the necessary dependencies and import the official docker yum source
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
sudo yum-config-manager --add-repo  https://download.docker.com/linux/centos/docker-ce.repo

# We install the latest version of docker directly
yum install docker-ce docker-ce-cli containerd.io

2.2 Configuring cgroup drivers

CentOS 7 uses systemd to initialize the system and manage processes. Initializing processes generates and uses a root control group (cgroup), and acts as a cgroup manager. Systemd is tightly integrated with cgroup and will assign a cgroup to each systemd unit. We can also configure the container runtime and kubelet to use cgroupfs. Using cgroupfs with systemd means that there will be two different cgroup managers. When both cgroupfs and systemd are present on a system, it tends to become unstable, so it is best to change the settings so that the container runtime and kubelet use systemd as the cgroup driver to make the system more stable. For Docker, you need to set the native.cgroupdriver=systemd parameter.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

sudo mkdir /etc/docker
cat <<EOF | sudo tee /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

sudo systemctl enable docker
sudo systemctl daemon-reload
sudo systemctl restart docker


# Lastly, check if the Cgroup Driver is systemd
$ docker info | grep systemd
 Cgroup Driver: systemd

2.3 About kubelet’s cgroup driver

k8s has detailed documentation on how to set kubelet’s cgroup driver. Note in particular that starting with version 1.22, if the kubelet cgroup driver is not set manually, it will be set to systemd by default.

Note: In v1.22, if the user is not setting the cgroupDriver field under KubeletConfiguration , kubeadm will default it to systemd .

A simpler way to specify a cgroup driver for a kubelet is to add a cgroupDriver field to kubeadm-config.yaml.

1
2
3
4
5
6
7
8

# kubeadm-config.yaml
kind: ClusterConfiguration
apiVersion: kubeadm.k8s.io/v1beta3
kubernetesVersion: v1.21.0
---
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemd

We can check the configmaps directly to see the kubeadm-config configuration of the cluster after initialization.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

$ kubectl describe configmaps kubeadm-config -n kube-system
Name:         kubeadm-config
Namespace:    kube-system
Labels:       <none>
Annotations:  <none>

Data
====
ClusterConfiguration:
----
apiServer:
  extraArgs:
    authorization-mode: Node,RBAC
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.23.6
networking:
  dnsDomain: cali-cluster.tclocal
  serviceSubnet: 10.88.0.0/18
scheduler: {}


BinaryData
====

Events:  <none>

Of course, since we need to install a version higher than 1.22.0 and use systemd, we don’t need to repeat the configuration.

3. Installing the kube triplet

Reference: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl

The kube triplet is kubeadm, kubelet and kubectl, the specific functions and roles of the three are as follows.

  • kubeadm: the command used to initialize the cluster.
  • kubelet: used on each node in the cluster to start Pods, containers, etc.
  • kubectl: Command line tool used to communicate with the cluster.

Some points to note are as follows.

  • kubeadm will not help us manage kubelet and kubectl, and the other two are the same, which means that the three are independent of each other and there is no case of who manages who.
  • The version of kubelet must be less than or equal to the version of API-server, otherwise compatibility issues are likely to arise.
  • kubectl does not need to be installed on every node in the cluster, nor does it have to be installed on a node in the cluster, it can be installed separately on top of your own local machine environment, and then with the kubeconfig file you can use the kubectl command to remotely manage the corresponding k8s cluster.

The installation of CentOS 7 is relatively simple, we can just use the official yum source. Note that you need to set the state of selinux here, but we have already turned off selinux, so we will skip this step.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

# Import the official Google yum sources directly
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

# Of course, if you can't connect to Google's source, you can consider using the domestic Ali mirror source
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF


# The next step is to install the kubectl triad directly
sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

# If the network environment is bad and the gpgcheck verification fails to read the yum source properly, consider disabling the repo_gpgcheck of the yum source
sed -i 's/repo_gpgcheck=1/repo_gpgcheck=0/g' /etc/yum.repos.d/kubernetes.repo
# Or disable gpgcheck during installation
sudo yum install -y kubelet kubeadm kubectl --nogpgcheck --disableexcludes=kubernetes



# If you want to install a specific version, you can use this command to view the relevant version information
sudo yum list --nogpgcheck kubelet kubeadm kubectl --showduplicates --disableexcludes=kubernetes
# Here we use docker-shim in order to preserve the use of docker-shim, so we follow the previous version 1.23.6 from version 1.24.0
sudo yum install -y kubelet-1.23.6-0 kubeadm-1.23.6-0 kubectl-1.23.6-0 --nogpgcheck --disableexcludes=kubernetes

# Configure boot-up kubelet after installation
sudo systemctl enable --now kubelet

4. Initialize the cluster

4.1 Writing the configuration file

After all the nodes in the cluster have performed the above three operations, we can start creating the k8s cluster. Since we are not involved in a high availability deployment this time, we can operate directly on top of our target master node during initialization.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# We'll first use the kubeadm command to check out the major image versions
# Since we previously specified an old 1.23.6 installation, the apiserver image version here will be rolled back as well
$ kubeadm config images list
I0507 14:14:34.992275   20038 version.go:255] remote version is much newer: v1.24.0; falling back to: stable-1.23
k8s.gcr.io/kube-apiserver:v1.23.6
k8s.gcr.io/kube-controller-manager:v1.23.6
k8s.gcr.io/kube-scheduler:v1.23.6
k8s.gcr.io/kube-proxy:v1.23.6
k8s.gcr.io/pause:3.6
k8s.gcr.io/etcd:3.5.1-0
k8s.gcr.io/coredns/coredns:v1.8.6

# To facilitate editing and management, we still export the initialization parameters to a configuration file
$ kubeadm config print init-defaults > kubeadm-flannel.conf
  • Considering that in most cases domestic(China) networks cannot use Google’s k8s.gcr.io image source, we can directly modify the imageRepository parameter in the configuration file to be Ali’s image source
  • kubernetesVersion field to specify the version of k8s we want to install
  • localAPIEndpoint parameter needs to be modified to the IP and port of our master node, which is the apiserver address of the k8s cluster after initialization
  • serviceSubnet and dnsDomain parameters can be changed by default, here I changed them according to my needs
  • The name parameter in nodeRegistration is changed to hostname of the corresponding master node
  • The new configuration block uses ipvs, which can be found in the official documentation
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 10.31.8.1
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  imagePullPolicy: IfNotPresent
  name: tiny-flannel-master-8-1.k8s.tcinternal
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: 1.23.6
networking:
  dnsDomain: flan-cluster.tclocal
  serviceSubnet: 10.8.0.0/18
  podSubnet: 10.8.64.0/18
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs

4.2 Initialize the cluster

At this point we check the mirror version in the corresponding configuration file, we will find that it has become the version corresponding to the AliCloud mirror source.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

# Check the corresponding image version to make sure the configuration file is valid
$ kubeadm config images list --config kubeadm-flannel.conf
registry.aliyuncs.com/google_containers/kube-apiserver:v1.23.6
registry.aliyuncs.com/google_containers/kube-controller-manager:v1.23.6
registry.aliyuncs.com/google_containers/kube-scheduler:v1.23.6
registry.aliyuncs.com/google_containers/kube-proxy:v1.23.6
registry.aliyuncs.com/google_containers/pause:3.6
registry.aliyuncs.com/google_containers/etcd:3.5.1-0
registry.aliyuncs.com/google_containers/coredns:v1.8.6

# After confirming that there is no problem we pull the image directly
$ kubeadm config images pull --config kubeadm-flannel.conf
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.23.6
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.23.6
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.23.6
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.23.6
[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.6
[config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.5.1-0
[config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.8.6

# Initialization
$ kubeadm init --config kubeadm-flannel.conf
[init] Using Kubernetes version: v1.23.6
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
... A bunch of output is omitted here...

When we see this output below, our cluster has been initialized successfully.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.31.8.1:6443 --token abcdef.0123456789abcdef \
        --discovery-token-ca-cert-hash sha256:d7160866920c0331731ad3c1c31a6e5b6c788b5682f86971cacaa940211db9ab

4.3 Configuring kubeconfig

After successful initialization, we can’t view the k8s cluster information right away. We need to configure kubeconfig related parameters in order to properly use kubectl to connect to apiserver to read cluster information.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# For non-root users, you can do this
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# If you are root user, you can import environment variables directly
export KUBECONFIG=/etc/kubernetes/admin.conf

# Add kubectl's auto-completion feature
echo "source <(kubectl completion bash)" >> ~/.bashrc

As we mentioned earlier, kubectl does not have to be installed in the cluster. In fact, you can install kubectl on any machine that can connect to the apiserver and configure kubeconfig according to the steps to use the kubectl command line to manage the corresponding k8s cluster.

Once the configuration is complete, we can then execute the relevant commands to view the information about the cluster.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

$ kubectl cluster-info
Kubernetes control plane is running at https://10.31.8.1:6443
CoreDNS is running at https://10.31.8.1:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

$ kubectl get nodes -o wide
NAME                                     STATUS     ROLES                  AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION                CONTAINER-RUNTIME
tiny-flannel-master-8-1.k8s.tcinternal   NotReady   control-plane,master   79s   v1.23.6   10.31.8.1     <none>        CentOS Linux 7 (Core)   3.10.0-1160.62.1.el7.x86_64   docker://20.10.14

$ kubectl get pods -A -o wide
NAMESPACE     NAME                                                             READY   STATUS    RESTARTS   AGE   IP          NODE                                     NOMINATED NODE   READINESS GATES
kube-system   coredns-6d8c4cb4d-2clkj                                          0/1     Pending   0          86s   <none>      <none>                                   <none>           <none>
kube-system   coredns-6d8c4cb4d-8mznz                                          0/1     Pending   0          86s   <none>      <none>                                   <none>           <none>
kube-system   etcd-tiny-flannel-master-8-1.k8s.tcinternal                      1/1     Running   0          91s   10.31.8.1   tiny-flannel-master-8-1.k8s.tcinternal   <none>           <none>
kube-system   kube-apiserver-tiny-flannel-master-8-1.k8s.tcinternal            1/1     Running   0          92s   10.31.8.1   tiny-flannel-master-8-1.k8s.tcinternal   <none>           <none>
kube-system   kube-controller-manager-tiny-flannel-master-8-1.k8s.tcinternal   1/1     Running   0          90s   10.31.8.1   tiny-flannel-master-8-1.k8s.tcinternal   <none>           <none>
kube-system   kube-proxy-dkvrn                                                 1/1     Running   0          86s   10.31.8.1   tiny-flannel-master-8-1.k8s.tcinternal   <none>           <none>
kube-system   kube-scheduler-tiny-flannel-master-8-1.k8s.tcinternal            1/1     Running   0          92s   10.31.8.1   tiny-flannel-master-8-1.k8s.tcinternal   <none>           <none>

4.4 Adding worker nodes

At this point we need to go ahead and add the remaining two nodes as worker nodes to run the load. Run the command directly on top of the remaining node that was output during successful cluster initialization to successfully join the cluster.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

$ kubeadm join 10.31.8.1:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:d7160866920c0331731ad3c1c31a6e5b6c788b5682f86971cacaa940211db9ab
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

It doesn’t matter if we accidentally don’t save the output of successful initialization, we can use the kubectl tool to view or generate the token.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

# View the list of existing tokens
$ kubeadm token list
TOKEN                     TTL         EXPIRES                USAGES                   DESCRIPTION                                                EXTRA GROUPS
abcdef.0123456789abcdef   23h         2022-05-08T06:27:34Z   authentication,signing   <none>                                                     system:bootstrappers:kubeadm:default-node-token

# If the token is no longer valid, then create a new token
$ kubeadm token create
pyab3u.j1a9ld7vk03znbk8
$ kubeadm token list
TOKEN                     TTL         EXPIRES                USAGES                   DESCRIPTION                                                EXTRA GROUPS
abcdef.0123456789abcdef   23h         2022-05-08T06:27:34Z   authentication,signing   <none>                                                     system:bootstrappers:kubeadm:default-node-token
pyab3u.j1a9ld7vk03znbk8   23h         2022-05-08T06:34:28Z   authentication,signing   <none>                                                     system:bootstrappers:kubeadm:default-node-token

# If you cannot find the --discovery-token-ca-cert-hash parameter, you can use the openssl tool on the master node to get it
$ openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
d6cdc5a3bc40cbb0ae85776eb4fcdc1854942e2dd394470ae0f2f97714dd9fb9

After adding the nodes, we can see that there are two more nodes in the cluster, but the state of the nodes is still NotReady, so we need to deploy CNI.

1
2
3
4
5

$ kubectl get nodes
NAME                                      STATUS     ROLES                  AGE     VERSION
tiny-flannel-master-8-1.k8s.tcinternal    NotReady   control-plane,master   7m49s   v1.23.6
tiny-flannel-worker-8-11.k8s.tcinternal   NotReady   <none>                 2m58s   v1.23.6
tiny-flannel-worker-8-12.k8s.tcinternal   NotReady   <none>                 102s    v1.23.6

5. Install CNI

5.1 Writing the manifest file

flannel should be one of the many open source CNI plugins with the lowest entry barrier to CNI, simple to deploy, easy to understand the principles, and the related documentation is abundant on the web.

1
2

# We first download the official yaml template, and then modify the key fields one by one
$ wget https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

For the kube-flannel.yml file, we need to modify some parameters to adapt to our cluster.

  • the net-conf.json parameter, which configures the pod’s network segment, here we use the previously planned 10.8.64.0/18

    1
2
3
4
5
6
7

    net-conf.json: |
{
    "Network": "10.8.64.0/18",
    "Backend": {
    "Type": "vxlan"
    }
}

5.2 Deploying flannel

Once the modifications are done, we can deploy it directly.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

$ kubectl apply -f kube-flannel.yml
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

# Check if the pod is running properly
$ kubectl get pods -A
NAMESPACE     NAME                                                             READY   STATUS    RESTARTS   AGE
kube-system   coredns-6d8c4cb4d-np7q2                                          1/1     Running   0          14m
kube-system   coredns-6d8c4cb4d-z8f5b                                          1/1     Running   0          14m
kube-system   etcd-tiny-flannel-master-8-1.k8s.tcinternal                      1/1     Running   0          14m
kube-system   kube-apiserver-tiny-flannel-master-8-1.k8s.tcinternal            1/1     Running   0          14m
kube-system   kube-controller-manager-tiny-flannel-master-8-1.k8s.tcinternal   1/1     Running   0          14m
kube-system   kube-flannel-ds-9fq4z                                            1/1     Running   0          12m
kube-system   kube-flannel-ds-ckstx                                            1/1     Running   0          7m18s
kube-system   kube-flannel-ds-qj55x                                            1/1     Running   0          8m25s
kube-system   kube-proxy-bncfl                                                 1/1     Running   0          14m
kube-system   kube-proxy-lslcm                                                 1/1     Running   0          7m18s
kube-system   kube-proxy-pmwhf                                                 1/1     Running   0          8m25s
kube-system   kube-scheduler-tiny-flannel-master-8-1.k8s.tcinternal            1/1     Running   0          14m

# Check flannel's pod log for reported errors
$ kubectl logs -f -l app=flannel -n kube-system

6. Deploy test cases

After the cluster is deployed we deploy an nginx in the k8s cluster to test if it works. First we create a namespace named nginx-quic, then we create a deployment named nginx-quic-deployment in this namespace to deploy pods, and finally we create a service to expose the service, here we first Here we first use nodeport to expose the port for testing purposes.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

$ cat nginx-quic.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: nginx-quic

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-quic-deployment
  namespace: nginx-quic
spec:
  selector:
    matchLabels:
      app: nginx-quic
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx-quic
    spec:
      containers:
      - name: nginx-quic
        image: tinychen777/nginx-quic:latest
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80

---

apiVersion: v1
kind: Service
metadata:
  name: nginx-quic-service
  namespace: nginx-quic
spec:
  selector:
    app: nginx-quic
  ports:
  - protocol: TCP
    port: 8080 # match for service access port
    targetPort: 80 # match for pod access port
    nodePort: 30088 # match for external access port
  type: NodePort

We check the status directly after the deployment is complete.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

# Direct Deployment
$ kubectl apply -f nginx-quic.yaml
namespace/nginx-quic created
deployment.apps/nginx-quic-deployment created
service/nginx-quic-service created

# Check the running status of the deployment
$ kubectl get deployment -o wide -n nginx-quic
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                          SELECTOR
nginx-quic-deployment   2/2     2            2           48s   nginx-quic   tinychen777/nginx-quic:latest   app=nginx-quic

# Check the running status of the service
$ kubectl get service -o wide -n nginx-quic
NAME                 TYPE       CLUSTER-IP   EXTERNAL-IP   PORT(S)          AGE   SELECTOR
nginx-quic-service   NodePort   10.8.4.218   <none>        8080:30088/TCP   62s   app=nginx-quic

# Check the running status of the pod
$ kubectl get pods -o wide -n nginx-quic
NAME                                     READY   STATUS    RESTARTS   AGE   IP          NODE                                      NOMINATED NODE   READINESS GATES
nginx-quic-deployment-696d959797-jm8w5   1/1     Running   0          73s   10.8.66.2   tiny-flannel-worker-8-12.k8s.tcinternal   <none>           <none>
nginx-quic-deployment-696d959797-lwcqz   1/1     Running   0          73s   10.8.65.2   tiny-flannel-worker-8-11.k8s.tcinternal   <none>           <none>

# View IPVS rules
$ ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.17.0.1:30088 rr
  -> 10.8.65.2:80                 Masq    1      0          0
  -> 10.8.66.2:80                 Masq    1      0          0
TCP  10.8.4.218:8080 rr
  -> 10.8.65.2:80                 Masq    1      0          0
  -> 10.8.66.2:80                 Masq    1      0          0
TCP  10.8.64.0:30088 rr
  -> 10.8.65.2:80                 Masq    1      0          0
  -> 10.8.66.2:80                 Masq    1      0          0
TCP  10.8.64.1:30088 rr
  -> 10.8.65.2:80                 Masq    1      0          0
  -> 10.8.66.2:80                 Masq    1      0          0
TCP  10.31.8.1:30088 rr
  -> 10.8.65.2:80                 Masq    1      0          0
  -> 10.8.66.2:80                 Masq    1      0          0

Finally we test that this image of nginx-quic returns by default the IP and port of the user request obtained in the nginx container.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

# First we test within the cluster
# Access the pod directly, the IP shown is the IP of the flannel.1 NIC on the master node
$ curl 10.8.66.2:80
10.8.64.0:38958
$ curl 10.8.65.2:80
10.8.64.0:46484
# Direct access to the service's ClusterIP, when the request will be forwarded to the pod
$ curl 10.8.4.218:8080
10.8.64.0:26305
# Direct access to nodeport, where the request is forwarded to the pod and does not go through the ClusterIP
$ curl 10.31.8.1:30088
10.8.64.0:6519

# Then we test outside the cluster
# Direct access to the nodeport of the three nodes, where the request will be forwarded to the pod and will not go through the ClusterIP
# Since the externalTrafficPolicy defaults to Cluster, the IP that nginx gets is the IP of the flannel.1 NIC of the node we are accessing
$ curl 10.31.8.1:30088
10.8.64.0:50688
$ curl 10.31.8.11:30088
10.8.65.1:41032
$ curl 10.31.8.12:30088
10.8.66.0:11422