Yesterday there was a problem with the network environment, the local virtual machine built Kubernetes environment does not have a fixed IP, the result of the node IP changed, of course the easiest way is to re-fix the node back to the previous IP address, but I stubbornly want to modify the IP address of the cluster, the results encountered a lot of problems, and not as simple as I thought.
Environment
First look at the previous environment.
1
2
3
4
|
➜ ~ cat /etc/hosts
192.168.0.111 master1
192.168.0.109 node1
192.168.0.110 node2
|
New IP address.
1
2
3
4
|
➜ ~ cat /etc/hosts
192.168.0.106 master1
192.168.0.101 node1
192.168.0.105 node2
|
So we need to change the IP addresses of all nodes.
Operation steps
First change the /etc/hosts
of all nodes to the new addresses.
Before manipulating any files it is highly recommended to backup .
master node
-
Back up the /etc/kubernetes
directory.
1
|
➜ cp -Rf /etc/kubernetes/ /etc/kubernetes-bak
|
-
Replace the APIServer address of all configuration files in /etc/kubernetes
.
1
2
3
4
5
6
7
8
|
➜ oldip=192.168.0.111
➜ newip=192.168.0.106
# 查看之前的
➜ find . -type f | xargs grep $oldip
# 替换IP地址
➜ find . -type f | xargs sed -i "s/$oldip/$newip/"
# 检查更新后的
➜ find . -type f | xargs grep $newip
|
-
Identify the certificate in /etc/kubernetes/pki
that has the old IP address as the alt name
.
1
2
3
4
5
6
|
➜ cd /etc/kubernetes/pki
➜ for f in $(find -name "*.crt"); do
openssl x509 -in $f -text -noout > $f.txt;
done
➜ grep -Rl $oldip .
➜ for f in $(find -name "*.crt"); do rm $f.txt; done
|
-
Find the ConfigMap in the kube-system
namespace that references the old IP.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
# 获取所有的 kube-system 命名空间下面所有的 ConfigMap
➜ configmaps=$(kubectl -n kube-system get cm -o name | \
awk '{print $1}' | \
cut -d '/' -f 2)
# 获取所有的ConfigMap资源清单
➜ dir=$(mktemp -d)
➜ for cf in $configmaps; do
kubectl -n kube-system get cm $cf -o yaml > $dir/$cf.yaml
done
# 找到所有包含旧 IP 的 ConfigMap
➜ grep -Hn $dir/* -e $oldip
# 然后编辑这些 ConfigMap,将旧 IP 替换成新的 IP
➜ kubectl -n kube-system edit cm kubeadm-config
➜ kubectl -n kube-system edit cm kube-proxy
|
This step is very, very important. I neglected this step when I was working on it, which caused Flannel CNI to fail to start and keep reporting errors, similar to the log message below.
1
2
3
4
|
➜ kubectl logs -f kube-flannel-ds-pspzf -n kube-system
I0512 14:46:26.044229 1 main.go:205] CLI flags config: {etcdEndpoints:http://127.0.0.1:4001,http://127.0.0.1:2379 etcdPrefix:/coreos.com/network etcdKeyfile: etcdCertfile: etcdCAFile: etcdUsername: etcdPassword: version:false kubeSubnetMgr:true kubeApiUrl: kubeAnnotationPrefix:flannel.alpha.coreos.com kubeConfigFile: iface:[ens33] ifaceRegex:[] ipMasq:true subnetFile:/run/flannel/subnet.env publicIP: publicIPv6: subnetLeaseRenewMargin:60 healthzIP:0.0.0.0 healthzPort:0 iptablesResyncSeconds:5 iptablesForwardRules:true netConfPath:/etc/kube-flannel/net-conf.json setNodeNetworkUnavailable:true}
W0512 14:46:26.044617 1 client_config.go:614] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
E0512 14:46:56.142921 1 main.go:222] Failed to create SubnetManager: error retrieving pod spec for 'kube-system/kube-flannel-ds-pspzf': Get "https://10.96.0.1:443/api/v1/namespaces/kube-system/pods/kube-flannel-ds-pspzf": dial tcp 10.96.0.1:443: i/o timeout
|
Actually, I couldn’t connect to the apiserver, and it took me a long time to check the kube-proxy logs, which showed the following error message.
1
|
E0512 14:53:03.260817 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://192.168.0.111:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&limit=500&resourceVersion=0": dial tcp 192.168.0.111:6443: connect: no route to host
|
This is because the apiserver address configured in kube-proxy’s ConfigMap is the old IP address, so be sure to replace it with the new one.
-
Delete the certificate and private key grep’d in step 3 and regenerate them.
1
2
3
4
5
6
|
➜ cd /etc/kubernetes/pki
➜ rm apiserver.crt apiserver.key
➜ kubeadm init phase certs apiserver
➜ rm etcd/peer.crt etcd/peer.key
➜ kubeadm init phase certs etcd-peer
|
It is of course possible to regenerate all of them.
1
|
➜ kubeadm init phase certs all
|
-
Generate a new kubeconfig file.
1
2
3
4
5
6
7
8
9
10
11
|
➜ cd /etc/kubernetes
➜ rm -f admin.conf kubelet.conf controller-manager.conf scheduler.conf
➜ kubeadm init phase kubeconfig all
I0513 15:33:34.404780 52280 version.go:255] remote version is much newer: v1.24.0; falling back to: stable-1.22
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
# 覆盖默认的 kubeconfig 文件
➜ cp /etc/kubernetes/admin.conf $HOME/.kube/config
|
-
Restart the kubelet.
1
2
|
➜ systemctl restart containerd
➜ systemctl restart kubelet
|
The normal Kubernetes clusters are now accessible.
1
2
3
4
5
|
➜ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1 Ready control-plane,master 48d v1.22.8
node1 NotReady <none> 48d v1.22.8
node2 NotReady <none> 48d v1.22.8
|
node nodes
Although the cluster is now accessible, we can see that the Node node is now in the NotReady
state, and we can go check the kubelet logs for the node2 node.
1
2
3
4
5
6
|
➜ journalctl -u kubelet -f
......
May 13 15:47:55 node2 kubelet[1194]: E0513 15:47:55.470896 1194 kubelet.go:2412] "Error getting node" err="node \"node2\" not found"
May 13 15:47:55 node2 kubelet[1194]: E0513 15:47:55.531695 1194 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://192.168.0.111:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp 192.168.0.111:6443: connect: no route to host
May 13 15:47:55 node2 kubelet[1194]: E0513 15:47:55.571958 1194 kubelet.go:2412] "Error getting node" err="node \"node2\" not found"
May 13 15:47:55 node2 kubelet[1194]: E0513 15:47:55.673379 1194 kubelet.go:2412] "Error getting node" err="node \"node2\" not found"
|
You can see that the previous APIServer address is still being accessed, so where is the APIServer address being used explicitly? We can check the kubelet startup parameters with the following command.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
➜ systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Fri 2022-05-13 14:37:31 CST; 1h 13min ago
Docs: https://kubernetes.io/docs/
Main PID: 1194 (kubelet)
Tasks: 15
Memory: 126.9M
CGroup: /system.slice/kubelet.service
└─1194 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kub...
May 13 15:51:08 node2 kubelet[1194]: E0513 15:51:08.787677 1194 kubelet.go:2412] "Error getting node" err="node \"node2... found"
May 13 15:51:08 node2 kubelet[1194]: E0513 15:51:08.888194 1194 kubelet.go:2412] "Error getting node" err="node \"node2... found"
......
|
The core configuration file is /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
and its contents are shown below.
1
2
3
4
5
6
7
8
9
10
11
12
|
➜ cat /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
|
There is a configuration KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet. conf
, where two configuration files bootstrap-kubelet.conf
and kubelet.conf
are mentioned, the first of which does not exist.
1
2
|
➜ cat /etc/kubernetes/bootstrap-kubelet.conf
cat: /etc/kubernetes/bootstrap-kubelet.conf: No such file or directory
|
The second configuration file is in the form of a kubeconfig file, and this file specifies the address of the APIServer, which you can see is still the same IP address as before.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
➜ cat /etc/kubernetes/kubelet.conf
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: <......>
server: https://192.168.0.111:6443
name: default-cluster
contexts:
- context:
cluster: default-cluster
namespace: default
user: default-auth
name: default-context
current-context: default-context
kind: Config
preferences: {}
users:
- name: default-auth
user:
client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
client-key: /var/lib/kubelet/pki/kubelet-client-current.pem
|
So our first thought must be to change the APIServer address here to a new IP address, but this is obviously problematic because the relevant certificate is still the same as before and needs to be regenerated, so how do we regenerate the file?
The first step is to backup the kubelet working directory.
1
2
|
➜ cp /etc/kubernetes/kubelet.conf /etc/kubernetes/kubelet.conf.bak
➜ cp -rf /var/lib/kubelet/ /var/lib/kubelet-bak
|
Remove the kubelet client certificate.
1
|
➜ rm /var/lib/kubelet/pki/kubelet-client*
|
Then go generate the kubelet.conf file at the master1 node (the node with the /etc/kubernetes/pki/ca.key
file).
1
2
|
# 在master1节点
➜ kubeadm kubeconfig user --org system:nodes --client-name system:node:node2 --config kubeadm.yaml > kubelet.conf
|
Then copy the kubelet.conf file to the node2 node /etc/kubernetes/kubelet.conf
, then restart the kubelet on the node2 node and wait for /var/lib/kubelet/pki/kubelet-client-current. pem
is recreated.
1
2
3
4
5
6
7
8
|
➜ systemctl restart kubelet
# 重启后等待重新生成 kubelet 客户端证书
➜ ll /var/lib/kubelet/pki/
total 12
-rw------- 1 root root 1106 May 13 16:32 kubelet-client-2022-05-13-16-32-35.pem
lrwxrwxrwx 1 root root 59 May 13 16:32 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2022-05-13-16-32-35.pem
-rw-r--r-- 1 root root 2229 Mar 26 14:39 kubelet.crt
-rw------- 1 root root 1675 Mar 26 14:39 kubelet.key
|
Ideally we can point to the rotating kubelet client certificate by manually editing kubelet.conf
, replacing client-certificate-data
and client-key-data
in the file with /var/lib/kubelet/pki/kubelet- client-current.pem
.
1
2
|
client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
client-key: /var/lib/kubelet/pki/kubelet-client-current.pem
|
Restart the kubelet again and the node2 node will now be Ready. Just configure the node1 node again in the same way.
Recommended operation
The above operation can accomplish our needs normally, but it requires some knowledge of the relevant certificates. There is a simpler way to do this.
First, stop the kubelet and back up the directory you want to work on.
1
2
3
|
➜ systemctl stop kubelet
➜ mv /etc/kubernetes /etc/kubernetes-bak
➜ mv /var/lib/kubelet/ /var/lib/kubelet-bak
|
Preserve the pki certificate directory.
1
2
3
4
5
6
7
|
➜ mkdir -p /etc/kubernetes
➜ cp -r /etc/kubernetes-bak/pki /etc/kubernetes
➜ rm /etc/kubernetes/pki/{apiserver.*,etcd/peer.*}
rm: remove regular file '/etc/kubernetes/pki/apiserver.crt'? y
rm: remove regular file '/etc/kubernetes/pki/apiserver.key'? y
rm: remove regular file '/etc/kubernetes/pki/etcd/peer.crt'? y
rm: remove regular file '/etc/kubernetes/pki/etcd/peer.key'? y
|
Now we use the following command to reinitialize the control plane nodes, but the most important point is to use etcd’s data directory, which can be done by telling kubeadm to use pre-existing etcd data with the --ignore-preflight-errors=DirAvailable--var-lib-etcd
flag.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
|
➜ kubeadm init --config kubeadm.yaml --ignore-preflight-errors=DirAvailable--var-lib-etcd
[init] Using Kubernetes version: v1.22.8
[preflight] Running pre-flight checks
[WARNING DirAvailable--var-lib-etcd]: /var/lib/etcd is not empty
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Using existing ca certificate authority
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [api.k8s.local kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master1] and IPs [10.96.0.1 192.168.0.106]
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost master1] and IPs [192.168.0.106 127.0.0.1 ::1]
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 12.003599 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.22" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master1 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node master1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.0.106:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:27993cae9c76d18a1b82b800182c4c7ebc7a704ba1093400ed886f65e709ec04
|
The above operation is almost the same as when we go to initialize the cluster, the only difference is the addition of the --ignore-preflight-errors=DirAvailable--var-lib-etcd
parameter, which means that the data from the previous etcd is used. Then we can verify that the IP address of the APIServer has changed to the new address.
1
2
3
4
5
6
7
|
➜ cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
cp: overwrite '/root/.kube/config'? y
➜ kubectl cluster-info
Kubernetes control plane is running at https://192.168.0.106:6443
CoreDNS is running at https://192.168.0.106:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
|
For node nodes we can reset and rejoin the cluster.
1
2
|
# 在node节点操作
➜ kubeadm reset
|
Just reset and rejoin the cluster.
1
2
3
|
# 在node节点操作
➜ kubeadm join 192.168.0.106:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:27993cae9c76d18a1b82b800182c4c7ebc7a704ba1093400ed886f65e709ec04
|
This way is much easier than the above way. The cluster is also normal after normal operation.
1
2
3
4
5
|
➜ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1 Ready control-plane,master 48d v1.22.8
node1 Ready <none> 48d v1.22.8
node2 Ready <none> 4m50s v1.22.8
|
Summary
It is best to use static IP addresses for Kubernetes cluster nodes to avoid the impact of IP changes on the business. If it is not a static IP, it is also highly recommended to add a custom domain name for signing, so that when the IP changes, you can directly remap the domain name. You only need to configure apiServer.certSANs
in the kubeadm configuration file via ClusterConfiguration
, as shown below.
1
2
3
4
5
6
7
8
9
|
apiVersion: kubeadm.k8s.io/v1beta3
apiServer:
timeoutForControlPlane: 4m0s
certSANs:
- api.k8s.local
- master1
- 192.168.0.106
kind: ClusterConfiguration
......
|
Add the address that needs to be signed to the certSANs configuration. For example, here we have added an additional api.k8s.local
address so that even if the IP changes in the future you can directly map this domain name to the new IP address, also if you want to access your cluster via an external access IP, then you need to add your external IP address for signature authentication.