The company’s PaaS platform uses the open source kubespray for the underlying kubernetes cluster deployment, and I was involved in the kubespray second-opening work. During this period, I mainly completed kubespray automated packaging and release pipeline, private deployment, added self-research CNI deployment, and some bugfixes. I recently took time to organize and summarize some of the pitfalls of using kubespray to deploy kubernetes clusters for local development and testing.
Prepare
- Need a deployment image repository and nginx
- A domain name is required, preferably with DNS resolution and SSL certificates already set up
- The cluster node needs to have at least two machines with access to the extranet
Although I have a large number of development machines on hand, I can’t resolve DNS to these Chinese servers because my domain name k8s.li
is special and difficult to file in China (and I don’t want to file). So I plan to resolve the domain name to a foreign server, and then use nginx rewrite to forward the request to AliCloud OSS; in addition, the backend storage of docker registry can also choose to use AliCloud OSS, so that when the client pulls the image, it will only get the manifest file of the image through my domain name, and the image’s During cluster deployment, the most important traffic for downloading files and mirrors will go through AliCloud OSS, which can save cluster deployment time and improve deployment efficiency, while leaving a server traffic cost.
Domain Name SSL Certificate Creation
If the certificate is self-signed or the mirror repository uses the HTTP protocol, this will cause docker or containerd to be unable to pull mirrors and require the insecure-registries
parameter to be configured for all nodes in the cluster. This is a bit of a pain, so it is recommended to add a non-self-signed SSL certificate to the mirror repository to reduce some of the unnecessary hassles. If you have a ready-made mirror repository with an SSL certificate configured, you can skip this step.
There are many ways to create domain certificates, but I recommend using acme.sh. It implements all the authentication protocols supported by the acme protocol, and supports dozens of domain name resolvers. As my domain is hosted on cloudflare, using acme.sh to issue certificates is particularly convenient, only two parameters need to be configured. The following to k8s.li this domain name issued a pan-domain certificate.
-
Install acme.sh
-
Issuing Certificate
1 2 3 4 5 6 7 8 9 10
export CF_Email="muzi502.li@gmail.com" # cloudflare 账户的邮箱 export CF_Key="xxxxxx" # "cloudflare中查看你的key" ~/.acme.sh/acme.sh --issue --dns dns_cf -d k8s.li -d *.k8s.li [Tue Apr 27 07:32:52 UTC 2021] Cert success. [Tue Apr 27 07:32:52 UTC 2021] Your cert is in /root/.acme.sh/k8s.li/k8s.li.cer [Tue Apr 27 07:32:52 UTC 2021] Your cert key is in /root/.acme.sh/k8s.li/k8s.li.key [Tue Apr 27 07:32:52 UTC 2021] The intermediate CA cert is in /root/.acme.sh/k8s.li/ca.cer [Tue Apr 27 07:32:52 UTC 2021] And the full chain certs is there: /root/.acme.sh/k8s.li/fullchain.cer
After the previous certificate is generated, you need to copy the certificate to the place where you really need to use it.
Note that the default generated certificates are placed in the installation directory
~/.acme.sh/
, please do not use the files in this directory directly, e.g. do not let thenginx/apache
configuration file use the files under it. The files in this directory are used internally and the directory structure may change.The correct way to use it is to use the
-installcert
command and specify the target location, then the certificate file will be copied to the appropriate location -
Installation certificate
Build mirror repository
-
config.yml
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
version: 0.1 log: fields: service: registry storage: cache: blobdescriptor: inmemory oss: accesskeyid: xxxx # 这里配置阿里云 OSS 的 accesskeyid accesskeysecret: xxxx # 这里配置阿里云 OSS 的 accesskeysecret region: oss-cn-beijing # 配置 OSS bucket 的区域,比如 oss-cn-beijing internal: false bucket: fileserver # 配置存储 bucket 的名称 rootdirectory: /kubespray/registry # 配置路径 delete: enabled: true http: addr: 0.0.0.0:5000 headers: X-Content-Type-Options: [nosniff] health: storagedriver: enabled: true interval: 10s threshold: 3
-
docker-compose
-
nginx.conf
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
server { listen 443 ssl; listen [::]:443; server_name hub.k8s.li; ssl_certificate /etc/nginx/ssl/fullchain.cer; ssl_certificate_key /etc/nginx/ssl/k8s.li.key; gzip_static on; client_max_body_size 4096m; if ($request_method !~* GET) { return 403; } location / { proxy_redirect off; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_pass http://localhost:5000; } }
File Server
The file server is used to store some binaries such as kubeadm, kubectl, kubelet, etc. The default download address of kubespray is particularly slow in China, so an http/https server needs to be built to download these binaries for cluster deployments.
-
nginx.conf
Note that the nginx configuration here uses rewrite instead of proxy_pass, so that when the client wants to request a file from my server, it will rewrite the client’s request and let the client request the AliCloud OSS address.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
server { listen 443; listen [::]:443; server_name dl.k8s.li; ssl_certificate /etc/nginx/ssl/fullchain.cer; ssl_certificate_key /etc/nginx/ssl/k8s.li.key; location / { rewrite ^/(.*)$ https://fileserver.oss-cn-beijing.aliyuncs.com/kubespray/files/$1; proxy_hide_header Content-Disposition; proxy_hide_header x-oss-request-id; proxy_hide_header x-oss-object-type; proxy_hide_header x-oss-hash-crc64ecma; proxy_hide_header x-oss-storage-class; proxy_hide_header x-oss-force-download; proxy_hide_header x-oss-server-time; } }
Compile and install skopeo
Installing skopeo to synchronize some used images to a private image repository is much faster than docker, and is highly recommended. skopeo installation can be found in the official documentation Installing from packages. However, I personally use go buid to compile a statically linked executable so that it can be used in all Linux distributions. Otherwise, the executable compiled on Debian cannot be used on CentOS because they use different dynamic link libraries!
|
|
Get the binaries needed for deployment
When deploying kubespray, we need to download some binaries from github.com or storage.googleapis.com, which are blocked in China, so we need to upload the files we depend on to our own file server. I wrote a script to get the binaries needed for kubespray deployment, and executed it in the root directory of the kubespray repo, where the downloaded files are stored in the temp/files
directory by default. After the download is complete, upload all subdirectories in this directory to your own file server. You can configure some parameters later by adding the URL of your file server in front of this address.
-
First clone repo to local
1
root@debian:/root# git clone https://github.com/kubernetes-sigs/kubespray && cd kubespray
-
Save the script
generate_list.sh
to the repo root directory and execute the footer to download the required files.1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57
#!/bin/bash set -eo pipefail CURRENT_DIR=$(cd $(dirname $0); pwd) TEMP_DIR="${CURRENT_DIR}/temp" REPO_ROOT_DIR="${CURRENT_DIR}" : ${IMAGE_ARCH:="amd64"} : ${ANSIBLE_SYSTEM:="linux"} : ${ANSIBLE_ARCHITECTURE:="x86_64"} : ${DOWNLOAD_YML:="roles/download/defaults/main.yml"} : ${KUBE_VERSION_YAML:="roles/kubespray-defaults/defaults/main.yaml"} mkdir -p ${TEMP_DIR} generate_versions() { # ARCH used in convert {%- if image_arch != 'amd64' -%}-{{ image_arch }}{%- endif -%} to {{arch}} if [ "${IMAGE_ARCH}" != "amd64" ]; then ARCH="${IMAGE_ARCH}"; fi cat > ${TEMP_DIR}/version.sh << EOF arch=${ARCH} image_arch=${IMAGE_ARCH} ansible_system=${ANSIBLE_SYSTEM} ansible_architecture=${ANSIBLE_ARCHITECTURE} EOF grep 'kube_version:' ${REPO_ROOT_DIR}/${KUBE_VERSION_YAML} \ | sed 's/: /=/g' >> ${TEMP_DIR}/version.sh grep '_version:' ${REPO_ROOT_DIR}/${DOWNLOAD_YML} \ | sed 's/: /=/g;s/{{/${/g;s/}}/}/g' | tr -d ' ' >> ${TEMP_DIR}/version.sh sed -i 's/kube_major_version=.*/kube_major_version=${kube_version%.*}/g' ${TEMP_DIR}/version.sh sed -i 's/crictl_version=.*/crictl_version=${kube_version%.*}.0/g' ${TEMP_DIR}/version.sh } generate_files_list() { echo "source ${TEMP_DIR}/version.sh" > ${TEMP_DIR}/files.sh grep 'download_url:' ${REPO_ROOT_DIR}/${DOWNLOAD_YML} \ | sed 's/: /=/g;s/ //g;s/{{/${/g;s/}}/}/g;s/|lower//g;s/^.*_url=/echo /g' >> ${TEMP_DIR}/files.sh bash ${TEMP_DIR}/files.sh | sort > ${TEMP_DIR}/files.list } generate_images_list() { KUBE_IMAGES="kube-apiserver kube-controller-manager kube-scheduler kube-proxy" echo "source ${TEMP_DIR}/version.sh" > ${TEMP_DIR}/images.sh grep -E '_repo:|_tag:' ${REPO_ROOT_DIR}/${DOWNLOAD_YML} \ | sed "s#{%- if image_arch != 'amd64' -%}-{{ image_arch }}{%- endif -%}#{{arch}}#g" \ | sed 's/: /=/g;s/{{/${/g;s/}}/}/g' | tr -d ' ' >> ${TEMP_DIR}/images.sh sed -n '/^downloads:/,/download_defaults:/p' ${REPO_ROOT_DIR}/${DOWNLOAD_YML} \ | sed -n "s/repo: //p;s/tag: //p" | tr -d ' ' | sed 's/{{/${/g;s/}}/}/g' \ | sed 'N;s#\n# #g' | tr ' ' ':' | sed 's/^/echo /g' >> ${TEMP_DIR}/images.sh echo "${KUBE_IMAGES}" | tr ' ' '\n' | xargs -L1 -I {} \ echo 'echo ${kube_image_repo}/{}:${kube_version}' >> ${TEMP_DIR}/images.sh bash ${TEMP_DIR}/images.sh | sort > ${TEMP_DIR}/images.list } generate_versions generate_files_list generate_images_list wget -x -P ${TEMP_DIR}/files -i ${TEMP_DIR}/files.list
The final result of the download is as follows, which basically maintains the original URL path and facilitates subsequent updates and version iterations.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59
temp/files ├── get.helm.sh │ └── helm-v3.5.4-linux-amd64.tar.gz ├── github.com │ ├── containerd │ │ └── nerdctl │ │ └── releases │ │ └── download │ │ └── v0.8.0 │ │ └── nerdctl-0.8.0-linux-amd64.tar.gz │ ├── containernetworking │ │ └── plugins │ │ └── releases │ │ └── download │ │ └── v0.9.1 │ │ └── cni-plugins-linux-amd64-v0.9.1.tgz │ ├── containers │ │ └── crun │ │ └── releases │ │ └── download │ │ └── 0.19 │ │ └── crun-0.19-linux-amd64 │ ├── coreos │ │ └── etcd │ │ └── releases │ │ └── download │ │ └── v3.4.13 │ │ └── etcd-v3.4.13-linux-amd64.tar.gz │ ├── kata-containers │ │ └── runtime │ │ └── releases │ │ └── download │ │ └── 1.12.1 │ │ └── kata-static-1.12.1-x86_64.tar.xz │ ├── kubernetes-sigs │ │ └── cri-tools │ │ └── releases │ │ └── download │ │ └── v1.20.0 │ │ └── crictl-v1.20.0-linux-amd64.tar.gz │ └── projectcalico │ ├── calico │ │ └── archive │ │ └── v3.17.3.tar.gz │ └── calicoctl │ └── releases │ └── download │ └── v3.17.3 │ └── calicoctl-linux-amd64 └── storage.googleapis.com └── kubernetes-release └── release └── v1.20.6 └── bin └── linux └── amd64 ├── kubeadm ├── kubectl └── kubelet
Get the images you need for deployment
For offline deployment, kubespray support is not very friendly. For example, to get the list of mirrors needed for deployment, the current solution is to deploy a cluster first and then get some resources to get the mirrors used by the pod through kubectl. Personally, I think this approach can be modified, for example, by generating a list of mirrors from the kubespray source code. The following is just a simple list of mirrors, with the following content
-
images.list
1 2 3 4 5 6 7 8 9 10 11 12 13 14
docker.io/nginx:1.19.0 docker.io/calico/cni:v3.17.3 docker.io/calico/node:v3.17.3 docker.io/calico/kube-controllers:v3.17.3 quay.io/coreos/flannel:v0.13.0 quay.io/coreos/flannel:v0.13.0-amd64 k8s.gcr.io/pause:3.2 k8s.gcr.io/coredns:1.7.0 k8s.gcr.io/kube-apiserver:v1.20.6 k8s.gcr.io/kube-controller-manager:v1.20.6 k8s.gcr.io/kube-proxy:v1.20.6 k8s.gcr.io/kube-scheduler:v1.20.6 k8s.gcr.io/dns/k8s-dns-node-cache:1.17.1 k8s.gcr.io/cpa/cluster-proportional-autoscaler-amd64:1.8.3
Since the master branch code is always being updated, the current version of the master branch may not be the same as the one here, so you need to change it to the version you need.
-
Based on the list of mirrors above, use skopeo to sync the mirrors to your own mirror repository, such as my
hub.k8s.li
1
for image in $(cat images.list); do skopeo copy docker://${image} docker://hub.k8s.li/${image#*/}; done
Synced to my mirror repository, the content will be as follows, by modifying the address of some mirror repositories during the deployment
1 2 3 4 5 6 7 8 9 10 11 12 13 14
hub.k8s.li/nginx:1.19.0 hub.k8s.li/calico/cni:v3.17.3 hub.k8s.li/calico/node:v3.17.3 hub.k8s.li/calico/kube-controllers:v3.17.3 hub.k8s.li/coreos/flannel:v0.13.0 hub.k8s.li/coreos/flannel:v0.13.0-amd64 hub.k8s.li/pause:3.2 hub.k8s.li/coredns:1.7.0 hub.k8s.li/kube-apiserver:v1.20.6 hub.k8s.li/kube-controller-manager:v1.20.6 hub.k8s.li/kube-proxy:v1.20.6 hub.k8s.li/kube-scheduler:v1.20.6 hub.k8s.li/dns/k8s-dns-node-cache:1.17.1 hub.k8s.li/cpa/cluster-proportional-autoscaler-amd64:1.8.3
The preparations are now largely complete, so let’s start configuring some parameters and inventory files for kubespray
Configuration
Follow the kubespray documentation to make a copy of the inventory/sample
directory and then control the deployment by modifying the parameters there.
|
|
inventory
-
deploy/inventory
Create a host inventory file in the following format.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
[all:vars] ansible_port=22 ansible_user=root ansible_ssh_private_key_file=/kubespray/.ssh/id_rsa [all] kube-control-1 ansible_host=192.168.4.11 kube-control-2 ansible_host=192.168.4.12 kube-control-3 ansible_host=192.168.4.13 kube-node-1 ansible_host=192.168.4.4 [kube_control_plane] kube-control-1 kube-control-2 kube-control-3 [etcd] kube-control-1 kube-control-2 kube-control-3 [kube-node] kube-control-1 kube-control-2 kube-control-3 kube-node-1 [calico-rr] [k8s-cluster:children] kube_control_plane kube-node calico-rr
-
ssh mutual trust
Kubespray uses ansible’s synchronize module to distribute files, based on the rsync protocol, so you must use ssh key pairs to connect to cluster nodes. inventory is configured as a path within the kubespray container, so you need to copy the ssh public and private keys to the repo’s .ssh directory. If the node does not have ssh-free login, you can use the authorized_key module of ansible to add the ssh public key to the authorized_key of the host. The procedure is as follows.
1 2 3 4 5 6 7
root@debian:/root/kubespray git:(master*) # mkdir -p .ssh # 生成 ssh 密钥对 root@debian:/root/kubespray git:(master*) # ssh-keygen -t rsa -f .ssh/id_rsa -P "" # 将 ssh 公钥添加到所有主机 root@debian:/root/kubespray git:(master*) # ansible -i deploy/inventory all -m authorized_key -a "user={{ ansible_user }} key='{{ lookup('file', '{{ ssh_cert_path }}') }}'" -e ssh_cert_path=./.ssh/id_rsa.pub -e ansible_ssh_pass=passwd
vars
Create and modify the following configuration file
-
deploy/env.yml
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
--- # 定义一些组件的版本 kube_version: v1.20.6 calico_version: "v3.17.3" pod_infra_version: "3.2" nginx_image_version: "1.19" coredns_version: "1.7.0" image_arch: "amd64" # docker registry domain registry_domain: "hub.k8s.li" # file download server url download_url: "https://dl.k8s.li" # docker-ce-repo mirrors docker_mirrors_url: "https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux" container_manager: "containerd" # 由于使用的是 containerd 作为 CRI,目前 etcd 不支持 containerd 容器化部署因此需要将该参数修改为 host ,使用 systemd 来部署 etcd_deployment_type: host etcd_cluster_setup: true etcd_events_cluster_setup: true etcd_events_cluster_enabled: true # kubernetes CNI type 配置集群 CNI 使用的类型 kube_network_plugin: canal
-
deploy/group_vars/all/download.yml
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
## Container registry define gcr_image_repo: "{{ registry_domain }}" kube_image_repo: "{{ registry_domain }}" docker_image_repo: "{{ registry_domain }}" quay_image_repo: "{{ registry_domain }}" # Download URLs kubelet_download_url: "{{ download_url }}/storage.googleapis.com/kubernetes-release/release/{{ kube_version }}/bin/linux/{{ image_arch }}/kubelet" kubectl_download_url: "{{ download_url }}/storage.googleapis.com/kubernetes-release/release/{{ kube_version }}/bin/linux/{{ image_arch }}/kubectl" kubeadm_download_url: "{{ download_url }}/storage.googleapis.com/kubernetes-release/release/{{ kubeadm_version }}/bin/linux/{{ image_arch }}/kubeadm" etcd_download_url: "{{ download_url }}/github.com/coreos/etcd/releases/download/{{ etcd_version }}/etcd-{{ etcd_version }}-linux-{{ image_arch }}.tar.gz" cni_download_url: "{{ download_url }}/github.com/containernetworking/plugins/releases/download/{{ cni_version }}/cni-plugins-linux-{{ image_arch }}-{{ cni_version }}.tgz" calicoctl_download_url: "{{ download_url }}/github.com/projectcalico/calicoctl/releases/download/{{ calico_ctl_version }}/calicoctl-linux-{{ image_arch }}" calico_crds_download_url: "{{ download_url }}/github.com/projectcalico/calico/archive/{{ calico_version }}.tar.gz" crictl_download_url: "{{ download_url }}/github.com/kubernetes-sigs/cri-tools/releases/download/{{ crictl_version }}/crictl-{{ crictl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz" helm_download_url: "{{ download_url }}/get.helm.sh/helm-{{ helm_version }}-linux-{{ image_arch }}.tar.gz" crun_download_url: "{{ download_url }}/github.com/containers/crun/releases/download/{{ crun_version }}/crun-{{ crun_version }}-linux-{{ image_arch }}" kata_containers_download_url: "{{ download_url }}/github.com/kata-containers/runtime/releases/download/{{ kata_containers_version }}/kata-static-{{ kata_containers_version }}-{{ ansible_architecture }}.tar.xz" nerdctl_download_url: "{{ download_url }}/github.com/containerd/nerdctl/releases/download/v{{ nerdctl_version }}/nerdctl-{{ nerdctl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"
docker-ce mirrors
When kubespray installs docker or containerd containers to run, you need to use the docker-ce source, and you can use Tsinghua’s image source in China. Depending on the Linux distribution, you can add these parameters to the deploy/group_vars/all/offline.yml
file. The parameter docker_mirrors_url
is the parameter set in env.yml
.
-
CentOS/Redhat
1 2 3 4 5 6 7
## CentOS/Redhat ### For EL7, base and extras repo must be available, for EL8, baseos and appstream ### By default we enable those repo automatically # rhel_enable_repos: false ### Docker / Containerd docker_rh_repo_base_url: "{{ docker_mirrors_url }}/centos/{{ ansible_distribution_major_version }}/{{ ansible_architecture }}/stable" docker_rh_repo_gpgkey: "{{ docker_mirrors_url }}/centos/gpg"
-
Fedora
1 2 3 4 5 6 7
## Fedora ### Docker docker_fedora_repo_base_url: "{{ docker_mirrors_url }}/fedora/{{ ansible_distribution_major_version }}/{{ ansible_architecture }}/stable" docker_fedora_repo_gpgkey: "{{ docker_mirrors_url }}/fedora/gpg" ### Containerd containerd_fedora_repo_base_url: "{{ docker_mirrors_url }}/fedora/{{ ansible_distribution_major_version }}/{{ ansible_architecture }}/stable" containerd_fedora_repo_gpgkey: "{{ docker_mirrors_url }}/fedora/gpg"
-
debian
1 2 3 4 5 6 7 8
## Debian ### Docker docker_debian_repo_base_url: "{{ docker_mirrors_url }}/debian" docker_debian_repo_gpgkey: "{{ docker_mirrors_url }}/debian/gpg" ### Containerd containerd_debian_repo_base_url: "{{ docker_mirrors_url }}/debian" containerd_debian_repo_gpgkey: "{{ docker_mirrors_url }}/debian/gpg" # containerd_debian_repo_repokey: 'YOURREPOKEY'
-
ubuntu
1 2 3 4 5 6 7
## Ubuntu ### Docker docker_ubuntu_repo_base_url: "{{ docker_mirrors_url }}/ubuntu" docker_ubuntu_repo_gpgkey: "{{ docker_mirrors_url }}/ubuntu/gpg" ### Containerd containerd_ubuntu_repo_base_url: "{{ docker_mirrors_url }}/ubuntu" containerd_ubuntu_repo_gpgkey: "{{ docker_mirrors_url }}/ubuntu/gpg"
Deployment
After preparing the configuration above, it’s time to start the deployment. When using ansible for deployment, I prefer to operate in a kubespray container rather than installing python3 on the local development machine. For offline deployments, it is more convenient to build the image in advance and use a docker container.
-
Build Mirror
1
root@debian:/root/kubespray git:(master*) # docker build -t kubespray:v2.15.1-kube-v1.20.6 .
-
Run the kubespray container
1
root@debian:/root/kubespray git:(master*) # docker run --rm -it --net=host -v $PWD:/kubespray kubespray:v2.15.1-kube-v1.20.6 bash
-
Test if the host is connected properly
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
root@debian:/kubespray# ansible -i cluster/inventory all -m ping kube-control-3 | SUCCESS => { "ansible_facts": { "discovered_interpreter_python": "/usr/bin/python" }, "changed": false, "ping": "pong" } kube-control-1 | SUCCESS => { "ansible_facts": { "discovered_interpreter_python": "/usr/bin/python" }, "changed": false, "ping": "pong" } kube-node-1 | SUCCESS => { "ansible_facts": { "discovered_interpreter_python": "/usr/bin/python" }, "changed": false, "ping": "pong" } kube-control-2 | SUCCESS => { "ansible_facts": { "discovered_interpreter_python": "/usr/bin/python" }, "changed": false, "ping": "pong" }
-
Start cluster deployment
1
root@debian:/kubespray# ansible-playbook -i deploy/inventory -e "@deploy/env.yml" cluster.yml
-
The deployment completion log is as follows, when failed is 0, it means the tasks have been successfully run
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
PLAY RECAP ****************************************************************** kube-control-1 : ok=526 changed=67 unreachable=0 failed=0 skipped=978 rescued=0 ignored=0 kube-control-2 : ok=524 changed=66 unreachable=0 failed=0 skipped=980 rescued=0 ignored=0 kube-control-3 : ok=593 changed=76 unreachable=0 failed=0 skipped=1125 rescued=0 ignored=1 kube-node-1 : ok=366 changed=34 unreachable=0 failed=0 skipped=628 rescued=0 ignored=0 localhost : ok=3 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 Wednesday 28 April 2021 10:57:57 +0000 (0:00:00.115) 0:15:21.190 ******* =============================================================================== kubernetes/control-plane : kubeadm | Initialize first master -------------- 65.88s kubernetes/control-plane : Joining control plane node to the cluster. ----- 50.05s kubernetes/kubeadm : Join to cluster -------------------------------------- 31.54s download_container | Download image if required --------------------------- 24.38s reload etcd --------------------------------------------------------------- 20.56s Gen_certs | Write etcd member and admin certs to other etcd nodes --------- 19.32s Gen_certs | Write node certs to other etcd nodes -------------------------- 19.14s Gen_certs | Write etcd member and admin certs to other etcd nodes --------- 17.45s network_plugin/canal : Canal | Create canal node manifests ---------------- 15.41s kubernetes-apps/ansible : Kubernetes Apps | Lay Down CoreDNS Template ----- 13.27s kubernetes/control-plane : Master | wait for kube-scheduler --------------- 11.97s download_container | Download image if required --------------------------- 11.76s Gen_certs | Write node certs to other etcd nodes -------------------------- 10.50s kubernetes-apps/ansible : Kubernetes Apps | Start Resources ---------------- 8.28s policy_controller/calico : Create calico-kube-controllers manifests -------- 7.61s kubernetes/control-plane : set kubeadm certificate key --------------------- 6.32s download : extract_file | Unpacking archive -------------------------------- 5.51s Configure | Check if etcd cluster is healthy ------------------------------- 5.41s Configure | Check if etcd-events cluster is healthy ------------------------ 5.41s kubernetes-apps/network_plugin/canal : Canal | Start Resources ------------- 4.85s
-
Cluster Status
1 2 3 4 5 6
[root@kube-control-1 ~]# kubectl get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME kube-control-1 Ready control-plane,master 5m24s v1.20.6 192.168.4.11 <none> CentOS Linux 7 (Core) 3.10.0-1160.el7.x86_64 containerd://1.4.4 kube-control-2 Ready control-plane,master 5m40s v1.20.6 192.168.4.12 <none> CentOS Linux 7 (Core) 3.10.0-1160.el7.x86_64 containerd://1.4.4 kube-control-3 Ready control-plane,master 6m28s v1.20.6 192.168.4.13 <none> CentOS Linux 7 (Core) 3.10.0-1160.el7.x86_64 containerd://1.4.4 kube-node-1 Ready <none> 3m53s v1.20.6 192.168.4.14 <none> CentOS Linux 7 (Core) 3.10.0-1160.el7.x86_64 containerd://1.4.4
-
Cluster Component Status
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46
[root@kube-control-1 ~]# kubectl get all -n kube-system NAME READY STATUS RESTARTS AGE pod/calico-kube-controllers-67d6cdb559-cwf62 0/1 CrashLoopBackOff 5 4m10s pod/canal-node-46x2b 2/2 Running 0 4m25s pod/canal-node-5rkhq 2/2 Running 0 4m25s pod/canal-node-fcsgn 2/2 Running 0 4m25s pod/canal-node-nhkp8 2/2 Running 0 4m25s pod/coredns-5d578c6f84-5nnp8 1/1 Running 0 3m33s pod/coredns-5d578c6f84-w2kvf 1/1 Running 0 3m39s pod/dns-autoscaler-6b675c8995-vp282 1/1 Running 0 3m34s pod/kube-apiserver-kube-control-1 1/1 Running 0 6m51s pod/kube-apiserver-kube-control-2 1/1 Running 0 7m7s pod/kube-apiserver-kube-control-3 1/1 Running 0 7m41s pod/kube-controller-manager-kube-control-1 1/1 Running 0 6m52s pod/kube-controller-manager-kube-control-2 1/1 Running 0 7m7s pod/kube-controller-manager-kube-control-3 1/1 Running 0 7m41s pod/kube-proxy-5dfx8 1/1 Running 0 5m17s pod/kube-proxy-fvrqk 1/1 Running 0 5m17s pod/kube-proxy-jd84p 1/1 Running 0 5m17s pod/kube-proxy-l2mjk 1/1 Running 0 5m17s pod/kube-scheduler-kube-control-1 1/1 Running 0 6m51s pod/kube-scheduler-kube-control-2 1/1 Running 0 7m7s pod/kube-scheduler-kube-control-3 1/1 Running 0 7m41s pod/nginx-proxy-kube-node-1 1/1 Running 0 5m20s pod/nodelocaldns-77kq9 1/1 Running 0 3m32s pod/nodelocaldns-fn5pd 1/1 Running 0 3m32s pod/nodelocaldns-lfjzb 1/1 Running 0 3m32s pod/nodelocaldns-xnc6n 1/1 Running 0 3m32s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/coredns ClusterIP 10.233.0.3 <none> 53/UDP,53/TCP,9153/TCP 3m38s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/canal-node 4 4 4 4 4 <none> 4m25s daemonset.apps/kube-proxy 4 4 4 4 4 kubernetes.io/os=linux 7m53s daemonset.apps/nodelocaldns 4 4 4 4 4 <none> 3m32s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/calico-kube-controllers 0/1 1 0 4m12s deployment.apps/coredns 2/2 2 2 3m39s deployment.apps/dns-autoscaler 1/1 1 1 3m34s NAME DESIRED CURRENT READY AGE replicaset.apps/calico-kube-controllers-67d6cdb559 1 1 0 4m12s replicaset.apps/coredns-5d578c6f84 2 2 2 3m39s replicaset.apps/dns-autoscaler-6b675c8995 1 1 1 3m34s