kubelet Remote Debugging

1. kubelet startup command analysis

kubelet is a systemd service. Take the v1.23.4 k8s cluster installed with Kubeadm tool as an example, the path of the configuration file of this service is /etc/systemd/system/kubelet.service.d/10-kubeadm.conf.

The contents are as follows.

# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

Take my test environment as an example, execute ps -ef |grep /usr/bin/kubelet , you can see the full command to start kubelet as follows.

/usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=remote --container-runtime-endpoint=/run/containerd/containerd.sock --pod-infra-container-image=k8s.gcr.io/pause:3.6

If you need to modify the kubelet command, you can shut down the service and start it with the same parameters. Or restart the kubelet service after modifying the systemd configuration file.

2. Compile kubelet

According to the source code analysis of the k8s makefile, the kubelet compilation command is as follows.

https://github.com/kubernetes/kubernetes/blob/v1.22.4/hack/lib/golang.sh#L679

kube::golang::build_some_binaries() {
    ...
    go install "${build_args[@]}" "$@"
    ...
}

where GOLDFLAGS, GOGCFLAGS are configured as follows.

https://github.com/kubernetes/kubernetes/blob/v1.22.4/hack/lib/golang.sh#L797-L799

kube::golang::build_binaries() {
    ...
    goldflags="${GOLDFLAGS=-s -w} $(kube::version::ldflags)"
    goasmflags="-trimpath=${KUBE_ROOT}"
    gogcflags="${GOGCFLAGS:-} -trimpath=${KUBE_ROOT}"
    ...
}

In order to preserve as much debugging information as possible, we need to reset both compilation parameters, so the command to compile the kubelet is as follows.

git clone https://github.com/kubernetes/kubernetes.git
cd kubernetes
git checkout v1.22.4
./build/shell.sh
make generated_files
make -o generated_files kubelet KUBE_BUILD_PLATFORMS=linux/amd64 GOLDFLAGS="" GOGCFLAGS="all=-N -l"

In order to preserve as much debugging information as possible, we need to reset both compilation parameters, so the command to compile the kubelet should be as follows.

After compilation, the kubelet binaries are located at _output/bin/kubelet.

3. Introduction to delve

delve is a debugger for the Go programming language. Although we can also use gdb to debug go language programs, delve is a better alternative to GDB when debugging Go programs built with the standard toolchain. It understands the Go runtime, data structures and expressions better than GDB.

You can install dlv using the following command:

`1`	`go install github.com/go-delve/delve/cmd/dlv@latest`

Use the following command to debug with dlv:

`1`	`dlv exec ./hello -- server --config conf/config.toml`

Taking kubelet as an example, the process of debugging using the dlv command line is as follows.

root@st0n3-host:~# dlv exec /usr/bin/kubelet -- --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=remote --container-runtime-endpoint=/run/containerd/containerd.sock --pod-infra-container-image=k8s.gcr.io/pause:3.6
Type 'help' for list of commands.
(dlv) b main.main
Breakpoint 1 set at 0x502e086 for main.main() _output/dockerized./cmd/kubelet/kubelet.go:39
(dlv) c
> main.main() _output/dockerized./cmd/kubelet/kubelet.go:39 (hits goroutine(1):1 total:1) (PC: 0x502e086)
    34:		_ "k8s.io/component-base/metrics/prometheus/restclient"
    35:		_ "k8s.io/component-base/metrics/prometheus/version" // for version metric registration
    36:		"k8s.io/kubernetes/cmd/kubelet/app"
    37:	)
    38:	
=>  39:	func main() {
    40:		command := app.NewKubeletCommand()
    41:	
    42:		// kubelet uses a config file and does its own special
    43:		// parsing of flags and that config file. It initializes
    44:		// logging after it is done with that. Therefore it does
(dlv)

4. Using GoLand to remotely debug kubelet

We can certainly debug using the command line form described above, but the kubernetes code is huge and using an IDE would be more convenient.

Click the Edit Configurations button to the left of the debug button to configure the address and port of the dlv.

Using GoLand to remotely debug kubelet

Start the kubelet using the command at the IDE prompt, or restart the service after configuring it into the systemd service.

root@st0n3-host:~# cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf 
...
ExecStart=/usr/bin/dlv --listen=:10086 --headless=true --api-version=2 --accept-multiclient exec /usr/bin/kubelet -- $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
root@st0n3-host:~# systemctl daemon-reload
root@st0n3-host:~# systemctl restart kubelet.service

At this point the kubelet command hasn’t actually started yet. The kubelet will only run after running the configuration you just added in GoLand and connecting to dlv.

After setting the breakpoints and clicking the debug button, we can debug the kubelet in the IDE.

debug the kubelet in the goland

5. Other container software debugging commands

5.1 runc

Compilation

1
2

make shell
make EXTRA_FLAGS='-gcflags="all=-N -l"'

Debugging

mv /usr/bin/runc /usr/bin/runc.bak
cat <<EOF > /usr/bin/runc
#!/bin/bash
dlv --listen=:2345 --headless=true --api-version=2 --accept-multiclient exec /usr/bin/runc.debug -- $*
chmod +x /usr/bin/runc

5.2 docker-cli

compile

Modify the compile command in scripts/build/binary as follows: remove LDFLAGS and add gcflags.

root@st0n3:~/cli# git diff
diff --git a/scripts/build/binary b/scripts/build/binary
index e4c5e12a6b..155528e501 100755
--- a/scripts/build/binary
+++ b/scripts/build/binary
@@ -74,7 +74,7 @@ fi
 echo "Building $GO_LINKMODE $(basename "${TARGET}")"
 
 export GO111MODULE=auto
-
-go build -o "${TARGET}" -tags "${GO_BUILDTAGS}" --ldflags "${LDFLAGS}" ${GO_BUILDMODE} "${SOURCE}"
+go build -o "${TARGET}" -tags "${GO_BUILDTAGS}" -gcflags="all=-N -l" ${GO_BUILDMODE} "${SOURCE}"
 
 ln -sf "$(basename "${TARGET}")" "$(dirname "${TARGET}")/docker"

1
2

make -f docker.Makefile shell
make binary

Debugging

cat <<EOF > docker.debug
#!/bin/bash
dlv --listen=:2344 --headless=true --api-version=2 --accept-multiclient exec ./docker-cli.debug -- $*
chmod +x docker.debug

5.3 dockerd

Compile

Modify the compile command in the hack/make/.binary file.

root@st0n3:~/moby# git diff
diff --git a/hack/make/.binary b/hack/make/.binary
index d56e3f3126..3e23865c81 100644
--- a/hack/make/.binary
+++ b/hack/make/.binary
@@ -81,11 +81,11 @@ hash_files() {
 
        echo "Building: $DEST/$BINARY_FULLNAME"
        echo "GOOS=\"${GOOS}\" GOARCH=\"${GOARCH}\" GOARM=\"${GOARM}\""
-       go build \
+       set -x
+       go build -gcflags "all=-N -l"  \
                -o "$DEST/$BINARY_FULLNAME" \
                "${BUILDFLAGS[@]}" \
                -ldflags "
-               $LDFLAGS
                $LDFLAGS_STATIC_DOCKER
                $DOCKER_LDFLAGS
        " \

1
2

make BIND_DIR=. shell
hack/make.sh binary

Debugging

`1`	`/root/go/bin/dlv --listen=:2343 --headless=true --api-version=2 --accept-multiclient exec /usr/bin/dockerd.debug -- -D -H unix:///var/run/docker.sock --containerd=/run/containerd/containerd.sock`

Table of Contents

1. kubelet startup command analysis

2. Compile kubelet

3. Introduction to delve

4. Using GoLand to remotely debug kubelet

5. Other container software debugging commands

5.1 runc

5.2 docker-cli

compile

5.3 dockerd

Compile