I recently had a requirement to collect extranet access to Kubernetes. Therefore, we have investigated and tried out related projects, and this post focuses on how to install Kindling and configure Grafana to view Kubernetes network connection data.

1. What is Kindling

There are currently two versions of Kindling, an open source version and a commercial version. The open source version, which collects data in insufficient detail and can only be observed through Grafana, and the commercial version, which has enhanced functionality, are described on the project’s Github home page and will not be repeated here.

This eBPF-based technique of converting kernel function calls into userspace events and then exposing them to user programs should be interesting in the coming years. Here’s a simple deployment, using the open source version of Kindling.

2. Installing Kindling

The Kindling open source community is not very well run, and the documentation and materials are not clear enough. I’ve put together a yml for installation.

2.1 Make sure the kernel version is greater than 4.14

1
2
3

uname -a

Linux node1 5.4.0-81-generic #91-Ubuntu SMP Thu Jul 15 19:09:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

2.2 Downloading Yaml files

1
2
3

git clone https://github.com/shaowenchen/demo

cd kindling/yaml

2.3 Installing Kindling

1

kubectl apply -f ./

2.4 Viewing Pod Status

1
2
3
4
5
6
7
8
9

kubectl -n kindling get pod

NAME                   READY   STATUS    RESTARTS   AGE
kindling-agent-8xt7c   1/1     Running   0          2d20h
kindling-agent-bvxzc   1/1     Running   0          2d20h
kindling-agent-l9phl   1/1     Running   0          2d20h
kindling-agent-nx5zh   1/1     Running   0          2d20h
kindling-agent-qd9cs   1/1     Running   0          2d20h
kindling-agent-sxglf   1/1     Running   0          2d20h

2.5 Pods may keep CrashLoopBackOff

Pod keeps CrashLoopBackOff, which is caused by a mismatch between the kindling-agent image and the current system. You need to recompile the image.

  • Install kernel headers

    Ubuntu Execute the following command.

    1

    apt-get -y install linux-headers-$(uname -r)

    CentOs executes the following command.

    1

    yum -y install kernel-devel-$(uname -r)

  • Compile and generate a new image

    1

    bash -c "$(curl -fsSL https://k8s-bpf-probes-public.oss-cn-hangzhou.aliyuncs.com/recompile-module.sh)"

  • Tag the image to the image defined in Yaml

    1

    docker tag kindlingproject/kindling-agent:bymyself shaowenchen/kindling-agent:ubuntu-20.04

  • Restart the Pod and you’re done!

    1

    kubeclt -n kindling delete pod kindling-agent-xxx

  • Tips for replacing kindling-agent images

    You need to change the mirror pull policy of the kindling-agent to IfNotPresent.

    1

        imagePullPolicy: IfNotPresent

Note here that if your infrastructure is relatively uniform, with only one OS and one kernel version, then you can tag the image as your own private image and push it to the remote end by modifying the kindling-agent Daemonset image address.

If your infrastructure is not uniform, and a cluster contains multiple hosts, multiple operating systems, and multiple kernel versions, then you can compile on each of these special systems. Since Daemonset can only set one image name, you need to keep the name of the kindling-agent image consistent across all Kubernetes nodes recompiled.

3. Installing the Grafana Plugin and Importing the Panel

3.1 Installing the topo-plugin plugin

Since Grafana, which I commonly use, is deployed using Docker, installing the plugin is a bit complicated.

  1. Download the plugin

    1
2

    git clone https://github.com/shaowenchen/demo
cd kindling/dashboard

  2. Copy the plugin into the container

    1

    docker cp topo-plugin.tar.gz 392fe26ae57f:/var/lib/grafana/plugins/

    where 392fe26ae57f is the container ID of the Grafana run.

  3. Enter the container creation directory

    1

    docker exec -it 392fe26ae57f sh

  4. In the container, create a directory to decompress the plugin

    1
2
3
4
5
6
7
8
9

    cd /var/lib/grafana/plugins/

mkdir kindlingproject-topology-panel

mv topo-plugin.tar.gz kindlingproject-topology-panel/

cd kindlingproject-topology-panel/

tar xvf topo-plugin.tar.gz

  5. Configuration plug-in

    /etc/grafana/grafana.ini is a read-only file, so it needs to be copied outside the container, modified, and then copied to overwrite the original file.

    1

    docker cp 392fe26ae57f:/etc/grafana/grafana.ini grafana.ini

    Edit the grafana.ini file locally and add the following.

    1
2

    [plugins]
allow_loading_unsigned_plugins = kindlingproject-topology-panel

    Copy the modified grafana.ini file back to the container to overwrite the original file.

    1

    docker cp grafana.ini  392fe26ae57f:/etc/grafana/grafana.ini

  6. Restart Grafana

    1

    docker restart 392fe26ae57f

3.2 Importing Grafana Panels

I am using Grafana version 8.3.1.

The relevant Dashboard Json file is backed up at https://github.com/shaowenchen/demo/tree/master/kindling/dashboard.

Compared to the official Dashboard provided by Kindling, the DataSource field has been added to switch the data source, making it easier to view monitoring data on different clusters.

4. View the data reported by the kindling-agent

Here is the screenshot.

kindling-agent

kindling-agent

kindling-agent

kindling-agent

In the panel, you can see some information related to DNS, quaternions, and even the network topology between command spaces and Services. The resource consumption is also acceptable.

1
2
3
4
5
6
7
8
9

kubectl -n kindling top pod

NAME                   CPU(cores)   MEMORY(bytes)   
kindling-agent-8xt7c   32m          444Mi           
kindling-agent-bvxzc   31m          337Mi           
kindling-agent-l9phl   63m          413Mi           
kindling-agent-nx5zh   62m          255Mi           
kindling-agent-qd9cs   99m          452Mi           
kindling-agent-sxglf   34m          701Mi

In some of the graphs above, you can find some data is vacant and some fields show NOT_FOUND_INTERNAL, which is not a good experience for the project.

With the PromQL statement, I get the IP list of the cluster for external access.

1

count by (dst_ip) (kindling_trace_request_duration_nanoseconds{dst_ip!~"127..*|10..*|172..*"})

PromQL

5. Reference

  • http://www.kindling.space:33215/project-1/doc-35/
  • https://github.com/CloudDectective-Harmonycloud/kindling
  • https://www.chenshaowen.com/blog/insight-kubernetes-network-by-kindling.html