Developing eBPF applications with Golang

In the previous article “Developing a Hello World level eBPF program from scratch using C”, we explained in detail how to develop an eBPF program (including its user state part) from scratch based on C and the libbpf library. That article was the basis for subsequent articles on eBPF program development, because until now the kernel state part of an eBPF program running in the kernel state had to be developed in C, no matter what language the user state part of the eBPF program was developed in. In this way, the competition between other programming languages is how to make the development of the user state part of eBPF programs easier, and Go is no exception.

The most active Go eBPF package used to develop the user state part of eBPF in the Go community would be the cilium project’s open source cilium/ebpf.

Isovalent, the company behind the > cilium project, is also one of the main drivers for the adoption of eBPF technology in the cloud-native space.

In this article we will talk about how to develop eBPF applications based on cilium/ebpf!

1. Exploring the cilium/ebpf project example

The cilium/ebpf project borrows the idea of libbpf-boostrap to build the user state part of the eBPF program by code generation with bpf program inline. In order to figure out how to develop eBPF programs based on cilium/ebpf, let’s first explore the sample code provided by the cilium/ebpf project.

Let’s first download and look at the structure of the ebpf example.

Download cilium/ebpf project

$ git clone https://github.com/cilium/ebpf.git
Cloning into 'ebpf'...
remote: Enumerating objects: 7054, done.
remote: Counting objects: 100% (183/183), done.
remote: Compressing objects: 100% (112/112), done.
remote: Total 7054 (delta 91), reused 124 (delta 69), pack-reused 6871
Receiving objects: 100% (7054/7054), 10.91 MiB | 265.00 KiB/s, done.
Resolving deltas: 100% (4871/4871), done.

Explore the ebpf project sample code structure

The ebpf example is in the examples directory. Let’s take a look at tracepoint_in_c as an example to see how it is organized.

$tree tracepoint_in_c
tracepoint_in_c
├── bpf_bpfeb.go
├── bpf_bpfeb.o
├── bpf_bpfel.go
├── bpf_bpfel.o
├── main.go
└── tracepoint.c

0 directories, 6 files

Judging from experience, tracepoint.c here corresponds to the kernel state part of the ebpf program, while main.go and bpf_bpfel.go/bpf_bpfeb.go are the user state part of the ebpf program, and as for bpf_bpfeb.o/bpf_bpfel.o should be some kind of intermediate target file.

Check this intermediate file by readingelf -a bpf_bpfeb.o.

$readelf -a bpf_bpfeb.o
ELF Header:
Magic:   7f 45 4c 46 02 02 01 00 00 00 00 00 00 00 00 00
Class:                             ELF64
Data:                              2's complement, big endian
Version:                           1 (current)
OS/ABI:                            UNIX - System V
ABI Version:                       0
Type:                              REL (Relocatable file)
Machine:                           Linux BPF
Version:                           0x1
Entry point address:               0x0
Start of program headers:          0 (bytes into file)
Start of section headers:          1968 (bytes into file)
Flags:                             0x0
Size of this header:               64 (bytes)
Size of program headers:           0 (bytes)
Number of program headers:         0
Size of section headers:           64 (bytes)
Number of section headers:         13
Section header string table index: 1
... ...

We see that this is an elf file containing linux bpf bytecode (Machine: Linux BPF).

After reading the documentation on cilium/ebpf, I figured out the relationship between these files and present it to you in the following schematic.

cilium/ebpf

The source code file of the ebpf program (e.g. tracepoint.c in the figure) is compiled (bpf2go calls clang) by bpf2go (a code generation tool provided by cilium/ebpf) into the ebpf bytecode files bpf_bpfeb.o (big end) and bpf_bpfel.o (small end). Then bpf2go will generate bpf_bpfeb.go or bpf_bpfel.go based on the ebpf bytecode file. The bytecode of the ebpf program will be embedded in these two go source files as binary data. Taking bpf_bpfel.go as an example, we can find the following in its code (using the go:embed feature).

1
2

//go:embed bpf_bpfel.o
var _BpfBytes []byte

main.go is the main program for the user state part of the ebpf program. Compiling main.go with either bpf_bpfeb.go or bpf_bpfel.go creates the ebpf program.

With an initial exploration of the cilium/ebpf project example, let’s build the ebpf sample code.

2. Build ebpf example code

cilium/ebpf provides a convenient build script, we just need to execute “make -C …” under ebpf/examples to build the sample code.

The make build process will start the build container based on the quay.io/cilium/ebpf-builder image.

If you are in China, you need to do a little modification to the Makefile content like below and add the GOPROXY environment variable, otherwise the go module can’t be pulled due to GFW.

$git diff ../Makefile
diff --git a/Makefile b/Makefile
index 3a1da88..d7b1712 100644
--- a/Makefile
+++ b/Makefile
@@ -48,6 +48,7 @@ container-all:
       ${CONTAINER_ENGINE} run --rm ${CONTAINER_RUN_ARGS} \
               -v "${REPODIR}":/ebpf -w /ebpf --env MAKEFLAGS \
               --env CFLAGS="-fdebug-prefix-map=/ebpf=." \
+               --env GOPROXY="https://goproxy.io" \
               --env HOME="/tmp" \
               "${IMAGE}:${VERSION}" \
               $(MAKE) all

Executing the build after this will give us the results we are looking for without any problems.

$ cd examples
$ make -C ..
make: Entering directory '/root/go/src/github.com/cilium/ebpf'
docker run --rm  --user "0:0" \
    -v "/root/go/src/github.com/cilium/ebpf":/ebpf -w /ebpf --env MAKEFLAGS \
    --env CFLAGS="-fdebug-prefix-map=/ebpf=." \
    --env GOPROXY="https://goproxy.io" \
    --env HOME="/tmp" \
    "quay.io/cilium/ebpf-builder:1648566014" \
    make all
make: Entering directory '/ebpf'
find . -type f -name "*.c" | xargs clang-format -i
go generate ./cmd/bpf2go/test
go: downloading golang.org/x/sys v0.0.0-20210906170528-6f6e22806c34
Compiled /ebpf/cmd/bpf2go/test/test_bpfel.o
Stripped /ebpf/cmd/bpf2go/test/test_bpfel.o
Wrote /ebpf/cmd/bpf2go/test/test_bpfel.go
Compiled /ebpf/cmd/bpf2go/test/test_bpfeb.o
Stripped /ebpf/cmd/bpf2go/test/test_bpfeb.o
Wrote /ebpf/cmd/bpf2go/test/test_bpfeb.go
go generate ./internal/sys
enum AdjRoomMode
enum AttachType
enum Cmd
enum FunctionId
enum HdrStartOff
enum LinkType
enum MapType
enum ProgType
enum RetCode
enum SkAction
enum StackBuildIdStatus
enum StatsType
enum XdpAction
struct BtfInfo
... ...
attr ProgRun
attr RawTracepointOpen
cd examples/ && go generate ./...
go: downloading github.com/cilium/ebpf v0.8.2-0.20220424153111-6da9518107a8
go: downloading golang.org/x/sys v0.0.0-20211001092434-39dca1131b70
Compiled /ebpf/examples/cgroup_skb/bpf_bpfel.o
Stripped /ebpf/examples/cgroup_skb/bpf_bpfel.o
Wrote /ebpf/examples/cgroup_skb/bpf_bpfel.go
Compiled /ebpf/examples/cgroup_skb/bpf_bpfeb.o
Stripped /ebpf/examples/cgroup_skb/bpf_bpfeb.o
Wrote /ebpf/examples/cgroup_skb/bpf_bpfeb.go
Compiled /ebpf/examples/fentry/bpf_bpfeb.o
Stripped /ebpf/examples/fentry/bpf_bpfeb.o
Wrote /ebpf/examples/fentry/bpf_bpfeb.go
Compiled /ebpf/examples/fentry/bpf_bpfel.o
Stripped /ebpf/examples/fentry/bpf_bpfel.o
Wrote /ebpf/examples/fentry/bpf_bpfel.go
Compiled /ebpf/examples/kprobe/bpf_bpfel.o
Stripped /ebpf/examples/kprobe/bpf_bpfel.o
Wrote /ebpf/examples/kprobe/bpf_bpfel.go
Stripped /ebpf/examples/uretprobe/bpf_bpfel_x86.o
... ...
Wrote /ebpf/examples/uretprobe/bpf_bpfel_x86.go
ln -srf testdata/loader-clang-14-el.elf testdata/loader-el.elf
ln -srf testdata/loader-clang-14-eb.elf testdata/loader-eb.elf
make: Leaving directory '/ebpf'
make: Leaving directory '/root/go/src/github.com/cilium/ebpf'

Taking ebpf under uretprobe as an example, let’s run it.

1
2

$go run -exec sudo uretprobe/*.go
2022/06/05 18:23:23 Listening for events..

Open a new terminal and execute vi .bashrc in the user home directory. in the above execution window of the uretprobe program we can see the following output.

1
2

2022/06/05 18:24:34 Listening for events..
2022/06/05 18:24:42 /bin/bash:readline return value: vi .bashrc

This indicates that the ebpf program under uretprobe is executing as expected.

3. Using cilium/ebpf to develop the user state part for the previous Hello World eBPF program

With an initial understanding of the cilium/ebpf sample program, let’s develop the user state part of the hello world ebpf program from the previous article “Developing a Hello World level eBPF program from scratch using C language”.

Review the C source code of that hello world ebpf program.

// github.com/bigwhite/experiments/tree/master/ebpf-examples/helloworld-go/helloworld.bpf.c
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>

SEC("tracepoint/syscalls/sys_enter_execve")

int bpf_prog(void *ctx) {
  char msg[] = "Hello, World!";
  bpf_printk("invoke bpf_prog: %s\n", msg);
  return 0;
}

char LICENSE[] SEC("license") = "Dual BSD/GPL";

Once this ebpf program is loaded into the kernel, it will be called once whenever the execve system call is executed, and we will see the corresponding log output in /sys/kernel/debug/tracing/trace_pipe.

3.1. Converting ebpf core state programs to Go code using bpf2go

Based on the “routine” we got when exploring the cilium/ebpf example program, the first thing we need to do is to convert helloworld.bpf.c into a Go code file, and the indispensable tool for this conversion process is the cilium/ebpf bpf2go tool, let’s install the tool first.

`1`	`$go install github.com/cilium/ebpf/cmd/bpf2go@latest`

Next, we can directly use the bpf2go tool to convert helloworld.ebpf.c to the corresponding go source file.

$GOPACKAGE=main bpf2go -cc clang-10 -cflags '-O2 -g -Wall -Werror' -target bpfel,bpfeb bpf helloworld.bpf.c -- -I /home/tonybai/test/ebpf/libbpf/include/uapi -I /usr/local/bpf/include -idirafter /usr/local/include -idirafter /usr/lib/llvm-10/lib/clang/10.0.0/include -idirafter /usr/include/x86_64-linux-gnu -idirafter /usr/include

Compiled /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfel.o
Stripped /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfel.o
Wrote /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfel.go
Compiled /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfeb.o
Stripped /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfeb.o
Wrote /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfeb.go

But there is a problem here, that is, the bpf2go command line is followed by a series of header files provided to the clang compiler with reference paths to the Makefile in the article “Developing a Hello World-level eBPF program from scratch using C. If we follow these header file paths, although the bpf2go conversion can work, we need to depend on and install the libbpf library, which is obviously not what we want.

cilium/ebpf provides a headers directory in examples that contains all the headers needed to develop the user state part of the ebpf program, and we use it as our header reference path. To build ebpf based on this headers directory, however, we need to modify the include statement in helloworld.bpf.c.

The original include statement.

1
2

#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>

The modified include statement.

`1`	`#include "common.h"`

Next we will perform the conversion with the bpf2go tool.

$GOPACKAGE=main bpf2go -cc clang-10 -cflags '-O2 -g -Wall -Werror' -target bpfel,bpfeb bpf helloworld.bpf.c -- -I /home/tonybai/go/src/github.com/cilium/ebpf/examples/headers

Compiled /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfel.o
Stripped /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfel.o
Wrote /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfel.go
Compiled /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfeb.o
Stripped /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfeb.o
Wrote /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfeb.go

We see that bpf2go smoothly generates ebpf bytecode with the corresponding Go source files.

3.2. Building the user state part of the helloworld ebpf program

The following is the main.go source code of the user state part of the helloword ebpf program built by referring to the cilium/ebpf example.

// github.com/bigwhite/experiments/ebpf-examples/helloworld-go/main.go
package main

import (
    "log"
    "os"
    "os/signal"
    "syscall"

    "github.com/cilium/ebpf/link"
    "github.com/cilium/ebpf/rlimit"
)

func main() {
    stopper := make(chan os.Signal, 1)
    signal.Notify(stopper, os.Interrupt, syscall.SIGTERM)

    // Allow the current process to lock memory for eBPF resources.
    if err := rlimit.RemoveMemlock(); err != nil {
        log.Fatal(err)
    }

    // Load pre-compiled programs and maps into the kernel.
    objs := bpfObjects{}
    if err := loadBpfObjects(&objs, nil); err != nil {
        log.Fatalf("loading objects: %s", err)
    }
    defer objs.Close()

    //SEC("tracepoint/syscalls/sys_enter_execve")
    // attach to xxx
    kp, err := link.Tracepoint("syscalls", "sys_enter_execve", objs.BpfProg, nil)
    if err != nil {
        log.Fatalf("opening tracepoint: %s", err)
    }
    defer kp.Close()

    log.Printf("Successfully started! Please run \"sudo cat /sys/kernel/debug/tracing/trace_pipe\" to see output of the BPF programs\n")

    // Wait for a signal and close the perf reader,
    // which will interrupt rd.Read() and make the program exit.
    <-stopper
    log.Println("Received signal, exiting program..")
}

We know that an ebpf program has several key components.

ebpf program data
map: used for data interaction between user state and kernel state
attach point

According to the description in cilium/ebpf architecture, the ebpf package abstracts the first two parts into a single data structure, bpfObjects.

// github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfel.go

// bpfObjects contains all objects after they have been loaded into the kernel.
//
// It can be passed to loadBpfObjects or ebpf.CollectionSpec.LoadAndAssign.
type bpfObjects struct {
    bpfPrograms
    bpfMaps
}

We see that the main function loads the ebpf program into the kernel via the generated loadBpfObjects function and populates the bpfObjects structure. Once the bpf program is loaded successfully, we can then use the fields in the bpfObjects structure to perform the rest of the operations, such as docking the bpf program to the target pendant node via a function in the link package (e.g., the link.Tracepoint function in the text). This way, bpf can be executed via callbacks after the corresponding event.

Compile and execute the helloworld example below.

1
2
3

$go run -exec sudo main.go bpf_bpfel.go
[sudo] password for tonybai:
2022/06/05 14:12:40 Successfully started! Please run "sudo cat /sys/kernel/debug/tracing/trace_pipe" to see output of the BPF programs

After that, open a new terminal and execute sudo cat /sys/kernel/debug/tracing/trace_pipe. When execve is called, we can see a log output like the following.

<...>-551077  [000] .... 6062226.208943: 0: invoke bpf_prog: Hello, World!
<...>-551077  [000] .... 6062226.209098: 0: invoke bpf_prog: Hello, World!
<...>-551079  [007] .... 6062226.215421: 0: invoke bpf_prog: Hello, World!
<...>-551079  [007] .... 6062226.215578: 0: invoke bpf_prog: Hello, World!
<...>-554756  [007] .... 6063476.785212: 0: invoke bpf_prog: Hello, World!
<...>-554756  [007] .... 6063476.785378: 0: invoke bpf_prog: Hello, World!

3.3. Using go generate to drive the conversion of bpf2go

In terms of code generation, the Go toolchain provides the go generate tool natively, and the cilium/ebpf examples also use go generate to drive bpf2go to convert bpf programs to Go source files. Let’s do a little transformation here as well.

First we add a go:generate instruction line to the main function of main.go.

// github.com/bigwhite/experiments/ebpf-examples/helloworld-go/main.go

// $BPF_CLANG, $BPF_CFLAGS and $BPF_HEADERS are set by the Makefile.
//go:generate bpf2go -cc $BPF_CLANG -cflags $BPF_CFLAGS -target bpfel,bpfeb bpf helloworld.bpf.c -- -I $BPF_HEADERS
func main() {
    stopper := make(chan os.Signal,  1)
    ... ...
}

This way when we explicitly execute the go generate statement, go generate will scan for that instruction statement and execute the command that follows. Several variables are used here, which are defined in the Makefile. Of course, if you don’t want to use the Makefile, you can replace the variables with the corresponding values. Here we use the Makefile, and here is the contents of the Makefile.

// github.com/bigwhite/experiments/ebpf-examples/helloworld-go/Makefile

CLANG ?= clang-10
CFLAGS ?= -O2 -g -Wall -Werror

LIBEBPF_TOP = /home/tonybai/go/src/github.com/cilium/ebpf
EXAMPLES_HEADERS = $(LIBEBPF_TOP)/examples/headers

all: generate

generate: export BPF_CLANG=$(CLANG)
generate: export BPF_CFLAGS=$(CFLAGS)
generate: export BPF_HEADERS=$(EXAMPLES_HEADERS)
generate:
    go generate ./...

With this Makefile, we can execute the make command to convert bpf2go to bpf programs.

$make
go generate ./...
Compiled /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfel.o
Stripped /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfel.o
Wrote /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfel.go
Compiled /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfeb.o
Stripped /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfeb.o
Wrote /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfeb.go

4. Summary

In this article we have explained how to develop the user state part of ebpf based on the cilium/ebpf package.
ebpf borrows the idea of libbpf to build the user state part of ebpf by generating code with data inline.
ebpf provides the bpf2go tool to convert the C source code of bpf to the corresponding go source code.
ebpf abstracts the bpf program into bpfObjects, completes the loading of the bpf program into the kernel by generating loadBpfObjects, and then implements the association of ebpf with kernel events using packages such as link provided by the ebpf library.
There are many more ways to play with ebpf packages, and this one is just to lay the groundwork. In subsequent articles, we will do further study and explanation for various types of bpf programs.
The code for this article can be downloaded at here.

Table of Contents