Running applications on different operating systems and processor architectures is a common scenario, so it is a common practice to build separate distributions for different platforms. This is not easy to achieve when the platform we use to develop the application is different from the target platform for deployment. For example, developing an application on an x86 architecture and deploying it to a machine on an ARM platform usually requires preparing the ARM platform infrastructure for development and compilation.

Building multiple deployments at once with image distribution dramatically improves the efficiency of application delivery, and building cross-platform images with docker buildx is a fast and efficient solution for scenarios that require cross-platform application deployment.

Prerequisites

Most image hosting platforms support multi-platform images, which means that a single tag in a image repository can contain multiple images for different platforms, take docker hub’s python image repository for example, the tag 3.9.6 contains 10 images for different systems and architectures (platform = system + architecture).

python image

When pulling an image with cross-platform support via docker pull or docker run, docker will automatically select the image that matches the current running platform. Thanks to this feature, when distributing images across platforms, we don’t need to do anything about the consumption of the images, we only need to care about the production of the images, i.e. how to build cross-platform images.

docker buildx

The default docker build command cannot do cross-platform build tasks, so we need to extend its functionality by installing the buildx plugin for the docker command line. buildx can use the additional features of build images provided by Moby BuildKit to create multiple builder instances, perform build tasks in parallel on multiple nodes, and build across platforms.

Enabling Buildx

Docker Desktop for macOS or Windows, and docker for Linux distributions installed via the deb or rpm packages have buildx built in and do not need to be installed separately.

If your docker does not have the buildx command, you can download the binary package to install it.

  1. First, find the binary for your platform from the release page of the Docker buildx project.
  2. Download the binaries locally and rename them to docker-buildx and move them to docker’s plugins directory ~/.docker/cli-plugins.
  3. Grant executable permissions to the binaries.

If your local docker version is higher than 19.03, you can build and install it directly locally by using the following command, which is more convenient.

1
2
3
4

$ export DOCKER_BUILDKIT=1
$ docker build --platform=local -o . git://github.com/docker/buildx
$ mkdir -p ~/.docker/cli-plugins
$ mv buildx ~/.docker/cli-plugins/docker-buildx

The way to build with buildx is as follows.

1

docker buildx build

The experience of using buildx and docker build commands is basically the same, and it also supports build common options such as -t, -f, etc.

builder instance

docker buildx manages build configurations and nodes through a builder instance object, where the command line sends build tasks to the builder instance, which in turn assigns them to eligible nodes for execution. We can create multiple builder instances based on the same docker service application and make them available to different projects to isolate the configuration of each project, or create a builder instance for a set of remote docker nodes to form a build array and quickly switch between arrays.

A builder instance can be created using the docker buildx create command, which will create a new builder instance for the node with the currently used docker service. To use a remote node, you can specify the remote port with the DOCKER_HOST environment variable when creating the example or switch to the remote node’s docker context in advance. The following creates a new builder instance with a remote node and specifies its driver, target platform, and instance name via command line options.

1
2
3

$ export DOCKER_HOST=tcp://10.10.150.66:2375
$ docker buildx create --driver docker-container --platform linux/amd64,linux/arm64 --name remote-builder
remote-builder

docker buildx ls will list all available builder instances and the nodes in the instance.

1
2
3
4
5
6

$ docker buildx ls
NAME/NODE         DRIVER/ENDPOINT         STATUS   PLATFORMS
remote-builder    docker-container                 
  remote-builder0 tcp://10.10.150.66:2375 inactive linux/amd64*, linux/arm64*
default *         docker                           
  default         default                 running  linux/amd64, linux/386

After an instance is created, you can continue to add new nodes to it by using the -append <node> option of the docker buildx create command to add nodes to the builder instance specified by the -name <builder> option.

1

$ docker buildx create --name default --append remote-builder0

The docker buildx inspect, docker buildx stop and docker buildx rm commands are used to manage the lifecycle of an instance.

docker buildx use <builder> will switch to the specified builder instance.

Build drivers

The buildx instance performs build tasks in two ways, the two execution methods are referred to as using different drivers.

  • docker driver: executes a build using the BuildKit library integrated in the Docker service application.
  • docker-container driver: starts a container containing BuildKit and executes the build in the container.

The docker driver cannot use a small number of buildx features (such as building multiple platform images at the same time in a single run), and there is also a difference in the default output format of the images: the docker driver outputs the build results directly to the docker image directory (usually /var/lib/ overlay2), and the output images can be listed by executing the docker images command, while docker container requires the --output option to specify the output format as a image or other format.

In order to build images for multiple platforms at once, we will use the docker container-driven builder instance below.

buildx’s cross-platform build strategy

Depending on the build node and target language, buildx supports the following three cross-platform build strategies.

  1. create a lightweight virtual machine via QEMU’s user-state mode and build the image in the virtual machine system.
  2. add multiple nodes of different target platforms to a builder instance and build the corresponding platform image from the native nodes.
  3. build and cross-compile to different target architectures in phases.

While QEMU is typically used to emulate a full operating system, it can also run in user state mode: register a binary conversion handler with binfmt_misc on the host system and dynamically translate the binary file at runtime, converting system calls from the target CPU architecture to the current system’s CPU architecture as needed. The end result is like running the target CPU architecture’s binaries in a virtual machine. qemu support is built into Docker Desktop, and other platforms that meet the requirements to run it can be installed in the following way.

Depending on the build node and target program language, buildx supports the following three cross-platform build strategies.

  1. create a lightweight virtual machine via QEMU’s user-state mode and build the image in the virtual machine system.
  2. add multiple nodes of different target platforms to a builder instance and build the corresponding platform image from the native nodes.
  3. build and cross-compile to different target architectures in phases.

While QEMU is typically used to emulate a full operating system, it can also run in user state mode: register a binary conversion handler with binfmt_misc on the host system and dynamically translate the binary file at runtime, converting system calls from the target CPU architecture to the current system’s CPU architecture as needed. The end result is like running the target CPU architecture’s binaries in a virtual machine. qemu support is built into Docker Desktop, and other platforms that meet the requirements to run it can be installed in the following way.

1

$ docker run --privileged --rm tonistiigi/binfmt --install all

This approach does not require any changes to the existing Dockerfile and is cheap to implement, but obviously not very efficient.

Adding native nodes from different system architectures to the builder instance can bring better support for cross-platform compilation and is more efficient, but requires sufficient infrastructure support.

If the build project uses a programming language that supports cross-compilation (e.g. C and Go), you can take advantage of the staged build feature provided by Dockerfile: first compile the binaries of the target architecture in the same architecture as the build node, and then copy these binaries to another image of the target architecture. A concrete example is implemented below using Go. This approach does not require additional hardware and yields better performance, but is only possible with specific programming languages.

Practice building multiple architecture Go images at once

Source code and Dockerfile

The following is an example of a simple Go project, assuming the sample program file main.go has the following contents.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

package main

import (
    "fmt"
    "runtime"
)

func main() {
    fmt.Println("Hello world!")
    fmt.Printf("Running in [%s] architecture.\n", runtime.GOARCH)
}

The Dockerfile that defines the build process is as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

FROM --platform=$BUILDPLATFORM golang:1.14 as builder

ARG TARGETARCH

WORKDIR /app
COPY main.go /app/main.go
RUN GOOS=linux GOARCH=$TARGETARCH go build -a -o output/main main.go

FROM alpine:latest
WORKDIR /root
COPY --from=builder /app/output/main .
CMD /root/main

The build process is divided into two phases.

  • In one phase, we pull a golang image for the same platform as the current build node and compile it into a binary for the target architecture using Go’s cross-compilation features.
  • Then we pull the alpine image of the target platform and copy the compilation results from the previous phase into the image.

Execute cross-platform builds

When executing the build command, in addition to specifying the image name, two other important options are to specify the target platform and the output format.

The -docker buildx build specifies the target platform for the build with the -platform option, and the FROM command in Dockerfile without the -platform flag will pull the base image with the target platform, and the resulting image will belong to the target platform. In addition, the value of this option can be used in Dockerfile with the BUILDPLATFORM, TARGETPLATFORM, BUILDARCH and TARGETARCH parameters. When using the docker-container driver, this option can accept multiple comma-separated values as input to specify multiple target platforms at the same time, and the build results for all platforms will be combined into a single list of images as output, so they cannot be directly output as local docker images images.

docker buildx build supports rich output behavior. The --output=[PATH,-,type=TYPE[,KEY=VALUE] option allows you to specify the output type and path of the build result, etc. The following output types are commonly used.

  • local: the build result will be written to the local path specified by dest in file system format, such as --output type=local,dest=. /output.
  • tar: the build result will be written to the local path specified by dest after packing.
  • oci: The build result is written to the local path specified by dest in OCI standard image format.
  • docker: The build result is written to the local path specified by dest in Docker standard image format or loaded into docker’s image repository. This option is not available when multiple target platforms are specified at the same time.
  • image: Export as an image or list of images, and support the push=true option to push directly to a remote repository, which can be used when multiple target platforms are specified at the same time.
  • registry: a compact representation of type=image,push=true.

For this example, we execute the following docker buildx build command.

1

$ docker buildx build --platform linux/amd64,linux/arm64,linux/arm -t registry.cn-hangzhou.aliyuncs.com/waynerv/arch-demo -o type=registry .

This command will build linux/amd64, linux/arm64 and linux/arm images in the current directory and push the output directly to the remote Aliyun image repository.

The build process can be broken down as follows.

  1. docker transfers the build context to the builder instance.
  2. builder builds images for each target platform specified by the -platform option on the command line, including pulling the base image and performing the build steps.
  3. export the build results, and the image file layer is pushed to the remote repository.
  4. generate a manifest JSON file and push it to the remote repository as a image tag.

Verify the build results

After running, you can probe the images that have been pushed to the remote repository with docker buildx imagetools.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

$ docker buildx imagetools inspect registry.cn-hangzhou.aliyuncs.com/waynerv/arch-demo:latest
Name:      registry.cn-hangzhou.aliyuncs.com/waynerv/arch-demo:latest
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest:    sha256:e2c3c5b330c19ac9d09f8aaccc40224f8673e12b88ff59cb68971c36b76e95ca
           
Manifests: 
  Name:      registry.cn-hangzhou.aliyuncs.com/waynerv/arch-demo:latest@sha256:cb6a7614ee3db03c8858e3680b1585f32a6fe3de9b371e37e25cf42a83f6e0ba
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/amd64
             
  Name:      registry.cn-hangzhou.aliyuncs.com/waynerv/arch-demo:latest@sha256:034aa0077a452a6c2585f8b4969c7c85d5d2bf65f801fcc803a00d0879ce900e
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm64
             
  Name:      registry.cn-hangzhou.aliyuncs.com/waynerv/arch-demo:latest@sha256:db0ee3a876fb789d2e733471385eef0a056f64ee12d9e7ef94e411469d054eb5
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm/v7

Finally, pull and run images on different platforms with the latest tag to verify that the build is correct. When using Docker Desktop, its own integrated virtualization feature allows you to run images from different platforms and pull images directly with sha256 values.

1
2
3

$ docker run --rm registry.cn-hangzhou.aliyuncs.com/waynerv/arch-demo:latest@sha256:cb6a7614ee3db03c8858e3680b1585f32a6fe3de9b371e37e25cf42a83f6e0ba
Hello world!
Running in [amd64] architecture.

1
2
3

$ docker run --rm registry.cn-hangzhou.aliyuncs.com/waynerv/arch-demo:latest@sha256:034aa0077a452a6c2585f8b4969c7c85d5d2bf65f801fcc803a00d0879ce900e
Hello world!
Running in [arm64] architecture.

How to cross-compile Golang’s CGO projects

Supporting cross-compilation to common operating systems and CPU architectures is one of Golang’s strengths, but the solution in the above example only works with pure Go code, which becomes more complicated if the project calls C code via cgo.

Preparing the cross-compiler environment and dependencies

In order to be able to compile C code to the target platform smoothly, you first need to install the C cross-compiler for the target platform (usually based on gcc) in the compilation environment. Common Linux distributions provide cross-compiler installation packages for most platforms, which can be installed directly through the package manager.

Next, you need to install the C standard libraries for the target platform (usually the standard libraries are installed as dependencies of the cross-compiler and do not need to be installed separately), and depending on the dependencies of the C code you are calling, you may need to install some additional C dependencies (e.g. libopus-dev or something like that).

We will use the official golang:1.14 image for the amd64 architecture as the base image for compilation, and the Linux distribution is Debian. assuming the target platform for cross-compilation is linux/arm64, the cross-compiler to be prepared is gcc-aarch64-linux-gnu and the C standard library is libc6-dev-arm64-cross, which is installed as follows.

1
2

$ apt-get update
$ apt-get install gcc-aarch64-linux-gnu

libc6-dev-arm64-cross will be installed at the same time.

Thanks to the multi-architecture installation capabilities provided by the Debian package manager dpkg, if our code relies on non-standard libraries such as libopus-dev, we can install their arm64 architecture installers via <library>:<architecture>.

1
2
3

$ dpkg --add-architecture arm64
$ apt-get update
$ apt-get install -y libopus-dev:arm64

Cross-compiling CGO examples

Suppose you have the following sample code for cgo.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

package main

/*
#include <stdlib.h>
*/
import "C"
import "fmt"

func Random() int {
    return int(C.random())
}

func Seed(i int) {
    C.srandom(C.uint(i))
}

func main()  {
    rand := Random()
    fmt.Printf("Hello %d\n", rand)
}

The Dockerfile that will be used is as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

FROM --platform=$BUILDPLATFORM golang:1.14 as builder

ARG TARGETARCH
RUN apt-get update && apt-get install -y gcc-aarch64-linux-gnu

WORKDIR /app
COPY . /app/

RUN if [ "$TARGETARCH" = "arm64" ]; then CC=aarch64-linux-gnu-gcc && CC_FOR_TARGET=gcc-aarch64-linux-gnu; fi && \
  CGO_ENABLED=1 GOOS=linux GOARCH=$TARGETARCH CC=$CC CC_FOR_TARGET=$CC_FOR_TARGET go build -a -ldflags '-extldflags "-static"' -o /main main.go

Dockerfile installs gcc-aarch64-linux-gnu as a cross-compiler via apt-get, and the example program is simple so no additional dependencies are needed. When running go build for compilation, you need to specify the cross-compiler to use via the CC and CC_FOR_TARGET environment variables.

In order to perform multiple builds on the same Dockerfile (assuming the target architecture is only amd64 / arm64), the RUN command at the bottom uses a trick to execute different build commands via Bash’s conditional syntax.

  • If the target platform of the build task is arm64, specify CC and CC_FOR_TARGET environment variables as installed cross-compilers (note that their values are different).
  • If the target platform for the build task is amd64, no cross-compiler-related variables are specified, and the default gcc is used as the compiler.

The final command to execute the build using buildx is as follows.

1

$ docker buildx build --platform linux/amd64,linux/arm64 -t registry.cn-hangzhou.aliyuncs.com/waynerv/cgo-demo -o type=registry .

Summary

With the help of the Buildx plugin, we can use docker to easily build cross-platform application images in the absence of infrastructure.

However, the default way of virtualizing target platform instructions via QEMU has a significant performance bottleneck, and if the language used to write the application supports cross-compilation, we can achieve higher efficiency by combining buildx with cross-compilation.

This article concludes with a solution for an advanced scenario: how to cross-compile a Golang project that uses CGO, and gives an example of compiling to the linux/arm64 platform.