Running applications on different operating systems and processor architectures is a common scenario, so it is a common practice to build separate distributions for different platforms. This is not easy to achieve when the platform we use to develop the application is different from the target platform for deployment. For example, developing an application on an x86 architecture and deploying it to a machine on an ARM platform usually requires preparing the ARM platform infrastructure for development and compilation.
Distributing images for multiple deployments in a single build dramatically improves the efficiency of application delivery, and for scenarios where applications need to be deployed across platforms, building images across platforms with docker buildx is a fast and efficient solution.
Prerequisites
Most mirror hosting platforms support multi-platform mirrors, which means that a single tag in a mirror repository can contain multiple mirrors from different platforms. Take the python
image repository of docker hub
for example, the 3.9.6
tag contains 10 images for different systems and architectures (platform = system + architecture).
When pulling an image with cross-platform support via docker pull
or docker run
, docker
will automatically select the image that matches the current running platform. Thanks to this feature, when distributing mirrors cross-platform, we don’t need to do anything about the consumption of the mirrors, we only need to care about the production of the mirrors, i.e. how to build cross-platform mirrors.
docker buildx
The default docker build
command cannot do cross-platform build tasks, we need to extend its functionality by installing the buildx
plugin for the docker
command line. buildx
can use the additional features of build images provided by Moby BuildKit to create multiple builder instances, perform build tasks in parallel on multiple nodes, and build across platforms.
Enabling Buildx
Docker Desktop for macOS or Windows, and docker
for Linux distributions installed via the deb
or rpm
packages have buildx
built in and do not need to be installed separately.
If your docker
does not have the buildx
command, you can download the binary package to install it.
- First, find the binary for your platform from the release page of the Docker buildx project.
- Download the binaries locally and rename them to
docker-buildx
and move them to docker’s plugins directory~/.docker/cli-plugins
. - Grant executable permissions to the binaries.
If the local docker
version is higher than 19.03, it is easier to build and install it directly locally with the following command.
Building with buildx
is done as follows.
|
|
buildx
and docker build
commands are basically the same experience, and also support build
common options such as -t
, -f
, etc.
builder instances
docker buildx
manages the build configuration and nodes through the builder instance object. The command line sends the build task to the builder instance, which in turn assigns it to a qualified node for execution. We can create multiple builder instances based on the same docker
service application and make them available to different projects to isolate the configuration of each project, or create a builder instance for a set of remote docker
nodes to form a build array and quickly switch between arrays.
A builder instance can be created using the docker buildx create
command, which will create a new builder instance for the node with the currently used docker service. To use a remote node, you can specify the remote port with the DOCKER_HOST
environment variable when creating the example or switch to the remote node’s docker context
in advance. The following creates a new builder instance with a remote node and specifies its driver, target platform, and instance name via command line options.
docker buildx ls
will list all available builder instances and the nodes in the instance.
After an instance is created, you can continue to add new nodes to it by using the -append <node>
option of the docker buildx create
command to add nodes to the builder instance specified by the -name <builder>
option.
|
|
The docker buildx inspect
, docker buildx stop
and docker buildx rm
commands are used to manage the lifecycle of an instance.
docker buildx use <builder>
will switch to the specified builder instance.
Build drivers
The buildx instance performs build tasks in two ways, the two execution methods are referred to as using different drivers
.
docker
driver: executes builds using the BuildKit library integrated in the Docker service application.docker-container
driver: starts a container containing BuildKit and executes the build in the container.
The docker
driver cannot use a small number of buildx
features (such as building multiple platform images at the same time in a single run), and there is also a difference in the default output format of the images: the docker
driver outputs the build results directly to the docker
image directory (usually /var/lib/ overlay2
), and the output images can be listed by executing the docker images
command, while docker container
requires the --output
option to specify the output format as a mirror or other format.
In order to build images for multiple platforms at once, we will use the docker container
-driven builder instance below.
Cross-platform build strategies for ## buildx
Depending on the build node and target language, buildx
supports the following three cross-platform build strategies.
- create a lightweight virtual machine via QEMU’s user-state mode and build the image in the virtual machine system.
- add multiple nodes of different target platforms to a builder instance and build the corresponding platform image from the native nodes.
- build and cross-compile to different target architectures in phases.
While QEMU is typically used to emulate a full operating system, it can also run in user state mode: register a binary conversion handler with binfmt_misc
on the host system and dynamically translate the binary file at runtime, converting system calls from the target CPU architecture to the current system’s CPU architecture as needed. The end result is like running the target CPU architecture’s binaries in a virtual machine. qemu support is built into Docker Desktop, and other platforms that meet the requirements to run it can be installed in the following way.
|
|
This approach does not require any changes to the existing Dockerfile and is cheap to implement, but obviously not very efficient.
Adding native nodes from different system architectures to the builder instance can bring better support for cross-platform compilation and is more efficient, but requires sufficient infrastructure support.
If the build project uses a programming language that supports cross-compilation (e.g. C and Go), you can take advantage of the staged build feature provided by Dockerfile: first compile the binaries of the target architecture in the same architecture as the build node, and then copy these binaries to another image of the target architecture. A concrete example is implemented below using Go. This approach does not require additional hardware and yields better performance, but is only possible with specific programming languages.
Build multiple architectures at once Go image practice
Source code and Dockerfile
The following will be a simple Go project as an example, assuming that the sample program file main.go
reads as follows.
The Dockerfile that defines the build process is as follows.
The build process is divided into two phases.
- In one phase, we pull a
golang
image for the same platform as the current build node and compile it into a binary for the target architecture using Go’s cross-compilation features. - Then pull the
alpine
image for the target platform and copy the compilation results from the previous phase into the image.
Executing a cross-platform build
When executing the build command, in addition to specifying the image name, the other two important options are to specify the target platform and the output format.
The -docker buildx build
specifies the target platform for the build via the -platform
option, and the FROM command in Dockerfile without the -platform
flag will pull the base image with the target platform, and the resulting image will belong to the target platform. In addition, the value of this option can be used in Dockerfile with the BUILDPLATFORM
, TARGETPLATFORM
, BUILDARCH
and TARGETARCH
parameters. When using the docker-container
driver, this option accepts multiple comma-separated values as input to specify multiple target platforms at the same time, and the build results for all platforms are combined into a single list of images as output, so they cannot be directly output as local docker images
images.
docker buildx build
supports rich output behavior. The --output=[PATH,-,type=TYPE[,KEY=VALUE]
option allows you to specify the output type and path of the build result.
- local: the build result will be written to the local path specified by
dest
in file system format, such as--output type=local,dest=. /output
. - tar: the build result will be written to the local path specified by
dest
after packing. - oci: The build result is written to the local path specified by
dest
in OCI standard image format. - docker: The build result is written to the local path specified by
dest
in Docker standard image format or loaded intodocker
’s image repository. This option is not available when multiple target platforms are specified at the same time. - image: Export as an image or list of images, and support the
push=true
option to push directly to a remote repository, which can be used when specifying multiple target platforms. - registry: a compact representation of
type=image,push=true
.
For this example, we execute the following docker buildx build
command.
|
|
This command will build linux/amd64
, linux/arm64
and linux/arm
images in the current directory and push the output directly to the remote Aliyun image repository.
The build process can be broken down as follows.
docker
transfers the build context to the builder instance. 2.- builder builds images for each target platform specified by the
-platform
option on the command line, including pulling the base image and performing the build steps. 3. - export the build results, and the image file layer is pushed to the remote repository. 4.
- Generate a manifest JSON file and push it to the remote repository as a mirror tag.
Validate the build results
After running you can probe the images that have been pushed to the remote repository with docker buildx imagetools
at the end of the run.
|
|
Finally, pull and run images on different platforms with the latest
tag to verify that the build is correct. When using Docker Desktop, its own integrated virtualization feature allows you to run images from different platforms and pull images directly with sha256
values.
How to cross-compile Golang’s CGO projects
Support for cross-compiling to common operating systems and CPU architectures is one of the great advantages of Golang, but the solution in the above example only works with pure Go code, and the situation becomes more complicated if the project calls C code via cgo
.
Preparing the cross-compilation environment and dependencies
In order to compile C code to the target platform smoothly, you need to install the C cross-compiler (usually based on gcc
) for the target platform in the compilation environment.
Next, you need to install the C standard libraries for the target platform (usually the standard libraries are installed as dependencies of the cross-compiler and do not need to be installed separately), and depending on the dependencies of the C code you are calling, you may need to install some additional C dependencies (e.g. libopus-dev
or something like that).
We will use the official golang:1.14
image for the amd64
architecture as the base image for compilation, and the Linux distribution is Debian. assuming the target platform for cross-compilation is linux/arm64
, the cross-compiler to be prepared is gcc-aarch64-linux-gnu
and the C standard library is libc6-dev-arm64-cross
, and the installer is as follows.
libc6-dev-arm64-cross
will be installed at the same time.
Thanks to the multi-architecture installation capabilities provided by the Debian package manager dpkg
, if our code relies on non-standard libraries such as libopus-dev
, we can install their arm64
architecture installers via <library>:<architecture>
.
Cross-compiling CGO examples
Suppose the following sample code for cgo
is available.
The Dockerfile that will be used is as follows.
|
|
Dockerfile installs gcc-aarch64-linux-gnu
as a cross-compiler via apt-get
, and the example program is simple so no additional dependencies are needed. When running go build
for compilation, you need to specify the cross-compiler to use via the CC
and CC_FOR_TARGET
environment variables.
In order to perform multiple builds on the same Dockerfile (assuming the target architecture is only amd64
/ arm64
), the RUN
command at the bottom uses a trick to execute different build commands via Bash’s conditional syntax.
- If the target platform of the build task is
arm64
, specifyCC
andCC_FOR_TARGET
environment variables as installed cross-compilers (note that their values are different). - If the target platform for the build task is
amd64
, no cross-compiler-related variables are specified, and the defaultgcc
is used as the compiler.
The final command to execute the build using buildx is as follows.
|
|
Summary
With the help of the Buildx
plugin, we can use docker
to easily build cross-platform application images in the absence of infrastructure.
However, the default way of virtualizing target platform instructions via QEMU has a significant performance bottleneck, and if the language used to write the application supports cross-compilation, we can achieve higher efficiency by combining buildx
with cross-compilation.
This article concludes with a solution for an advanced scenario: how to cross-compile a Golang project that uses CGO, and gives an example of compiling to the linux/arm64
platform.