Now the deployment of the compilation process will certainly use Docker, regardless of testing and deployment are implemented in Docker as far as possible, to achieve environmental isolation. But how to shorten Docker in the compilation of Image time, this is a problem, this article to introduce you to an experimental feature is BuildKit.
Prepare beforehand
Since BuildKit is an experimental feature, the default installation of Docker will not enable this feature. Currently, only compiled Linux containers are supported. Please use the following method to start it:
1
|
DOCKER_BUILDKIT=1 docker build .
|
After placing the command, you will see that the whole output result is different, the interface looks better, and you can see how long each layer takes to compile.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
|
[+] Building 0.1s (15/15) FINISHED
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 545B 0.0s
=> [internal] load metadata for docker.io/library/golang:1.14-alpine 0.0s
=> [1/10] FROM docker.io/library/golang:1.14-alpine 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 184B 0.0s
=> CACHED [2/10] RUN apk add bash ca-certificates git gcc g++ libc-dev 0.0s
=> CACHED [3/10] WORKDIR /app 0.0s
=> CACHED [4/10] COPY go.mod . 0.0s
=> CACHED [5/10] COPY go.sum . 0.0s
=> CACHED [6/10] RUN go mod download 0.0s
=> CACHED [7/10] COPY main.go . 0.0s
=> CACHED [8/10] COPY foo/foo.go foo/ 0.0s
=> CACHED [9/10] COPY bar/bar.go bar/ 0.0s
=> CACHED [10/10] RUN go build -o /app -v -tags netgo -ldflags '-w -extldflags "-static"' . 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:6cc56539b3191d5efd87fb4d05181993d013411299b5cefb74047d2447b4d0c9 0.0s
=> => naming to docker.io/appleboy/demo 0.0s
|
For a detailed compilation step, please add --progress=plain to see the detailed process. In fact, I think the point is that each step actually adds time, which is very helpful in development or CI/CD process. In addition, you can add config to the docker daemon so that you don’t have to add the DOCKER_BUILDKIT environment variable.
1
2
3
4
5
6
7
|
{
"debug": true,
"experimental": true,
"features": {
"buildkit": true
}
}
|
Please remember to restart Docker for the new settings to take effect.
Compile without BuildKit
Let’s test the basic Go language example to see how much time is saved. The code can be found here, and the examples are below:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
|
package main
import (
"net/http"
"gin/bar"
"gin/foo"
"github.com/gin-gonic/gin"
)
func main() {
r := gin.Default()
r.GET("/ping", func(c *gin.Context) {
c.JSON(http.StatusOK, gin.H{
"message": "pong",
})
})
r.GET("/ping2", func(c *gin.Context) {
c.JSON(http.StatusOK, gin.H{
"message": "pong2",
})
})
r.GET("/ping100", func(c *gin.Context) {
c.JSON(http.StatusOK, gin.H{
"message": foo.Foo(),
})
})
r.GET("/ping101", func(c *gin.Context) {
c.JSON(http.StatusOK, gin.H{
"message": bar.Bar(),
})
})
r.Run() // listen and serve on 0.0.0.0:8080 (for windows "localhost:8080")
}
|
Then write the Dockerfile.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
FROM golang:1.14-alpine
LABEL maintainer="Bo-Yi Wu <appleboy.tw@gmail.com>"
RUN apk add bash ca-certificates git gcc g++ libc-dev
WORKDIR /app
# Force the go compiler to use modules
ENV GO111MODULE=on
# We want to populate the module cache based on the go.{mod,sum} files.
COPY go.mod .
COPY go.sum .
RUN go mod download
COPY main.go .
COPY foo/foo.go foo/
COPY bar/bar.go bar/
ENV GOOS=linux
ENV GOARCH=amd64
RUN go build -o /app -v -tags netgo -ldflags '-w -extldflags "-static"' .
CMD ["/app"]
|
As you can see, if there is no change in go.mode and go.sum, basically the go module file can be processed through the docker cache layer. But every time there is any change in the code, the final go build will be compiled from scratch, please see the result below:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
|
docker build --progress=plain -t appleboy/docker-demo -f Dockerfile .
#14 [10/10] RUN go build -o /app -v -tags netgo -ldflags '-w -extldflags "-s...
#14 0.391 gin/foo
#14 0.403 gin/bar
#14 0.412 github.com/go-playground/locales/currency
#14 0.438 github.com/gin-gonic/gin/internal/bytesconv
#14 0.441 github.com/go-playground/locales
#14 0.449 golang.org/x/sys/unix
#14 0.464 net
#14 0.471 github.com/gin-gonic/gin/internal/json
#14 0.508 github.com/go-playground/universal-translator
#14 0.511 github.com/leodido/go-urn
#14 0.694 github.com/golang/protobuf/proto
#14 0.754 gopkg.in/yaml.v2
#14 1.535 github.com/mattn/go-isatty
#14 1.789 net/textproto
#14 1.790 crypto/x509
#14 1.920 vendor/golang.org/x/net/http/httpproxy
#14 1.978 vendor/golang.org/x/net/http/httpguts
#14 2.019 github.com/go-playground/validator/v10
#14 2.434 crypto/tls
#14 3.043 net/http/httptrace
#14 3.085 net/http
#14 4.211 net/rpc
#14 4.212 github.com/gin-contrib/sse
#14 4.212 net/http/httputil
#14 4.372 github.com/ugorji/go/codec
#14 6.322 github.com/gin-gonic/gin/binding
#14 6.322 github.com/gin-gonic/gin/render
#14 6.517 github.com/gin-gonic/gin
#14 6.819 gin
#14 DONE 7.8s
|
It took a total of 7.8 seconds, but think about it, when you develop on your own computer, it will not take that long, but will only be compiled based on the corrected Go files, but how can you do that in the CI/CD process? In fact, we can find that the compiled files are cached in the computer. In Linux environment, it will be /root/.cache/go-build. How can we speed up the compilation with buildKit?
Compiling with BuildKit
Let’s take a look at how Dockerfile can be improved to speed up compilation. Take a look at the following
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
|
# syntax = docker/dockerfile:experimental
FROM golang:1.14-alpine
LABEL maintainer="Bo-Yi Wu <appleboy.tw@gmail.com>"
RUN --mount=type=cache,target=/var/cache/apk apk add bash ca-certificates git gcc g++ libc-dev
WORKDIR /app
# Force the go compiler to use modules
ENV GO111MODULE=on
# We want to populate the module cache based on the go.{mod,sum} files.
COPY go.mod .
COPY go.sum .
RUN --mount=type=cache,target=/go/pkg/mod go mod download
COPY main.go .
COPY foo/foo.go foo/
COPY bar/bar.go bar/
ENV GOOS=linux
ENV GOARCH=amd64
RUN --mount=type=cache,target=/go/pkg/mod --mount=type=cache,target=/root/.cache/go-build go build -o /app -v -tags netgo -ldflags '-w -extldflags "-static"' .
CMD ["/app"]
|
To the first line is necessary to fill in the following content.
1
|
# syntax = docker/dockerfile:experimental
|
Then use the -mount method to cache the file, which can be done at any RUN step. So you can see that it is used in go build.
1
2
|
RUN --mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/root/.cache/go-build
|
You can see that this step will cache all the files after go module and build, so that next time when compiling, the files will be automatically placed in the corresponding location by default to speed up the compilation process.
1
2
3
4
5
|
docker build --progress=plain -t appleboy/docker-buildkit -f Dockerfile.buildkit .
#16 [stage-0 10/10] RUN --mount=type=cache,target=/go/pkg/mod --mount=type=c...
#16 0.381 gin/foo
#16 0.447 gin
#16 DONE 1.2s
|
You can see that after modifying the file, the result of the compilation is exactly the same as on your own computer, shortening the time by six seconds, which is a lot of time saved in a large Go project.
Summary
Now the CI/CD tools are not sure they all support docker buildKit, so you may have to do your own experiments to try it out, like the GitHub Action official also does not support docker buildkit. If you are setting up all your own, you can basically use docker buildKit + docker cache-from together, which will save a lot of time. See here for code examples