CI/CD Flow

Nowadays, everyone is definitely containerizing their services, and how to effectively manage and upgrade the containers without affecting the existing services is an important issue. However, in the CI/CD flow, there are definitely two steps that are necessary, the first is to package the environment into a Docker Image and upload it to the company’s private Docker Registry, after uploading, connect to the machine via SSH and pull the new image file, and then restart the running service through the Graceful Shutdown mechanism.

This article is to bring you a new tool Watchtower to automatically update the running containers, so that the CD process can be simplified one more step, the developer just upload the Docker Image, the remote server can automatically update.

The architecture will look like this.

Watchtower architecture

What is Watchtower

Watchtower is an application developed in Go that monitors running Docker containers and watches for changes to the Docker Image used when they were originally started. If watchtower detects that the image has changed, it will automatically restart the container with the new image.

With watchtower, developers can simply update the running version of a containerized application by pushing a new image file to Docker Hub or your own Docker Registry. watchtower will download your new image, gracefully shut down the existing container, and then restart it using the same options used for the initial deployment.

For example, suppose you are running watchtower and an instance of an image called ghcr.io/go-training/example53.

Every few minutes, watchtower will download the latest ghcr.io/go-training/example53 image file and compare it to the image used to run the “example53” container. If it finds that the image has changed, it will stop/delete the “example53” container and restart it with the new image and the same docker run option that was used to start the container in the first place.

Usage

Watchtower itself is packaged as a Docker container, so installation is very simple, just pull the containrrrr/watchtower image. If you are using an ARM-based architecture, please pull the appropriate containrrrr/watchtower:armhf-tag image from Docker Hub.

Since the watchtower code needs to interact with the Docker API to monitor the running container, you need to mount /var/run/docker.sock to the container using the -v flag when running the container.

Run the watchtower container with the following command.

1
2
3
4

docker run -d \
  --name watchtower \
  -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower

If pulling an image file from the private Docker registry, use the environment variables REPO_USER and REPO_PASS or mount the host docker profile to the container (in the root directory / of the container file system).

1
2
3
4
5
6

docker run -d \
  --name watchtower \
  -e REPO_USER=username \
  -e REPO_PASS=password \
  -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower container_to_watch --debug

Alternatively, if you have 2FA authentication set up on Docker Hub, providing an account and password will not be sufficient. Instead, you can run the docker login command to store the credentials in the $HOME/.docker/config.json file, and then mount this configuration file to make it available to Watchtower containers.

1
2
3
4
5

docker run -d \
  --name watchtower \
  -v $HOME/.docker/config.json:/config.json \
  -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower container_to_watch --debug

Example

Next, we use docker-compose to test the running container.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

version: "3"
services:
  example53:
    image: ghcr.io/go-training/example53:latest
    restart: always
    labels:
      - "com.centurylinklabs.watchtower.enable=true"
    ports:
      - "8080:8080"

  watchtower:
    image: containrrr/watchtower
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command: --interval 5

After booting, you can see the following Log message.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

example53_1   | [GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
example53_1   |
example53_1   | [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
example53_1   |  - using env: export GIN_MODE=release
example53_1   |  - using code: gin.SetMode(gin.ReleaseMode)
example53_1   |
example53_1   | [GIN-debug] GET    /ping                     --> main.main.func1 (3 handlers)
example53_1   | [GIN-debug] GET    /                         --> main.main.func2 (3 handlers)
example53_1   | [GIN-debug] [WARNING] You trusted all proxies, this is NOT safe. We recommend you to set a value.
example53_1   | Please check https://pkg.go.dev/github.com/gin-gonic/gin#readme-don-t-trust-all-proxies for details.
example53_1   | [GIN-debug] Environment variable PORT is undefined. Using port :8080 by default
example53_1   | [GIN-debug] Listening and serving HTTP on :8080
watchtower_1  | time="2023-03-02T01:13:07Z" level=info msg="Watchtower 1.5.3"
watchtower_1  | time="2023-03-02T01:13:07Z" level=info msg="Using no notifications"
watchtower_1  | time="2023-03-02T01:13:07Z" level=info msg="Checking all containers (except explicitly disabled with label)"
watchtower_1  | time="2023-03-02T01:13:07Z" level=info msg="Scheduling first run: 2023-03-02 01:13:12 +0000 UTC"
watchtower_1  | time="2023-03-02T01:13:07Z" level=info msg="Note that the first check will be performed in 4 seconds"
watchtower_1  | time="2023-03-02T01:13:14Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
watchtower_1  | time="2023-03-02T01:13:19Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
watchtower_1  | time="2023-03-02T01:13:24Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no

You can adjust the --interval 5 parameter according to the time interval you need to monitor, here set 5 seconds first, watchtower is set to monitor all containers of Host, so if some containers don’t want to update, you can set label as below.

1
2

labels:
  - "com.centurylinklabs.watchtower.enable=false"

In addition, after each upgrade, the old container or image file will still be in the host and will take up some space. You can use the --cleanup parameter to make watchtower delete the old image file after restarting the container with the new image file.

1
2
3
4
5
6

  watchtower:
    image: containrrr/watchtower
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command: --interval 5 --cleanup

Pull to the new Image and you will see the following message, it will send SIGTERM signal to the container for Gracefully Shutdown first.

1
2

watchtower_1  | time="2023-03-02T01:35:15Z" level=info msg="Found new ghcr.io/go-training/example53:latest image (040d01951ee2)"
watchtower_1  | time="2023-03-02T01:35:17Z" level=info msg="Stopping /root_example53_1 (57fc95adf8cd) with SIGTERM"

If you want to change Stop Signals, you can convert it by Label, please change Dockerfile.

1

LABEL com.centurylinklabs.watchtower.stop-signal="SIGHUP"

Or you can add the following parameter when starting the container.

1

docker run -d --label=com.centurylinklabs.watchtower.stop-signal=SIGHUP someimage

Summary

In the future, the team can focus on packaging images in the CI/CD flow and uploading them to the Docker Registry, and all services on the machine will be monitored by Watchtower, and the uploaded images will follow the semver principle. This reduces the workflow of writing shell scripts.