I’ve been doing a five-day Kubernetes education training at an enterprise for the past few weeks, and our Lab environment is all based on lightweight MicroK8s. But the course didn’t start well because there were about 20 people in the class, and they were all using the company’s network, and they all shared the same IP connection to the extranet, so they ran into the annoying Rate limit on Docker Hub problem. So some people can’t set up the cluster successfully, because the image can’t be downloaded. In this article, I’ll explain how to solve the problem of not being able to download images from Docker Hub in MicroK8s.

Problem detection

When the problem occurs, you should see ErrImagePull or ImagePullBackOff with kubectl get po, which means that the Image is pulling with an exception.

At this point you can use kubectl describe po/$PODNAME to view the Events event data to determine the cause of the problem, the error message is as follows.

Warning Failed 19m (x2 over 19m) kubelet Failed to pull image “docker.io/calico/cni:v3.19.1”: rpc error: code = Unknown desc = failed to pull and unpack image “docker.io/calico/cni:v3.19.1”: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/calico/cni/manifests/sha256:f301171be0add870152483fcce71b28cafb8e910f61ff003032e9b1053b062c4: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

This kubectl describe is the command we will definitely use when we do Troubleshooing in Kubernetes!

Solutions

In fact, there are many different solutions, and you can see them in the DockerHub download rate limits file on the MicroK8s website.

  1. Set up direct login to DockerHub by pull image authentication
  2. Use an alternative Registry source to fix the problem by editing the content of the existing Deployment
  3. Use your own private image registry and use a proxy server to fetch the image content from DockerHub

In this post, I’m going to share two standard ways to log into DockerHub.

Solution 1: Adjust containerd’s Docker Hub Registry settings

Since MicroK8s mainly uses containerd as the container engine (Container Runtime), the most effective way is to directly adjust the containerd setting of “each node” so that any image to be downloaded from Docker Hub Any image to be downloaded from Docker Hub can enter Docker Hub’s authentication information (account password) to lift Docker Hub’s Rate limit limit.

The following are the steps to set up

  1. First get the account and password of Docker Hub, it is recommended not to use your password in Docker Hub directly, but go to Security of Docker Hub to set up a Access Token as the password. security), create an Access Token as a password, and select Read-only in the Access permissions to ensure your account is secure!

    Docker Hub

  2. Edit the /var/snap/microk8s/current/args/containerd-template.toml file in the cluster node

    1

    multipass exec 'microk8s-vm' -- vi /var/snap/microk8s/current/args/containerd-template.toml

    When you first enter the vim editor, you may not be able to see some of the text clearly due to the Vim default theme setting, so it is recommended that you enter the following command to make the text appear more clearly.

    1

    :set background=dark

    Next, at the bottom of the file, add the following: (typography is not important)

    1
2
3

    [plugins."io.containerd.grpc.v1.cri".registry.configs."registry-1.docker.io".auth]
username = "DOCKERHUB_Username"
password = "DOCKERHUB_AccessToken"

    Please be sure to do the same settings in each node, but whether the cluster will retain the settings after the upgrade, I have not tested, I will wait for the next upgrade to test to see!

    The following is another technique for adding content to /var/snap/microk8s/current/args/containerd-template.toml without opening vim.

    Start by entering the VM’s shell environment.

    1

    multipass shell microk8s-vm

    Add the content one at a time.

    1
2
3
4
5

    cat <<EOF >> /var/snap/microk8s/current/args/containerd-template.toml
[plugins."io.containerd.grpc.v1.cri".registry.configs."registry-1.docker.io".auth]
    username = "DOCKERHUB_Username"
    password = "DOCKERHUB_AccessToken"
EOF

  3. Reboot MicroK8s and you’re good to go!

    1
2

    microk8s stop
microk8s start

  4. Test Deployment Application

    1
2
3

    kubectl create deployment microbot --image=dontrebootme/microbot:v1
kubectl expose deployment microbot --type=NodePort --port=80 --name=microbot-service
kubectl describe pod

Solution 2: Use a custom Private Registry as an image source

Many of our customers and ourselves set up our own private registries as image sources, such as ACR (Azure Container Registry) or Harbor, etc. When you want to deploy self-made images and upload them to Private Registry, you must be authenticated, otherwise you will not be able to download any images back.

Here are the steps to set up

  1. First, get the account and password of Private Registry. Here I also use Docker Hub as an example, instead of using your Docker Hub password, we suggest you go to the Security page of Docker Hub and create an Access Token as your password. security), create an Access Token as a password, and select Read-only in Access permissions to ensure your account is secure!

  2. Create a Secret object called regcred

    Bash

    1
2
3
4
5

    kubectl create secret docker-registry regcred \
--docker-server="https://index.docker.io/v1/" \
--docker-username="<Username>" \
--docker-password="<AccessToken>" \
--docker-email="<Email>"

    PowerShell

    1
2
3
4
5

    kubectl create secret docker-registry regcred `
--docker-server="https://index.docker.io/v1/" `
--docker-username="<Username>" `
--docker-password="<AccessToken>" `
--docker-email="<Email>"

    After the creation is complete, you can see the actual settings with the following command.

    1

    kubectl get secret regcred --output="jsonpath={.data.\.dockerconfigjson}" | base64 --decode | jq -C

    Obtain the initial account and password.

    1
2

    kubectl get secret regcred --output="jsonpath={.data.\.dockerconfigjson}" | base64 -d | jq '.auths."https://index.docker.io/v1/".username' -Cr
kubectl get secret regcred --output="jsonpath={.data.\.dockerconfigjson}" | base64 -d | jq '.auths."https://index.docker.io/v1/".password' -Cr

    Specify the imagePullSecrets field as regcred in the Container Spec of the Pod Template.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10

    apiVersion: v1
kind: Pod
metadata:
name: private-reg
spec:
containers:
    - name: private-reg-container
    image: <your-private-image>
imagePullSecrets:
    - name: regcred

    Once set up, your Pod will properly download images from the Private Registry when it starts! 👍

Reference