The cluster configuration is adjusted according to the number of cluster users to achieve the purpose of controlling the amount of resources used in a specific namespace and ultimately achieving fair usage and cost control of the cluster.

The functions to be implemented are as follows.

  • Limit the amount of compute resources used by Pods in the running state.
  • Limit the number of persistent volumes to control access to storage.
  • Limit the number of load balancers to control costs.
  • Prevent abuse of scarce resources such as network ports.
  • Provide default compute resource requests for more optimal system scheduling.

1. Create namespace

Create a namespace named quota-example with the following contents in the namespace.yaml file.

1
2
3
4

apiVersion: v1
kind: Namespace
metadata:
  name: quota-example

Create namespace.

1
2

[root@master1 ~]# kubectl create -f namespace.yaml 
namespace/quota-example created

View namespace.

1
2
3
4

[root@master1 ~]# kubectl get namespaces | grep -v 'kube'
NAME              STATUS   AGE
default           Active   17d
quota-example     Active   2m13s

2. Set resource quotas that limit the number of objects

You can control the number of persistent storage volumes, load balancers, and NodePort resources by setting a resource quota that limits the number of pairs.

Create a ResourceQuota named object-counts. object-counts.yaml file has the following contents.

1
2
3
4
5
6
7
8
9

apiVersion: v1
kind: ResourceQuota
metadata:
  name: object-counts
spec:
  hard:
    persistentvolumeclaims: "2"
    services.loadbalancers: "2"
    services.nodeports: "0"

Create resource.

1
2

[root@master1 ~]# kubectl create -f object-counts.yaml --namespace=quota-example
resourcequota/object-counts created

The quota system detects the creation of resource item quotas, counts and limits the consumption of resources in that namespace.

Check if this configuration is in effect.

1
2
3
4
5
6
7
8

[root@master1 ~]# kubectl describe quota object-counts --namespace=quota-example 
Name:                   object-counts
Namespace:              quota-example
Resource                Used  Hard
--------                ----  ----
persistentvolumeclaims  0     2
services.loadbalancers  0     2
services.nodeports      0     0

At this point, the quota system automatically blocks requests that push resource usage over the resource quota limit.

3. Set a resource quota that limits computing resources

Create a compute-resources-qualified resource quota to limit the total amount of compute resources used in this namespace.

Create a ResourceQuota named compute-resources. compute-resources.yaml file has the following contents.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-resources
spec:
  hard:
    pods: "4"
    requests.cpu: "1"
    requests.memory: 1Gi
    limits.cpu: "2"
    limits.memory: 2Gi

Create resource.

1
2

[root@master1 ~]# kubectl create -f compute-resources.yaml --namespace=quota-example 
resourcequota/compute-resources created

Check if the quota is in effect.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

[root@master1 ~]# kubectl describe quota compute-resources --namespace=quota-example 
Name:            compute-resources
Namespace:       quota-example
Resource         Used  Hard
--------         ----  ----
limits.cpu       0     2
limits.memory    0     2Gi
pods             0     4
requests.cpu     0     1
requests.memory  0     1Gi

The quota system automatically prevents more than 4 non-terminating Pods in the namespace at the same time, and because the resource quota limits the total number of CPU and memory Limits and Requests, it forces all containers in the namespace to explicitly define the CPU and memory Limits, Requests (default values can be used, Requests equals Limits by default).

4. Configuring Default Requests and Limits

If you try to create a Pod in a namespace that does not specify Requests and Limits when the namespace has been configured with a limited compute resource quota, the Pod creation may fail. Here is an example of this failure.

Create a Deployment of Nginx.

1
2

[root@master1 ~]# kubectl create deployment nginx --image=nginx --replicas=1 --namespace=quota-example
deployment.apps/nginx created

When you look at the created Pod, you will see that the Pod was not created successfully.

1
2
3
4
5

[root@master1 ~]# kubectl get pods --namespace=quota-example 
No resources found in quota-example namespace.
[root@master1 ~]# kubectl get deployments --namespace=quota-example 
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   0/1     0            0           2m38s

Further details on Deployment.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

[root@master1 ~]# kubectl describe deployments nginx --namespace=quota-example 
Name:                   nginx
Namespace:              quota-example
CreationTimestamp:      Tue, 02 Aug 2022 15:31:30 +0800
Labels:                 app=nginx
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=nginx
Replicas:               1 desired | 0 updated | 0 total | 0 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=nginx
  Containers:
   nginx:
    Image:        nginx
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type             Status  Reason
  ----             ------  ------
  Progressing      True    NewReplicaSetCreated
  Available        False   MinimumReplicasUnavailable
  ReplicaFailure   True    FailedCreate
OldReplicaSets:    <none>
NewReplicaSet:     nginx-6799fc88d8 (0/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  104s  deployment-controller  Scaled up replica set nginx-6799fc88d8 to 1

The Deployment tries to create a Pod, but fails. See the details of the ReplicaSet there.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

[root@master1 ~]# kubectl describe replicasets nginx-6799fc88d8  --namespace=quota-example 
Name:           nginx-6799fc88d8
Namespace:      quota-example
Selector:       app=nginx,pod-template-hash=6799fc88d8
Labels:         app=nginx
                pod-template-hash=6799fc88d8
Annotations:    deployment.kubernetes.io/desired-replicas: 1
                deployment.kubernetes.io/max-replicas: 2
                deployment.kubernetes.io/revision: 1
Controlled By:  Deployment/nginx
Replicas:       0 current / 1 desired
Pods Status:    0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  app=nginx
           pod-template-hash=6799fc88d8
  Containers:
   nginx:
    Image:        nginx
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type             Status  Reason
  ----             ------  ------
  ReplicaFailure   True    FailedCreate
Events:
  Type     Reason        Age                 From                   Message
  ----     ------        ----                ----                   -------
  Warning  FailedCreate  4m5s                replicaset-controller  Error creating: pods "nginx-6799fc88d8-vt4t2" is forbidden: failed quota: compute-resources: must specify limits.cpu,limits.memory,requests.cpu,requests.memory
  Warning  FailedCreate  4m5s                replicaset-controller  Error creating: pods "nginx-6799fc88d8-xrj66" is forbidden: failed quota: compute-resources: must specify limits.cpu,limits.memory,requests.cpu,requests.memory
  Warning  FailedCreate  4m5s                replicaset-controller  Error creating: pods "nginx-6799fc88d8-scqxt" is forbidden: failed quota: compute-resources: must specify limits.cpu,limits.memory,requests.cpu,requests.memory
  Warning  FailedCreate  4m5s                replicaset-controller  Error creating: pods "nginx-6799fc88d8-m672m" is forbidden: failed quota: compute-resources: must specify limits.cpu,limits.memory,requests.cpu,requests.memory
  Warning  FailedCreate  4m5s                replicaset-controller  Error creating: pods "nginx-6799fc88d8-62pcx" is forbidden: failed quota: compute-resources: must specify limits.cpu,limits.memory,requests.cpu,requests.memory
  Warning  FailedCreate  4m5s                replicaset-controller  Error creating: pods "nginx-6799fc88d8-j7p8m" is forbidden: failed quota: compute-resources: must specify limits.cpu,limits.memory,requests.cpu,requests.memory
  Warning  FailedCreate  4m5s                replicaset-controller  Error creating: pods "nginx-6799fc88d8-5tv47" is forbidden: failed quota: compute-resources: must specify limits.cpu,limits.memory,requests.cpu,requests.memory
  Warning  FailedCreate  4m4s                replicaset-controller  Error creating: pods "nginx-6799fc88d8-4dfch" is forbidden: failed quota: compute-resources: must specify limits.cpu,limits.memory,requests.cpu,requests.memory
  Warning  FailedCreate  4m4s                replicaset-controller  Error creating: pods "nginx-6799fc88d8-hffdc" is forbidden: failed quota: compute-resources: must specify limits.cpu,limits.memory,requests.cpu,requests.memory
  Warning  FailedCreate  81s (x7 over 4m3s)  replicaset-controller  (combined from similar events): Error creating: pods "nginx-6799fc88d8-p2rmx" is forbidden: failed quota: compute-resources: must specify limits.cpu,limits.memory,requests.cpu,requests.memory

Prompting a creation failure, the Master rejects the creation of a Pod by this ReplicaSet because the Requests, Limits for CPU and memory are not specified in this Pod.

To avoid this failure, you can use LimitRange to provide a default value for resource configuration for all Pods in this namespace.

Create a LimitRange named limits. limits.yaml file has the following contents.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

apiVersion: v1
kind: LimitRange
metadata:
  name: limits
spec:
  limits:
    - default:
        cpu: 200m
        memory: 512Mi
      defaultRequest:
        cpu: 100m
        memory: 256Mi
      type: Container

Create resource.

1
2

[root@master1 ~]# kubectl create -f limits.yaml --namespace=quota-example 
limitrange/limits created

View LimitsRange resource configuration.

1
2
3
4
5
6
7

[root@master1 ~]# kubectl describe limitranges --namespace=quota-example 
Name:       limits
Namespace:  quota-example
Type        Resource  Min  Max  Default Request  Default Limit  Max Limit/Request Ratio
----        --------  ---  ---  ---------------  -------------  -----------------------
Container   cpu       -    -    100m             200m           -
Container   memory    -    -    256Mi            512Mi          -

After a LimitRange is successfully created, if a user creates a Pod in that namespace without specifying a resource limit, the system automatically sets a default resource limit for that Pod.

For example, each newly created Pod without a resource limit is equivalent to using the following resource limit.

1
2

[root@master1 ~]# kubectl run nginx --image=nginx --requests=cpu=100m,memory=256Mi --limits=cpu=200m,memory=512Mi --namespace=quota-example
pod/nginx created

At this point, the namespace has been configured with default compute resources, and the ReplicaSet is ready to create Pods.

1
2
3

[root@master1 10.4]# kubectl get pods --namespace=quota-example
NAME                     READY   STATUS    RESTARTS   AGE
nginx-6799fc88d8-kvbpm   1/1     Running   0          39s

Next, check the resource quota usage.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

[root@master1 ~]# kubectl  describe quota --namespace=quota-example 
Name:            compute-resources
Namespace:       quota-example
Resource         Used   Hard
--------         ----   ----
limits.cpu       200m   2
limits.memory    512Mi  2Gi
pods             1      4
requests.cpu     100m   1
requests.memory  256Mi  1Gi


Name:                   object-counts
Namespace:              quota-example
Resource                Used  Hard
--------                ----  ----
persistentvolumeclaims  0     2
services.loadbalancers  0     2
services.nodeports      0     0

You can see that each Pod consumes a specified amount of resources when it is created.

5. Specifying the scope of resource quotas

Suppose that instead of configuring a default compute resource quota for a namespace, you want to limit the total number of Pods running in the namespace with a QoS of BestEffort (all container Requests and Limits in the Pod are undefined and have the lowest priority).

For example, by having some resources in the cluster running services with non-BestEffort QoS and idle resources running services with BestEffort QoS, you can avoid having all resources of the cluster exhausted by only a large number of BestEffort Pods. This can be achieved by creating two resource quotas.

Create a namespace named quota-scopes.

1
2

[root@master1 ~]# kubectl create namespace quota-scopes
namespace/quota-scopes created

Create a ResourceQuota named best-effort, specify the Scope as BestEffort, and the contents of the best-effort.yaml file as follows.

1
2
3
4
5
6
7
8
9

apiVersion: v1
kind: ResourceQuota
metadata:
  name: best-effort
spec:
  hard:
    pods: "10"
  scopes:
    - BestEffort

Create resource.

1
2

[root@master1 ~]# kubectl create -f best-effort.yaml --namespace=quota-scopes 
resourcequota/best-effort created

Create another ResourceQuota named not-best-effort, specifying the Scope as NotBestEffort, and the content of the not-best-effort.yaml file as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

apiVersion: v1
kind: ResourceQuota
metadata:
  name: not-best-effort
spec:
  hard:
    pods: "4"
    requests.cpu: "1"
    requests.memory: 1Gi
    limits.cpu: "2"
    limits.memory: 2Gi
  scopes:
    - NotBestEffort

Create resource.

1
2

[root@master1 ~]# kubectl create -f not-best-effort.yaml --namespace=quota-scopes 
resourcequota/not-best-effort created

View the successfully created ResourceQuota.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

[root@master1 ~]# kubectl describe quota --namespace=quota-scopes 
Name:       best-effort
Namespace:  quota-scopes
Scopes:     BestEffort
 * Matches all pods that do not have resource requirements set. These pods have a best effort quality of service.
Resource  Used  Hard
--------  ----  ----
pods      0     10


Name:       not-best-effort
Namespace:  quota-scopes
Scopes:     NotBestEffort
 * Matches all pods that have at least one resource requirement set. These pods have a burstable or guaranteed quality of service.
Resource         Used  Hard
--------         ----  ----
limits.cpu       0     2
limits.memory    0     2Gi
pods             0     4
requests.cpu     0     1
requests.memory  0     1Gi

Pods that are not configured with Requests will be restricted by a ResourceQuota named best-effort; Pods that are configured with Requests will be restricted by a ResourceQuota named not-best-effort.

Create a Deployment of bestEffort. The best-effort-nginx.yaml file is as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

apiVersion: apps/v1
kind: Deployment
metadata:
  name: best-effort-nginx
spec:
  replicas: 8
  selector:
    matchLabels:
      app: best-effort-nginx
  template:
    metadata:
      labels:
        app: best-effort-nginx
    spec:
      containers:
        - name: nginx
          image: nginx

Create the Deployment for NotbestEffort. not-best-effort-nginx.yaml file has the following contents.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

apiVersion: apps/v1
kind: Deployment
metadata:
  name: not-best-effort-nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: not-best-effort-nginx
  template:
    metadata:
      labels:
        app: not-best-effort-nginx
    spec:
      containers:
        - name: nginx
          image: nginx
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              cpu: 200m
              memory: 512Mi

Create two resources.

1
2
3

[root@master1 ~]# kubectl create -f best-effort-nginx.yaml -f not-best-effort-nginx.yaml  --namespace=quota-scopes 
deployment.apps/best-effort-nginx created
deployment.apps/not-best-effort-nginx created

The Deployment named best-effort-nginx has a QoS level of BestEffort because it does not have Requests and Limits configured, so its creation process is limited by the best-effort resource quota item, while the not-best-effort resource quota item does not limit it. The best-effort resource quota item does not limit Requests and Limits, so best-effort-nginx Deployment can successfully create 8 Pods.

Deployment named not-best-effort-nginx has a Burstable QoS level because it is configured with Requests and Limits, which are not equal, so its creation process is limited by the not-best-effort resource quota item, while the best-effort resource The not-best-effort resource quota restricts the total limit of Requests and Limits for the Pod. not-best-effort-nginx Deployment does not exceed this limit, so it can successfully create two Pods.

Looking at the Pods that have been created, you can see that all 10 Pods were created successfully.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

[root@master1 ~]# kubectl get pods --namespace=quota-scopes 
NAME                                    READY   STATUS    RESTARTS   AGE
best-effort-nginx-78bb6d5d9d-6cxg7      1/1     Running   0          45s
best-effort-nginx-78bb6d5d9d-ckrdx      1/1     Running   0          44s
best-effort-nginx-78bb6d5d9d-jw6cq      1/1     Running   0          44s
best-effort-nginx-78bb6d5d9d-llm42      1/1     Running   0          44s
best-effort-nginx-78bb6d5d9d-pzkkk      1/1     Running   0          45s
best-effort-nginx-78bb6d5d9d-rhjps      1/1     Running   0          45s
best-effort-nginx-78bb6d5d9d-tl6g5      1/1     Running   0          44s
best-effort-nginx-78bb6d5d9d-zghvp      1/1     Running   0          40s
not-best-effort-nginx-54cfd5bd8-v68dr   1/1     Running   0          45s
not-best-effort-nginx-54cfd5bd8-v7mt8   1/1     Running   0          45s

When you look at the usage of the two resource quota entries, you see that the best-effort resource quota entry has counted the resource usage of the 8 Pods created in best-effort-nginx Deployment. The not-best-effort resource quota item has also counted the resource usage of the two Pods created in not-best-effort-nginx Deployment.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

[root@master1 ~]# kubectl describe quota --namespace=quota-scopes 
Name:       best-effort
Namespace:  quota-scopes
Scopes:     BestEffort
 * Matches all pods that do not have resource requirements set. These pods have a best effort quality of service.
Resource  Used  Hard
--------  ----  ----
pods      8     10


Name:       not-best-effort
Namespace:  quota-scopes
Scopes:     NotBestEffort
 * Matches all pods that have at least one resource requirement set. These pods have a burstable or guaranteed quality of service.
Resource         Used   Hard
--------         ----   ----
limits.cpu       400m   2
limits.memory    1Gi    2Gi
pods             2      4
requests.cpu     200m   1
requests.memory  512Mi  1Gi

This example reveals that the resource quota scopes (Scopes) provide a mechanism to partition the set of resources, making it easier for cluster administrators to monitor and restrict the use of various types of resources by different types of objects, while providing better flexibility and convenience for resource allocation and restriction.

6. Resource Management Summary

Resource management in Kubernetes is based on the resource allocations (Requests and Limits) of containers and Pods. A container’s resource allocation specifies the resources requested by the container and the maximum amount of resources the container can use, and a Pod’s resource allocation is the sum of the resource allocations of all the containers in the Pod.

The resource quota mechanism allows us to limit the total number of resources used by all Pods in a namespace, as well as the number of objects of a specified type in that namespace. Using scopes allows resource quotas to be restricted only to objects that fit a specific scope, so the scoping mechanism allows for a richer, more flexible resource quota policy.

If more restrictions need to be placed on the resource allocation of a user’s Pod or container, this can be achieved using a LimitRange, which effectively limits the maximum and minimum range of resource allocations for Pods and containers, as well as limiting the maximum ratio of Limits to Requests for Pods and containers. LimitRange can also provide default resource allocation for containers in Pods.

Kubernetes implements resource quality of service (QoS) based on the resource allocation of Pods. Pods with different QoS levels have different priorities in the system: Pods with higher priorities have higher reliability and can be used to run services with higher reliability requirements; Pods with lower priorities can oversell cluster resources and effectively improve cluster resource utilization.