QoS Levels

QoS is closely related to resource limits and, as shown under, is the configuration of the resource limits section of a Pod.

1
2
3
4
5
6
7

resources:
  limits:
    cpu: 200m
    memory: 1G
  requests:
    cpu: 500m
    memory: 1G

Kubernetes classifies Pod QoS into three levels of QoS based on CPU and memory configurations.

  • Guaranteed: guaranteed, set only limits or if requests is the same as limits
  • Burstable: burstable, only if requests or requests is lower than limits
  • Best-effort: default, or this level if not set

Why you should care about Pod QoS levels

In Kubernetes, resources are divided into two categories: compressible resources “CPU” and incompressible resources “memory”. When compressible resources are exhausted, they will not be terminated and evicted, while when incompressible resources are exhausted, i.e. when a Pod is out of memory, it will be killed by OOMKiller, i.e. evicted and other operations, and knowing the QoS level of a Pod can effectively prevent critical Pods from being evicted.

Pod QoS classification

Sourceļ¼šhttps://doc.kaas.thalesdigital.io/docs/BestPractices/QOS

As you can see from the diagram above, Pods at the BestEffort level are able to use all the resources on the node, which will cause resource problems for other Pods. So these Pods have the LowestEffort priority and will be killed first if the system runs out of memory.

How Pods are evicted

When a node is running low on compute resources, kubelet will initiate an eviction, this operation is to avoid system OOM events, and the QoS level determines the priority of the eviction, Pods of type BestEffort with no limit on resources are evicted first, next Guaranteed and Burstable with lower resource usage than Requests will not be evicted by the resource usage of other Pods, and secondly for such Pods, if the Pod uses resources more than the configuration (Requests), eviction will be based on the priority of these two levels of Pods. BestEffort and Burstable will be evicted in order of priority, then resource usage.

For disk pressure, the eviction order is based on BestEffort ==> Burstable ==> Guaranteed.

How to check the QoS level of a Pod

The Status field in the Pod resource list represents the Pod QoS level.

1

kubectl get pod <pod_name> -o jsonpath='{.status.qosClass}'

How to configure the default level of QoS

If you don’t want to configure resource limits for each Pod, Kubernetes provides an API LimitRange that allows you to specify a default QoS, provide a default resource limit for the Pod, and then the access controller will add the default resource limit k8s.io/kubernetes/plugin/pkg/admission/limitranger. As the official example is given.

Notes: The access controller is not the same concept as the Pod controller, the access controller is the handler chain for kube-apiserver requests

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

apiVersion: v1
kind: LimitRange
metadata:
  name: cpu-resource-constraint
spec:
  limits:
  - default: # this section defines default limits
      cpu: 500m
    defaultRequest: # this section defines default requests
      cpu: 500m
    max: # max and min define the limit range
      cpu: "1"
    min:
      cpu: 100m

The access controller checks this, and LimitRange is also a criterion that restricts all Pods to resource limit criteria (Request and limits) that must be less than or equal to the format of the LimitRange configuration, for example the following configuration will not be accessed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

apiVersion: v1
kind: Pod
metadata:
  name: example-conflict-with-limitrange-cpu
spec:
  containers:
  - name: demo
    image: registry.k8s.io/pause:2.0
    resources:
      requests:
        cpu: 700m

As the Pod is not configured with Limits and does not conform to the specification, the Pod will not be dispatched with the following error.

1

Pod "example-conflict-with-limitrange-cpu" is invalid: spec.containers[0].resources.requests: Invalid value: "700m": must be less than or equal to cpu limit

If both request and limit are set, the new Pod will be dispatched successfully even if it is larger than the LimitRange configuration.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

apiVersion: v1
kind: Pod
metadata:
  name: example-no-conflict-with-limitrange-cpu
spec:
  containers:
  - name: demo
    image: registry.k8s.io/pause:2.0
    resources:
      requests:
        cpu: 700m
      limits:
        cpu: 700m