“Graceful termination” means that when a service needs to be taken offline or restarted, there are measures and methods to let other services perceive the current service being taken offline as soon as possible on the one hand, and minimize the impact on the current processing requests on the other. Elegant termination can improve the high availability of services, reduce the service jitter caused by going offline, and improve service stability and user experience.

Taking services offline is not just a job at the Ops level. It requires the cooperation of the whole RPC implementation, service architecture and operation and maintenance system to achieve graceful service offline perfectly. This article will analyze how to achieve graceful termination of microservices based on the whole process of service offline

  • Active offline of service registry
  • Analysis of how gRPC can achieve graceful termination based on the source code of gRPC-Go
  • Exploring the graceful termination of k8s

Active offline of the service registry

If the service uses a service registry (e.g. Consul, etc.), then the first step is to first take the service offline from the registry. This will ensure that new requests are not sent to this node as soon as possible.

Although most service registries have a heartbeat and timeout auto-cleanup mechanism for the node, the heartbeat has a fixed interval and the registry needs to wait until the preset heartbeat timeout to find out that the node is offline. Therefore, active offline can greatly shorten the process of exception discovery.

If the service is managed and scheduled based on k8s, it is very convenient to do this.

First, k8s itself comes with a reliable service discovery, and k8s will naturally be the first to sense when pods are going up or down on k8s.

If you are using an external name service, you can use the preStop feature of k8s. k8s natively supports container lifecycle callbacks, we can define the pod’s preStop hook to enable cleanup operations before the service goes offline. As follows.

Example.

1
2
3
4
5
6
7

containers:
- name: my-app-container
  image: my-app-image
  lifecycle:
    preStop:
      exec:
        command: ["/bin/sh","-c","/app/pre_stop.sh"]

Before a pod goes offline, it first executes the /app/pre_stop.sh command, in which we can do a lot of pre-cleanup policies.

Graceful termination of RPC

Removing a service node from the name service can block new traffic from entering the node, which is the first step towards graceful termination. However, for the established client connections on that node, a hasty disconnect would cause an abrupt suspension of the ongoing business logic. Therefore, we need to implement RPC-level graceful termination of connections and request processing to ensure that the business logic is minimally affected.

Take gRPC-Go as an example. gRPC implements two stop interfaces GracefulStop and Stop, representing graceful and non-graceful termination of the service, respectively. Let’s look at how gRPC terminates gracefully.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

func (s *Server) GracefulStop() {
    s.quit.Fire()
    defer s.done.Fire()

    ...
    s.mu.Lock()

    // 首先关闭监听 socket，保证不会有新的连接到来
    for lis := range s.lis {
        lis.Close()
    }

    s.lis = nil
    if !s.drain {
        for st := range s.conns {
            st.Drain()
        }
        s.drain = true
    }

    // Wait for serving threads to be ready to exit.  Only then can we be sure no
    // new conns will be created.
    s.mu.Unlock()
    s.serveWG.Wait()
    s.mu.Lock()

    for len(s.conns) != 0 {
        s.cv.Wait()
    }
    ...
    s.mu.Unlock()
}
  • Step 1: Call s.quit.Fire() . When this statement is executed, gRPC discards all new Accept arrivals directly.
  • Step 2: Call lis.Close() one by one. Close the listening socket so that no more new connections will arrive.
  • Step 3: Call st.Drain() one by one for the established connections. Since gRPC is based on the HTTP2 implementation, the goAway frame of HTTP2 will be applied here. The goAway frame is equivalent to the server-side active signal to the client to close the connection, and upon receiving this signal, the client will close all HTTP2 streams on that connection. This way the client side can actively sense that the connection is closed and will not continue to send new requests over.
  • Step 4: Call s.serveWG.Wait(). Ensure that the Serve function of gRPC has exited properly.
  • Step 5: Call s.cv.Wait() . This logic is used to wait for the normal end of all business processing logic for established connections. This way no abnormalities in the business logic will be caused by the sudden shutdown of the service.

The above is the graceful termination process of gRPC. In short, gRPC needs to ensure the proper closure of each layer of logic from the outside in.

However, there is a problem here that may be easy to overlook. The last step calls s.cv.Wait(), which is used to wait for the business processing logic to finish properly. But there may be an exception here that if the business logic has a deadlock or dead loop due to a code bug, then the business logic will never finish and s.cv.Wait() will always be stuck. As a result, GracefulStop will also never end.

To address this issue, you need to work with an external deployment system to force a timeout on the service. Next, let’s look at how k8s does this.

Graceful termination of k8s

Before k8s takes a pod offline, the cluster does not forcibly kill the pod, but instead needs to perform a series of steps before the pod is decently taken offline.

  1. Check the pod lifecycle and execute the preStop hook first if it is configured with preStop. We can do things like pre-cleaning and service registry offline.
  2. send SIGTERM signal to the pod. SIGTERM is actually the equivalent of the linux command kill -15. This requires the RPC to listen for the SIGTERM signal itself and perform a graceful termination once it receives the signal; 3.
  3. wait for a while, if the pod still does not stop by itself, then send SIGKILL signal to the pod, which is equivalent to the linux command kill -9, and the pod will be forcibly terminated. And the waiting time depends on the pod’s configured graceful termination time terminationGracePeriodSeconds parameter, the default is 30 seconds.

Summary of the process of graceful termination

The above content talks about how to achieve graceful termination of services in various aspects respectively, and summarizes the whole process of graceful termination.

  1. First, the service node is actively taken offline from the service registry to ensure the service registry. If the service is scheduled and managed based on k8s, the preStop callback can be used to take the service registry offline. 2.
  2. RPC needs to implement a complete set of graceful termination logic. Ensure that existing business logic is not damaged as much as possible.
  3. k8s waits for the pod graceful termination period to expire and force the pod to stop.

Based on the above set of processes, you can achieve graceful termination of the service, which is basically enough for stateless services. However, for stateful services, the challenge of graceful termination is a bit more difficult.

This article focuses on graceful termination of services, so since there is graceful termination, it also corresponds to graceful startup of services. The graceful termination of the service is from the outside to the inside, first shutting down the outermost traffic entry, and then gradually stopping the logic inward. The graceful start of the service is from inside to outside to ensure that the logic of each layer is opened normally, in order to complete the final online.