Istio Sidecar's interception mechanism for traffic

The basic process of flowing traffic through a host

1. Inbound traffic passes through the NIC and enters the host’s network stack.
2. the protocol stack checks the messages against pre-customized network rules (iptables/netfilter).
3. after the stack rules check, compliant Inbound traffic enters from kernel space to user space and enters the process that specifies the listening port.
4. the user process in the user state receives the network traffic message for processing and returns the processed result to the network protocol stack in kernel space via user space.
5. the network protocol stack examines the messages and sends the resultant messages out through the NIC according to the specified network policy.

Sidecar Traffic Interception Basic Process

The injection mechanism of Sidecar has been introduced in the previous article, and the following two containers are injected into the Pod.

• istio-init: InitContainer, which is used for the initial configuration of iptables in the Pod during the initialization of the Pod.
• istio-proxy: responsible for communication with pilot components and traffic control; this container will start two key processes pilot-agent and envoy when it runs.The pilot-agent process will communicate with istio’s pilot components regularly, and the envoy process will receive inbound and outbound network traffic.

Note: both istio-proxy and kube-proxy in Kubernetes handle network traffic via iptables/netfilter. Only istio-proxy is located in pod cyberspace and handles network traffic within the pod, while kube-proxy is located in host cyberspace and handles network traffic within the host (since kube-proxy is a daemonset, it is located on every node node of the k8s cluster).

Sidecar traffic interception is actually based on iptables rules, which are first set by the init container when the Pod is started.

iptables rules to intercept the application container Inbound/Outbound traffic, currently can only intercept TCP traffic, can not intercept UDP, because Envoy currently does not support the forwarding of UDP.

The following to analyze the Inbound traffic a series of direction.

1. inbound traffic is first intercepted by iptables rules as it enters the Pod’s network stack.
2. the iptables rules forward the packets to the Envoy process in the istio-proxy container.
3. Envoy then forwards the traffic to the application process in the app container according to the configuration of its own listener.

NOTE: Envoy will also flow through the kernel stack when forwarding traffic to applications processed by iptables rules, where the rules set by the init container do not intercept, so the iptables process is omitted in the middle.

Then to analyze the Outbound traffic a series of direction.

1. outbound traffic is sent from the application and is first intercepted by the iptables rules.
2. the iptables rules forward the outbound packets to the Envoy process of the istio-proxy container.
3. Envoy then decides whether to forward the traffic outside the container according to its own configuration.

Traffic interception implementation details

Istio traffic proxy forwarding mode

In Istio, the implementation of traffic interception relies on the setup of initContainer iptables rules. There are currently two traffic interception modes: REDIRECT mode and TPROXY mode.

REDIRECT mode

REDIRECT mode allows all traffic to be redirected to a specific port through iptables.

For example, the following REDIRECT rule.

1

iptables -t nat -A PREROUTING -p tcp -j REDIRECT --to-port 15001

This means that all traffic is redirected to port 15001. In general, when we use iptables to do DNAT, we need to specify the target IP and port so that iptables will know what to change the destination of the packet to, while REDIRECT mode, only the port needs to be specified, and iptables will automatically help us determine the IP address that needs to be set.

REDIRECT mode will perform source address translation but it is still the default setting because with the telemetry data provided by Istio it is still possible to perform call chain analysis. In the Kubernetes platform Pods and their IP addresses are not persistent and will be dynamically destroyed and migrated with the resource status of the cluster, so the traditional software system of recording clients like source address is not suitable for the cloud-native application platform Kubernetes.

TPROXY mode

In addition to using REDIRECT mode, Istio also provides TPROXY mode, for TPROXY mode, you need to use iptables and routing, a little more complicated, used to do transparent proxy, the operation is mangle table.

Also requires the original client socket to set the IP_TRANSPARENT option, Linux provides an IP_TRANSPARENT option , this option allows the program to bind an address that does not belong to the local machine, as a client, it can use an IP address that does not belong to the local address as the source IP to initiate a connection, as a server, it can listen on an IP address that does not belong to the local machine, which is necessary for a transparent proxy.

Since TPROXY mode does not change the packets, the original IP port information is obtained directly through getsockname.

Combine with iptables to analyze Inbound/Outbound traffic direction

Inbound traffic

• Inbound matches the first rule of the PREROUTING chain of the iptables nat table first, so Inbound traffic is routed to the ISTIO_INBOUND chain.
• In the ISTIO_INBOUND chain, the traffic is routed to the ISTIO_IN_REDIRECT chain, matching the Xth rule of that chain, based on the access port.
• traffic routed to the ISTIO_IN_REDIRECT chain eventually hits the Envoy process listening to port xxxx from the kernel state to the user state.
• After the Envoy process processes the traffic, it will pass the traffic from the user state process back to the network stack in the kernel state, and according to the predefined stack rules, the traffic will flow through the OUTPUT chain, and the OUTPUT chain will in turn route the traffic to the ISTIO_OUTPUT chain again according to the rules.
• Because Envoy will eventually re-route the traffic to the application process on the destination port after processing, the rule X in the ISTIO_OUTPUT chain will be matched (because Envoy is in the same network namespace as the application container process, so the loopback address lo is matched), and the traffic will then return from the kernel state to the user state and enter the application process listening on port X.
• The application process finishes processing the traffic and returns the result to the envoy process (user state) via a socket connection.
• Envoy then returns the traffic to the user through the POSTROUTING chain, NIC, and returns the response traffic to the user.

Outbound Traffic

• Outbound traffic first enters the ISTIO_OUTPUT chain via the OUTPUT chain of the iptables nat table.
• Matching to the ISTIO_REDIRECT chain, based on the destination address.
• The ISTIO_REDIRECT chain routes the traffic to the Envoy process.
• Envoy re-routes the processed Outbound traffic through the user state to the kernel state network stack, where the traffic will first go through the OUTPUT chain to the ISTIO_OUTPUT chain.
• The traffic will eventually go to the NIC via the POSTROUTING chain and then be sent out.