MPTCP: an overlooked new feature in Go 1.21

Go 1.21 will be released in two or three months, and many people have already summarised the new features in Go 1.21 to build up the momentum for the new Go release, but I haven’t seen anyone specifically cover one of the new features added to the net library in Go 1.21, so I’m dedicating a new article to it.

There is a dedicated issue (#56539) to follow up and discuss this new feature of MPTCP. It is a single-path extension to TCP, defined by the RFC8684 specification.

Multipath TCP (MPTCP) is a protocol at the transport layer designed to augment the traditional single-path TCP protocol by enabling the simultaneous transmission of data over multiple network paths. MPTCP allows multiple paths to be utilised simultaneously for data transmission, providing higher bandwidth, better load balancing and greater reliability.

The traditional TCP protocol is designed for single paths, by transferring data over a single connection between end-to-ends. MPTCP, on the other hand, makes it possible for a single TCP connection to operate on multiple network paths simultaneously by introducing additional functionality.

MPTCP works as follows:

Connection Establishment: MPTCP’s connection establishment process is similar to traditional TCP, except that during the initial handshake, both parties exchange capability options to determine if MPTCP is supported.
Subflow Establishment: Once an MPTCP connection is established, it can initiate multiple subflows (subflow), each of which transmits data over a different network path. These subflows can be identified by different IP addresses and port numbers.
Path Management: MPTCP uses a path management mechanism to select and manage multiple network paths. It allows path selection based on path quality, latency, bandwidth and other metrics, and dynamically adjusts the use of paths according to network conditions.
Data Transfer: MPTCP splits data into appropriately sized blocks and sends them on different sub-streams. The receiving end reassembles the data based on the sequence number of the block and the sub-stream to which the block belongs.

The benefits of MPTCP include:

Bandwidth Enhancement: MPTCP can utilize the bandwidth of multiple paths simultaneously, thus providing higher overall bandwidth.
Load balancing: MPTCP can dynamically adjust data transmission based on path quality and available bandwidth, enabling load balancing and improving network resource utilisation.
Fault Tolerance: Because data can be transmitted over multiple paths, MPTCP can provide better fault tolerance. Even if a path fails, data can still be transmitted through other available paths.
Mobility Support: MPTCP can maintain connectivity when mobile devices switch networks without having to re-establish connectivity, providing a smoother mobile experience.

MPTCP has become a standardised protocol and is used in a wide range of multi-path transport scenarios, such as intra-data centre communications, wireless networks and mobile networks.

For example, the official apple documentation states that

iOS supports Multipath TCP (MPTCP) and allows the iPhone or iPad to establish a backup TCP connection to the target host over a cellular data connection. iPhone and iPad use MPTCP with a valid cellular data connection to establish two connections:

a primary TCP connection over Wi-Fi

Backup connection via cellular data

If Wi-Fi is unavailable or unresponsive, iOS will use the cellular data connection.

https://support.apple.com/zh-cn/HT201373

MPTCP is also specifically covered in the official Red Hat help documentation.

MPTCP is widely supported in the Linux kernel and has become part of the Linux kernel. chatGPT says that MPTCP has been included in the mainline kernel since Linux kernel version 3.6 and that users can configure and use MPTCP features, but the MPTCP community website says that MPTCP v1 is only supported since 5.6.

Linux distributions such as Ubuntu, Fedora and Debian usually include MPTCP support by default. As of Linux 5.19, MPTCP includes the following features:

Support for setting the IPPROTO_MPTCP protocol in the socket system call
Fallback from MPTCP to TCP if the other party or intermediate device does not support MPTCP
Use in-kernel or userspace path manager for path management
Sockets options that also use TCP sockets
Debugging features, including MIB counters, diagnostic support (using the ss command) and trace points

After much effort by the Go community and the MPTCP community, a convenient way to support MPTCP has finally been found and implemented in Go 1.21, summarised in the following four methods.

For the TCP client, you can set up and check for MPTCP support by using the following methods.

1
2

func (*Dialer) SetMultipathTCP(enabled bool)
func (*Dialer) MultipathTCP() bool

For the TCP server you can set up and check if MPTCP is supported in the following way.

1
2

func (*ListenConfig) SetMultipathTCP(enabled bool)
func (*ListenConfig) MultipathTCP() bool

So for a system, both the client and server side need to be set up for MPTCP to take effect.

I will use a simple example to demonstrate how MPTCP can be used.

The server-side code is as follows, enabling mptcp for the Listener, and the connection established with the client may support mptcp, or may degrade to normal tcp.

package main
import (
    "context"
    "errors"
    "flag"
    "fmt"
    "io"
    "net"
)
var (
    addr = flag.String("addr", ":8080", "service address")
)
func main() {
    flag.Parse()
    lc := &net.ListenConfig{}
    if lc.MultipathTCP() { // By default, mptcp is disabled
        panic("MultipathTCP should be off by default")
    }
    lc.SetMultipathTCP(true) // Proactively enable mptcp
    ln, err := lc.Listen(context.Background(), "tcp", *addr) // Normal tcp listening
    if err != nil {
        panic(err)
    }
    for {
        conn, err := ln.Accept()
        if err != nil {
            panic(err)
        }
        go func() {
            defer conn.Close()
            isMultipathTCP, err := conn.(*net.TCPConn).MultipathTCP() // Check if the connection supports mptcp
            fmt.Printf("accepted connection from %s with mptcp: %t, err: %v\n", conn.RemoteAddr(), isMultipathTCP, err)
            for {
                buf := make([]byte, 1024)
                n, err := conn.Read(buf)
                if err != nil {
                    if errors.Is(err, io.EOF) {
                        return
                    }
                    panic(err)
                }
                if _, err := conn.Write(buf[:n]); err != nil {
                    panic(err)
                }
            }
        }()
    }
}

The client code is as follows, we enable mptcp for the dialer, and check if the established connection really supports mptcp, because if the client or server does not support mptcp, it will degrade to normal tcp.

package main
import (
    "flag"
    "fmt"
    "net"
    "time"
)
var (
    addr = flag.String("addr", "127.0.0.1:8080", "service address")
)
func main() {
    flag.Parse()
    d := &net.Dialer{}
    if d.MultipathTCP() { // Not enabled by default.
        panic("MultipathTCP should be off by default")
    }
    d.SetMultipathTCP(true) // Enabling.
    if !d.MultipathTCP() { 
        panic("MultipathTCP is not on after having been forced to on")
    }
    c, err := d.Dial("tcp", *addr)
    if err != nil {
        panic(err)
    }
    defer c.Close()
    tcp, ok := c.(*net.TCPConn)
    if !ok {
        panic("struct is not a TCPConn")
    }
    mptcp, err := tcp.MultipathTCP() //Does the connection established really support mptcp
    if err != nil {
        panic(err)
    }
    fmt.Printf("outgoing connection from %s with mptcp: %t\n", *addr, mptcp)
    if !mptcp { // mptcp is not supported, then panic
        panic("outgoing connection is not with MPTCP")
    }
    for {
        snt := []byte("MPTCP TEST")
        if _, err := c.Write(snt); err != nil {
            panic(err)
        }
        b := make([]byte, len(snt))
        if _, err := c.Read(b); err != nil {
            panic(err)
        }
        fmt.Println(string(b))
        time.Sleep(time.Second)
    }
}

Ref: https://colobu.com/2023/07/03/mptcp-a-go-1-21-new-feature/