We know that, in general, TCP/UDP ports can only be bound to one socket. When we try to listen to a port that is already listened to by another process, the bind call fails and errno is set to 98 EADDRINUSE. This is also known as port occupation.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

int fd1 = socket(AF_INET, SOCK_DGRAM, 0);
int fd2 = socket(AF_INET, SOCK_DGRAM, 0);

struct sockaddr_in addr = {0};
addr.sin_family = AF_INET;
addr.sin_port = htons(1234);
addr.sin_addr.s_addr = inet_addr("127.0.0.1");

bind(fd1, (struct sockaddr*)&addr, sizeof(addr)); // 0
bind(fd2, (struct sockaddr*)&addr, sizeof(addr)); // -1, errno = 98

But can a port be listened to by only one process? Obviously not. For example, we can bind a socket and then fork it, so that both child processes are listening on the same port. This is how Nginx does it, with all its worker processes listening on the same port. We can also do the same thing by using a UNIX domain socket to pass a file and “sending” an fd to another process.

In fact, according to TCP/IP standards, ports themselves are allowed to be multiplexed. The essence of port binding is that when the system receives a TCP message or UDP datagram, it can find the corresponding process based on the port field in its header and pass the data to the corresponding process. In addition, for broadcast and multicast, port multiplexing is necessary because they are inherently multi-delivery.

setsockopt

In Linux, we can call setsockopt to set the SO_REUSEADDR and SO_REUSEPORT options to 1 to enable address and port multiplexing.

1
2
3

int val = 1;
setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &val, sizeof(val));
setsockopt(fd, SOL_SOCKET, SO_REUSEPORT, &val, sizeof(val));

It’s a little trickier for Go, because Go does all the socket() , bind() , listen() steps at once when calling net.Listen. So we have to use net.ListenConfig to set up a callback function to control the intermediate process. After getting the raw file descriptor in the callback function, we can call syscall.SetsockoptInt to set the socket options, which is similar to the original setsockopt system call.

1
2
3
4
5
6
7
8
9

cfg := net.ListenConfig{
    Control: func(network, address string, c syscall.RawConn) error {
        return c.Control(func(fd uintptr) {
            syscall.SetsockoptInt(int(fd), syscall.SOL_SOCKET, unix.SO_REUSEADDR, 1)
            syscall.SetsockoptInt(int(fd), syscall.SOL_SOCKET, unix.SO_REUSEPORT, 1)
        })
    },
}
tcp, err := cfg.Listen(context.Background(), "tcp", "127.0.0.1:1234")

Role

What does port multiplexing do? According to the TCP/IP standard, for unicast datagrams, if port multiplexing exists, it can only be delivered to one of the processes (as opposed to multicast and broadcast, where it is delivered to all processes). We can make the server more efficient by having multiple processes listening on the same port, allowing them to receive and process data in a preemptive manner. This is how Nginx does it.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

func main() {
    cfg := net.ListenConfig{
        Control: func(network, address string, c syscall.RawConn) error {
            return c.Control(func(fd uintptr) {
                syscall.SetsockoptInt(int(fd), syscall.SOL_SOCKET, unix.SO_REUSEADDR, 1)
                syscall.SetsockoptInt(int(fd), syscall.SOL_SOCKET, unix.SO_REUSEPORT, 1)
            })
        },
    }
    tcp, err := cfg.Listen(context.Background(), "tcp", "127.0.0.1:1234")
    if err != nil {
        fmt.Println("listen failed", err)
        return
    }

    buf := make([]byte, 1024)
    for {
        conn, err := tcp.Accept()
        if err != nil {
            fmt.Println("accept failed", err)
            continue
        }
        for {
            n, err := conn.Read(buf)
            if err != nil {
                fmt.Println("read failed", err)
                break
            }

            fmt.Println(string(buf[:n]))
        }
    }
}

The above is an iterative TCP server, which can only handle one connection at a time. But if we enable port multiplexing, if we start N such servers at the same time, they can handle N connections at the same time, and the process is preemptive.

In addition, we can listen to the same ports, with different IP addresses, to process different datagrams at the same time. For example, we can create two goroutines, one listening to 127.0.0.1:1234 and the other listening to 0.0.0.0:1234, to handle different sources differently.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

func serve(addr string) {
    cfg := net.ListenConfig{
        Control: func(network, address string, c syscall.RawConn) error {
            return c.Control(func(fd uintptr) {
                syscall.SetsockoptInt(int(fd), syscall.SOL_SOCKET, unix.SO_REUSEADDR, 1)
            })
        },
    }
    udp, err := cfg.ListenPacket(context.Background(), "udp", addr)

    if err != nil {
        fmt.Println("listen failed", err)
        return
    }

    buf := make([]byte, 1024)
    for {
        n, caddr, err := udp.ReadFrom(buf)
        if err != nil {
            fmt.Println("read failed", err)
            continue
        }

        fmt.Println(addr, caddr, string(buf[:n]))
    }
}

func main() {
    go serve("127.0.0.1:1234")
    go serve("0.0.0.0:1234")
    select {}
}

In the above example it is possible to distribute to different goroutines based on different destination IP addresses.