This article explains the sendfile command in the http module in nginx and how it works.

1. Introduction to sendfile()

The http module of nginx has a sendfile directive, which is on by default. The official website documentation explains it as follows.

Enables or disables the use of sendfile().

Starting from nginx 0.8.12 and FreeBSD 5.2.1, aio can be used to pre-load data for sendfile():

1
2
3
4
5

location /video/ {
   sendfile       on;
   tcp_nopush     on;
   aio            on;
}

In this configuration, sendfile() is called with the SF_NODISKIO flag which causes it not to block on disk I/O, but, instead, report back that the data are not in memory. nginx then initiates an asynchronous data load by reading one byte. On the first read, the FreeBSD kernel loads the first 128K bytes of a file into memory, although next reads will only load data in 16K chunks. This can be changed using the read_ahead directive.

Before version 1.7.11, pre-loading could be enabled with aio sendfile;.

Simply put, enabling the sendfile() system call to replace read() and write() calls reduces system context switching and thus improves performance, which can greatly improve nginx performance when it is a static file server, but is of little use when it is a reverse proxy server. Let’s analyze how this sendfile works.

2. Principle analysis

First of all, we need to know that the biggest difference between sendfile() and read() and write() is that the former is a system call, while the latter is a function call, let’s look at the following diagram.

difference between sendfile() and read() and write()

It is easy to see that nginx belongs to Applicaiton, while read() and write() belong to function calls, that is, in the Lib Func layer, sendfile() system call, located in the System Call layer, and want to operate on the hard disk, is the kernel only has the permission. All of the above layers need to be called downwards.

As a comparison let’s take a look at the operation process if nginx calls read() and write() functions under normal circumstances.

We all know that data is stored on the hard disk, and when it is called, it is loaded into memory and then cascaded to the CPU. Here we simplify this process.

  • Step 1: First, nginx calls the read function, at which time the data is loaded from the harddisk into the Kernel Buffer (Hard Disk), which is the user mode from the beginning to the kernel mode in order to complete the operation.
  • Step 2: Then the data is transferred from the Kernel Buffer (Hard Disk) to the User Buffer because the data needs to be operated by the write() function, so the data is switched from the kernel mode back to the user mode.
  • Step 3: Then the data is written from user buffer to Kernel Buffer (Socket Engine) by write() function, which is switched from user mode to kernel mode.
  • Step 4: data is transferred from the Kernel Buffer (Socket Engine) to the Socket Engine, which switches from kernel mode back to user mode.

Two points need to be clarified here, one is that switching between user state and kernel state requires a context switch operation, which is a very system resource and time consuming operation, and the other is that read() and write() are function calls, which do not have permission to operate in the kernel and cannot transfer data directly from Kernel Buffer (Hard Disk) to Kernel Buffer (Socket Engine). They cannot transfer data directly from Kernel Buffer (Hard Disk) to Kernel Buffer (Socket Engine).

What about using sendfile()? Since it is a system call, it is not necessary to transfer the data to the User Buffer in steps 2 and 3, and the operation is performed directly in the kernel, eliminating two state switches, i.e., two context switches, thus significantly improving performance.

Let’s take a look at the following diagram.

sendfile

Finally, let’s explain why sendfile() is not useful when nginx is used as a reverse proxy server.

As the name implies, sendfile() is used to send files, which means that the receiving end of the data is a file handle and the sending end is a socket, but when you are a reverse proxy server, both ends are sockets, so you can’t use sendfile() and there is no performance improvement.