Principles of the Go language HTTP standard library implementation

The Hypertext Transfer Protocol (HTTP) is today’s most widely used application layer protocol, drafted by Tim Berners-Lee at CERN in 1989, and has become the core of data transfer on the Internet. Over the past few years, HTTP/2 and HTTP/3 have also updated the existing protocol to provide more secure and faster transfers. The existing protocols have been updated to provide more secure and faster transfers. Most programming languages implement HTTP/1.1 and HTTP/2.0 in their standard libraries to meet the daily development needs of engineers, and the network library for the Go language that we are presenting today also implements both major versions of the HTTP protocol.

Design Principles

HTTP is an application layer protocol, and in general we use TCP as the underlying transport layer protocol to transfer packets, but HTTP/3 implements a new transport layer protocol, QUIC, on top of the UDP protocol and uses QUIC to transfer data, which also means that HTTP can run on both TCP and UDP.

Before analyzing the internal implementation principles, let’s take a look at some of the design of the HTTP protocol and the relationship between the hierarchy and modules within the standard library.

Requests and responses

The most common concepts in the HTTP protocol are the HTTP request and response, which can be understood as messages passed between the client and the server, where the client sends an HTTP request to the server, and the server receives the HTTP request and computes it and sends it to the client as an HTTP response.

Unlike other binary protocols, which are text transfer protocols, the HTTP protocol headers are all text data. The first line of the HTTP request header will contain the method, path, and protocol version of the request, followed by multiple HTTP protocol headers and the load carried.

GET / HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE5.01; Windows NT)
Host: draveness.me
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Content-Length: <length>
Connection: Keep-Alive

<html>
    ...
</html>

HTTP responses also have a relatively similar structure, which also contains the protocol version, status code, response headers and load of the response, so we won’t go into that here.

Message Boundary

The HTTP protocol currently runs mainly on the TCP protocol, which is a connection-oriented, reliable, byte-stream-based transport layer communication protocol. The data handed over by the application layer to the TCP protocol is not transmitted to the destination host as a message, but in some cases is combined into a data segment and sent to the destination host. Because the TCP protocol is byte-stream based, TCP-based application layer protocols all need to delineate the boundaries of their messages themselves.

The HTTP protocol actually implements both of these solutions, and in most cases the HTTP protocol adds Content-Length to the protocol header to indicate the length of the load, and the recipient of the message parses the header to determine the current The recipient of the message parses the protocol header to determine where the current HTTP request/response ends and separates the different HTTP messages, as illustrated by the following example of using Content-Length to delimit the message boundary.

HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Content-Length: 138
...
Connection: close

<html>
  <head>
    <title>An Example Page</title>
  </head>
  <body>
    <p>Hello World, this is a very simple HTML document.</p>
  </body>
</html>

When HTTP uses the Chunked Transfer mechanism, the HTTP header no longer contains the Content-Length, but uses the HTTP message with a load size of 0 as the terminator to indicate the message boundary.

Hierarchy

The Go language wraps both HTTP client and server implementations in net/http, and to support better extensibility, it introduces the net/http.RoundTripper and net/http. The caller takes the request as an argument to get the response to the request, while net/http.Handler is mainly used by HTTP servers to respond to client requests: net/http.

1
2
3

type RoundTripper interface {
    RoundTrip(*Request) (*Response, error)
}

The receiver of an HTTP request can implement the net/http.Handler interface, which implements the logic for processing HTTP requests. will get the HTTP response, write the data to the load and write the response header.

type Handler interface {
    ServeHTTP(ResponseWriter, *Request)
}

type ResponseWriter interface {
	Header() Header
	Write([]byte) (int, error)
	WriteHeader(statusCode int)
}

Both the client and the server face a two-way HTTP request and response, with the client constructing the request and waiting for a response, and the server processing the request and returning a response. HTTP requests and responses have more than one implementation in the standard library, and they all contain a hierarchy. net/http.RoundTripper in the standard library contains the hierarchy shown below.

Each implementation of the net/http.RoundTripper interface contains a procedure for making requests to the remote; the standard library also provides multiple implementations of net/http.Handler to provide different services for client HTTP requests.

Clients

The client can either make HTTP requests directly via net/http.Get using the default client net/http.DefaultClient, or you can build your own new net/http.Client to implement a custom HTTP transaction. However, it should be noted that requests made with the default client do not have a timeout, so in some scenarios they will wait forever. In addition to custom HTTP transactions, we can also implement a custom net/http.CookieJar interface to manage and use cookies in HTTP requests.

Transactions and cookies are two of the most important modules we have available to us in the HTTP client package. In this section, we will analyze the principles of the client implementation, starting with the HTTP GET request and following the modules of building the request, transferring the data, getting the connection and waiting for the response. When we call net/http.Client.Get to send an HTTP, the following steps are performed.

call net/http.NewRequest to construct the request based on the method name, URL and request body.
call net/http.Transport.RoundTrip to open an HTTP transaction, obtain a connection, and send the request.
waiting for a response in the net/http.persistConn.readLoop method of an HTTP persistent connection.

The client side of HTTP contains several important constructs, namely net/http.

net/http.Client is the HTTP client, which defaults to the HTTP client using net/http.DefaultTransport.
net/http.Transport is an implementation of the net/http.RoundTripper interface, whose primary role is to support HTTP/HTTPS requests and HTTP proxies.
net/http.persistConn encapsulates a TCP persistent connection and is the handle (Handle) that we use to exchange messages with the remote.

Client net/http.Client is the higher level abstraction that provides some details of HTTP, including cookies and redirects; and net/http.Transport will handle the underlying implementation details of the HTTP/HTTPS protocol, which will include functions such as connection reuse, building requests, and sending requests.

Constructing a request

net/http.Request represents a request received by an HTTP service or sent by an HTTP client, and contains fields for the method, URL, protocol version, protocol header, and request body of the HTTP request, in addition to these fields, it holds a reference to the HTTP response at

type Request struct {
	Method string
	URL *url.URL

	Proto      string // "HTTP/1.0"
	ProtoMajor int    // 1
	ProtoMinor int    // 0

	Header Header
	Body io.ReadCloser

	...
	Response *Response
}

NewRequest is a method provided by the standard library for creating requests. This method checks the HTTP request fields and assembles them into a new request structure based on the input parameters.

func NewRequestWithContext(ctx context.Context, method, url string, body io.Reader) (*Request, error) {
	if method == "" {
		method = "GET"
	}
	if !validMethod(method) {
		return nil, fmt.Errorf("net/http: invalid method %q", method)
	}
	u, err := urlpkg.Parse(url)
	if err != nil {
		return nil, err
	}
	rc, ok := body.(io.ReadCloser)
	if !ok && body != nil {
		rc = ioutil.NopCloser(body)
	}
	u.Host = removeEmptyPort(u.Host)
	req := &Request{
		ctx:        ctx,
		Method:     method,
		URL:        u,
		Proto:      "HTTP/1.1",
		ProtoMajor: 1,
		ProtoMinor: 1,
		Header:     make(Header),
		Body:       rc,
		Host:       u.Host,
	}
	if body != nil {
		...
	}
	return req, nil
}

The request assembly process is relatively simple, it checks and verifies the input method, URL and load, however, after initializing the new net/http.Request structure, the process of handling the load is slightly more complex, we use different methods to wrap them into io.ReadCloser types depending on the type of load.

Opening a transaction

Once we have built the HTTP request using the standard library, we open the HTTP transaction to send the HTTP request and wait for a response from the remote, and after the following sequence of calls, we end up at the structure where the standard library implements the underlying HTTP protocol - net/http.Transport.

net/http.Client.Do
net/http.Client.do
net/http.Client.send
net/http.send
net/http.Transport.RoundTrip

Transport implements the net/http.RoundTripper interface, which is the most important and complex structure in the entire request process, and sends HTTP requests and waits for responses in net/http. We can divide the execution of this function into two parts.

Finding and executing a custom implementation of net/http.RoundTripper based on the protocol of the URL.
fetching or initializing a new persistent connection from the connection pool and calling the connection’s net/http.persistConn.roundTrip to make the request.

We can register the net/http.RoundTripper implementation for different protocols by calling net/http.Transport.RegisterProtocol in the standard library’s net/http.Transport, and in the following code the corresponding implementation will be selected based on the protocol in the URL instead of the default logic.

func (t *Transport) roundTrip(req *Request) (*Response, error) {
	ctx := req.Context()
	scheme := req.URL.Scheme

	if altRT := t.alternateRoundTripper(req); altRT != nil {
		if resp, err := altRT.RoundTrip(req); err != ErrSkipAltProtocol {
			return resp, err
		}
	}
	...
}

By default, HTTP requests are handled using net/http.persistConn persistent connections, which first fetches the connection used to send the request and then calls net/http.persistConn.roundTrip.

func (t *Transport) roundTrip(req *Request) (*Response, error) {
	...
	for {
		select {
		case <-ctx.Done():
			return nil, ctx.Err()
		default:
		}

		treq := &transportRequest{Request: req, trace: trace}
		cm, err := t.connectMethodForRequest(treq)
		if err != nil {
			return nil, err
		}

		pconn, err := t.getConn(treq, cm)
		if err != nil {
			return nil, err
		}

		resp, err := pconn.roundTrip(treq)
		if err == nil {
			return resp, nil
		}
	}
}

net/http.Transport.getConn is the method to get the connection which will be used to send the request by two methods.

func (t *Transport) getConn(treq *transportRequest, cm connectMethod) (pc *persistConn, err error) {
	req := treq.Request
	ctx := req.Context()

	w := &wantConn{
		cm:         cm,
		key:        cm.key(),
		ctx:        ctx,
		ready:      make(chan struct{}, 1),
	}

	if delivered := t.queueForIdleConn(w); delivered {
		return w.pc, nil
	}

	t.queueForDial(w)
	select {
	case <-w.ready:
		...
		return w.pc, w.err
	...
	}
}

call net/http.Transport.queueForIdleConn to wait in the queue for an idle connection.
call net/http.Transport.queueForDial to wait in the queue for a new connection to be established.

Connections are a relatively expensive resource, and establishing a new connection before each HTTP request can consume more time and incur more overhead. Allocating and reusing resources through connection pooling can effectively improve the overall performance of HTTP requests, and most network library clients adopt a similar strategy for reusing resources.

When we call net/http.Transport.queueForDial to try to establish a connection with the remote, the standard library starts a new Goroutine internally to execute net/http.Transport.dialConnFor for connection establishment, and from the final call to net/http. dialConn we can find the TCP connection and the net library in the final call: net/http.

func (t *Transport) dialConn(ctx context.Context, cm connectMethod) (pconn *persistConn, err error) {
	pconn = &persistConn{
		t:             t,
		cacheKey:      cm.key(),
		reqch:         make(chan requestAndChan, 1),
		writech:       make(chan writeRequest, 1),
		closech:       make(chan struct{}),
		writeErrCh:    make(chan error, 1),
		writeLoopDone: make(chan struct{}),
	}

	conn, err := t.dial(ctx, "tcp", cm.addr())
	if err != nil {
		return nil, err
	}
	pconn.conn = conn

	pconn.br = bufio.NewReaderSize(pconn, t.readBufferSize())
	pconn.bw = bufio.NewWriterSize(persistConnWriter{pconn}, t.writeBufferSize())

	go pconn.readLoop()
	go pconn.writeLoop()
	return pconn, nil
}

After creating a new TCP connection, we also create two Goroutines in the background for the current connection to read data from or write data to the TCP connection, respectively.

Waiting for requests

A persistent TCP connection implements net/http.persistConn.roundTrip to handle writing HTTP requests and waiting for the return of the response in the select statement.

func (pc *persistConn) roundTrip(req *transportRequest) (resp *Response, err error) {
	writeErrCh := make(chan error, 1)
	pc.writech <- writeRequest{req, writeErrCh, continueCh}

	resc := make(chan responseAndError)
	pc.reqch <- requestAndChan{
		req:        req.Request,
		ch:         resc,
	}

	for {
		select {
		case re := <-resc:
			if re.err != nil {
				return nil, pc.mapRoundTripError(req, startBytesWritten, re.err)
			}
			return re.res, nil
		...
		}
	}
}

Each HTTP request is cyclically written by net/http.persistConn.writeLoop in another Goroutine, which executes independently and communicates through a channel. net/http.Request.write composes TCP data segments according to the HTTP protocol based on the fields in the net/http. Request structure will compose TCP data segments according to the HTTP protocol.

func (pc *persistConn) writeLoop() {
	defer close(pc.writeLoopDone)
	for {
		select {
		case wr := <-pc.writech:
			startBytesWritten := pc.nwrite
			wr.req.Request.write(pc.bw, pc.isProxy, wr.req.extra, pc.waitForContinue(wr.continueCh))
			...
		case <-pc.closech:
			return
		}
	}
}

When we call net/http.Request.write to write data to the request, it is actually written directly to the TCP connection in net/http.persistConnWriter, and the TCP stack takes care of sending the contents of the HTTP request to the target server:

type persistConnWriter struct {
	pc *persistConn
}

func (w persistConnWriter) Write(p []byte) (n int, err error) {
	n, err = w.pc.conn.Write(p)
	w.pc.nwrite += int64(n)
	return
}

Another read loop in a persistent connection, net/http.persistConn.readLoop, is responsible for reading data from the TCP connection and sending it to the caller of the HTTP request, but it is net/http.ReadResponse that is really responsible for parsing the HTTP protocol.

func ReadResponse(r *bufio.Reader, req *Request) (*Response, error) {
	tp := textproto.NewReader(r)
	resp := &Response{
		Request: req,
	}

	line, _ := tp.ReadLine()
	if i := strings.IndexByte(line, ' '); i == -1 {
		return nil, badStringError("malformed HTTP response", line)
	} else {
		resp.Proto = line[:i]
		resp.Status = strings.TrimLeft(line[i+1:], " ")
	}

	statusCode := resp.Status
	if i := strings.IndexByte(resp.Status, ' '); i != -1 {
		statusCode = resp.Status[:i]
	}
	resp.StatusCode, err = strconv.Atoi(statusCode)

	resp.ProtoMajor, resp.ProtoMinor, _ = ParseHTTPVersion(resp.Proto)

	mimeHeader, _ := tp.ReadMIMEHeader()
	resp.Header = Header(mimeHeader)

	readTransfer(resp, r)
	return resp, nil
}

We can see the general framework of the HTTP response structure in the above method, which contains the status code, protocol version, request headers, etc. The response body is still parsed in the read loop net/http.persistConn.readLoop based on the HTTP protocol headers.

Server

The Go language standard library, the net/http package, provides a very easy-to-use interface that allows us to quickly build new HTTP services using the functionality provided by the standard library as follows.

func handler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hi there, I love %s!", r.URL.Path[1:])
}

func main() {
    http.HandleFunc("/", handler)
    log.Fatal(http.ListenAndServe(":8080", nil))
}

The main function above calls only two functions provided by the standard library, they are net/http.HandleFunc function for registering processors and net/http.ListenAndServe for listening and handling requests, most server frameworks will include these two types of interfaces for registering processors and handling external requests respectively, which A very common pattern, and we will also describe here how the standard library supports HTTP server implementations along these two dimensions.

Registering processors

The HTTP service is composed of a set of processors that implement the net/http.Handler interface and process HTTP requests by selecting the appropriate processor based on the request route.

When we call net/http.HandleFunc directly to register a handler, the standard library uses the default HTTP server net/http.DefaultServeMux to process the request, and this method will call net/http.ServeMux.HandleFunc directly

1
2
3

func (mux *ServeMux) HandleFunc(pattern string, handler func(ResponseWriter, *Request)) {
	mux.Handle(pattern, HandlerFunc(handler))
}

The above method converts the processor to the net/http.Handler interface type calling net/http.ServeMux.Handle to register the processor.

func (mux *ServeMux) Handle(pattern string, handler Handler) {
	if _, exist := mux.m[pattern]; exist {
		panic("http: multiple registrations for " + pattern)
	}

	e := muxEntry{h: handler, pattern: pattern}
	mux.m[pattern] = e
	if pattern[len(pattern)-1] == '/' {
		mux.es = appendSorted(mux.es, e)
	}

	if pattern[0] != '/' {
		mux.hosts = true
	}
}

The route and corresponding processor are composed into net/http.DefaultServeMux, which holds a net/http.muxEntry hash that stores the mapping from URLs to processors, which is used by the HTTP server to find processors when processing requests.

Processing requests

The standard library provides net/http.ListenAndServe to listen for TCP connections and process requests. This function initializes an HTTP server, net/http.Server, with the incoming listen address and processor, and calls the server’s net/http.Server.ListenAndServe method This function initializes an HTTP server with the incoming listener address and handler, calls the net/http.

func ListenAndServe(addr string, handler Handler) error {
	server := &Server{Addr: addr, Handler: handler}
	return server.ListenAndServe()
}

Server.ListenAndServe listens for TCP connections at the corresponding address and processes client requests via net/http.Server.

func (srv *Server) ListenAndServe() error {
	if addr == "" {
		addr = ":http"
	}
	ln, err := net.Listen("tcp", addr)
	if err != nil {
		return err
	}
	return srv.Serve(ln)
}

Serve listens for external TCP connections in the loop and calls net/http.Server.newConn for each connection to create a new net/http.conn, which is the server-side representation of the HTTP connection.

func (srv *Server) Serve(l net.Listener) error {
	l = &onceCloseListener{Listener: l}
	defer l.Close()

	baseCtx := context.Background()
	ctx := context.WithValue(baseCtx, ServerContextKey, srv)
	for {
		rw, err := l.Accept()
		if err != nil {
			select {
			case <-srv.getDoneChan():
				return ErrServerClosed
			default:
			}
			...
			return err
		}
		connCtx := ctx
		c := srv.newConn(rw)
		c.setState(c.rwc, StateNew) // before Serve can return
		go c.serve(connCtx)
	}
}

After creating a server-side connection, the implementation in the standard library creates a separate Goroutine for each HTTP request and calls the net/http.Conn.serve method in it. requests.

func (c *conn) serve(ctx context.Context) {
	c.remoteAddr = c.rwc.RemoteAddr().String()

	ctx = context.WithValue(ctx, LocalAddrContextKey, c.rwc.LocalAddr())
	ctx, cancelCtx := context.WithCancel(ctx)
	c.cancelCtx = cancelCtx
	defer cancelCtx()

	c.r = &connReader{conn: c}
	c.bufr = newBufioReader(c.r)
	c.bufw = newBufioWriterSize(checkConnErrorWriter{c}, 4<<10)

	for {
		w, _ := c.readRequest(ctx)
		serverHandler{c.server}.ServeHTTP(w, w.req)
		w.finishRequest()
		...
	}
}

The above code snippet is our simplified connection handling process, which consists of reading the HTTP request, calling the Handler to process the HTTP request, and calling the completion of the request. The read HTTP request calls net/http.Conn.readRequest, which takes the HTTP request from the connection and constructs a variable net/http.response that implements the net/http.ResponseWriter interface, and any data written to this structure is forwarded to the buffer it holds in.

func (w *response) write(lenData int, dataB []byte, dataS string) (n int, err error) {
	...
	w.written += int64(lenData)
	if w.contentLength != -1 && w.written > w.contentLength {
		return 0, ErrContentLength
	}
	if dataB != nil {
		return w.w.Write(dataB)
	} else {
		return w.w.WriteString(dataS)
	}
}

After parsing the HTTP request and initializing net/http.ResponseWriter, we can then call net/http.serverHandler.ServeHTTP lookup handler to process the HTTP request.

type serverHandler struct {
	srv *Server
}

func (sh serverHandler) ServeHTTP(rw ResponseWriter, req *Request) {
	handler := sh.srv.Handler
	if handler == nil {
		handler = DefaultServeMux
	}
	if req.RequestURI == "*" && req.Method == "OPTIONS" {
		handler = globalOptionsHandler{}
	}
	handler.ServeHTTP(rw, req)
}

If the current HTTP server does not contain any processors, we use the default net/http.DefaultServeMux to handle external HTTP requests.

ServeMux is a multiplexer for HTTP requests that receives external HTTP requests, matches and calls the most appropriate processor based on the requested URL: net/http.

func (mux *ServeMux) ServeHTTP(w ResponseWriter, r *Request) {
	h, _ := mux.Handler(r)
	h.ServeHTTP(w, r)
}

After a series of function calls, the above process culminates in a call to the HTTP server’s net/http.ServerMux.match, which traverses the previously registered routing table and matches it according to specific rules.

func (mux *ServeMux) match(path string) (h Handler, pattern string) {
	v, ok := mux.m[path]
	if ok {
		return v.h, v.pattern
	}

	for _, e := range mux.es {
		if strings.HasPrefix(path, e.pattern) {
			return e.h, e.pattern
		}
	}
	return nil, ""
}

If the path of the request and the table entry in the route match successfully, we call the corresponding processor in the table entry, and the business logic contained in the processor builds the response corresponding to the HTTP request via net/http.ResponseWriter and sends it back to the client over a TCP connection.

Summary

The Go language HTTP standard library provides a very rich set of features. Many languages’ standard libraries provide only the most basic features, and the implementation of HTTP clients and servers often requires the use of other open source frameworks, but many Go language projects use the standard library directly to implement HTTP servers, which also illustrates the value of the Go language standard library from the side.

Table of Contents