Data Reading in Golang

The standard library of Go provides several methods to read data from io.Reader (io, ioutil), and this article demonstrates the application scenarios by reading data from net.

Accessing a website using a TCP connection, using the HTTP 1.0 protocol, keeping the TCP connection short and closing it after reading the response, thus simulating the io.EOF error.

conn, err := net.Dial("tcp", "rpcx.site:80")
if err != nil {
	fmt.Println("dial error:", err)
	return
}
defer conn.Close()

io.Reader.Read

The io.Reader interface defines the Read(p []byte) (n int, err error) method, which we can use to read a batch of data from the Reader.

Read up to len(p) length of data at a time
If the read encounters error (io.EOF or other errors), the number of bytes of data read and error will be returned.
Even if the number of bytes read < len(p), it will not change the size of p (obviously, because normally it can’t change it)
When the input stream ends, calling it may return err == EOF or err == nil and n >= 0, but the next call will definitely return n = 0, err = io.EOF

Often this method is used to read data in bulk from the input stream until the input stream is read to the head, but it is important to note that we should always process the n bytes read first, and then process the error.

Reader interface, when implementing the Read method, it is not recommended that you return 0,nil unless the input len(p)==0, because this can cause confusion, and generally use 0,io.EOF to indicate that the input stream has been read to the head.

The following is an example demonstrating this method.

package main
import (
	"fmt"
	"io"
	"net"
	"strings"
)
func main() {
	// Establishing a connection
	conn, err := net.Dial("tcp", "rpcx.site:80")
	if err != nil {
		fmt.Println("dial error:", err)
		return
	}
	defer conn.Close()
	// Send request, http 1.0 protocol
	fmt.Fprintf(conn, "GET / HTTP/1.0\r\n\r\n")
	// Read response
	var sb strings.Builder
	buf := make([]byte, 256)
	for {
		n, err := conn.Read(buf)
		if err != nil {
			if err != io.EOF {
				fmt.Println("read error:", err)
			}
			break
		}
		sb.Write(buf[:n])
	}
	// Show results
	fmt.Println("response:", sb.String())
	fmt.Println("total response size:", sb.Len())
}

io.Copy

func Copy(dst Writer, src Reader) (write int64, err error) copies data from src to dst (the Go implementation generally puts the destination argument in front of the source argument) until.

EOF
Other errors

It returns the number of bytes read and the first error encountered

Note that after successfully reading the input stream err is not io.EOF, but nil. It does this internally by calling CopyBuffer(dst Writer, src Reader, buf []byte) (write int64, err error). CopyBuffer may use a specified buf to copy the data, if the length of buf is 0, it will panic.

In two special cases buf will not be used, but copied using the methods of the interface.

src implements the WriterTo interface
dst implements the ReadFrom interface

package main
import (
	"fmt"
	"io"
	"net"
	"strings"
)
func main() {
	// Establishing a connection
	conn, err := net.Dial("tcp", "rpcx.site:80")
	if err != nil {
		fmt.Println("dial error:", err)
		return
	}
	defer conn.Close()
	// Send request, http 1.0 protocol
	fmt.Fprintf(conn, "GET / HTTP/1.0\r\n\r\n")
	// Read response
	var sb strings.Builder
	_, err = io.Copy(&sb, conn)
	if err != nil {
		fmt.Println("read error:", err)
	}
	// Show results
	fmt.Println("response:", sb.String())
	fmt.Println("total response size:", sb.Len())
}

ioutil.ReadAll

ReadAll(r io.Reader) ([]byte, error) provides a method to read all the data from the input stream until an error (error) or a read to the head (EOF) occurs.

A successful header read does not return io.EOF, but rather the data and nil.

We often use it to read the trusted http.Response

package main
import (
	"fmt"
	"io"
	"io/ioutil"
	"net"
)
func main() {
	// Establishing a connection
	conn, err := net.Dial("tcp", "rpcx.site:80")
	if err != nil {
		fmt.Println("dial error:", err)
		return
	}
	defer conn.Close()
	// Send request, http 1.0 protocol
	fmt.Fprintf(conn, "GET / HTTP/1.0\r\n\r\n")
	// Read response
	data, err := ioutil.ReadAll(conn)
	if err != nil {
		if err != io.EOF {
			fmt.Println("read error:", err)
		}
		panic(err)
	}
	// Show results
	fmt.Println("response:", string(data)) 
	fmt.Println("total response size:", len(data))
}

io.ReadFull

ReadFull(r Reader, buf []byte) (n int, err error) reads exactly len(buf) bytes of data from the input stream.

Read 0 bytes and encounter EOF, return EOF
Read to 0<n<len(buf) bytes and encounter EOF, return ErrUnexpectedEOF
Read n==len(buf), even if error is encountered, return err=nil
return err ! = nil then definitely n<len(buf)

package main
import (
	"fmt"
	"io"
	"net"
	"strings"
)
func main() {
	// Establishing a connection
	conn, err := net.Dial("tcp", "rpcx.site:80")
	if err != nil {
		fmt.Println("dial error:", err)
		return
	}
	defer conn.Close()
	// Send request, http 1.0 protocol
	fmt.Fprintf(conn, "GET / HTTP/1.0\r\n\r\n")
	// Read response
	var sb strings.Builder
	buf := make([]byte, 256)
	for {
		n, err := io.ReadFull(conn, buf)
		if err != nil {
			if err != io.EOF && err != io.ErrUnexpectedEOF {
				fmt.Println("read error:", err)
			}
			break
		}
		sb.Write(buf[:n])
	}
	// Show results
	fmt.Println("response:", sb.String())
	fmt.Println("total response size:", sb.Len())
}

io.ReadAtLeast

ReadAtLeast(r Reader, buf []byte, min int) (n int, err error) Reads at least min bytes from the input stream into buf until buf is full or error is encountered, including EOF.

If n < min, an error must be returned
No data to read returns n=0 and EOF
n < min, and EOF is encountered, return n and ErrUnexpectedEOF
If min > len(buf), return 0, ErrShortBuffer
If at least min bytes are read, return err=nil even if an error is encountered

package main
import (
	"fmt"
	"io"
	"net"
	"strings"
)
func main() {
	// Establishing a connection
	conn, err := net.Dial("tcp", "rpcx.site:80")
	if err != nil {
		fmt.Println("dial error:", err)
		return
	}
	defer conn.Close()
	// Send request, http 1.0 protocol
	fmt.Fprintf(conn, "GET / HTTP/1.0\r\n\r\n")
	// Read response
	var sb strings.Builder
	buf := make([]byte, 256)
	for {
		n, err := io.ReadAtLeast(conn, buf, 256)
		if err != nil {
			if err != io.EOF && err != io.ErrUnexpectedEOF {
				fmt.Println("read error:", err)
			}
			break
		}
		sb.Write(buf[:n])
	}
	// Show results
	fmt.Println("response:", sb.String())
	fmt.Println("total response size:", sb.Len())
}

io.LimitReader

LimitReader(r Reader, n int64) Reader returns a Reader with restricted content length, when n bytes are read from this Reader for a while it will encounter EOF.

Imagine if you didn’t have this protection, say you were sent a picture and it ended up sending you a video of 3G size, which could potentially spike your memory.

It is to provide a protection function, the rest is no different from the ordinary Reader.

package main
import (
	"fmt"
	"io"
	"net"
	"strings"
)
func main() {
	// Establishing a connection
	conn, err := net.Dial("tcp", "rpcx.site:80")
	if err != nil {
		fmt.Println("dial error:", err)
		return
	}
	defer conn.Close()
	// Send request, http 1.0 protocol
	fmt.Fprintf(conn, "GET / HTTP/1.0\r\n\r\n")
	// Read response
	var sb strings.Builder
	buf := make([]byte, 256)
	rr := io.LimitReader(conn, 102400)
	for {
		n, err := io.ReadAtLeast(rr, buf, 256)
		if err != nil {
			if err != io.EOF && err != io.ErrUnexpectedEOF {
				fmt.Println("read error:", err)
			}
			break
		}
		sb.Write(buf[:n])
	}
	// Show results
	fmt.Println("response:", sb.String())
	fmt.Println("total response size:", sb.Len())
}

References

Reference https://colobu.com/2019/02/18/read-data-from-net-Conn/

Table of Contents