Use the reflect package to read and write variables of all types in the reflection world

golang reflect

The reflection package provided by Go in the standard library gives Go programs the ability to reflect at runtime (reflection), but this reflection ability is also a “double-edged sword”, it has the advantage of solving a specific class of problems, but also brings a lack of logic, performance problems, and difficult to find problems and debug It also brings disadvantages such as unclear logic, performance problems, and difficulty in problem identification and debugging. However, the reflect package, which has been released with Go since its inception, is an important and indispensable capability for Go. Whether you use it or not, it is important to master the basic methods of interacting with the type system using reflect, such as how to read and write variables of all types in a reflect world. In this article, we’ll take a quick look at using the reflect package to read and write Go basic type variables, composite type variables, and their applications.

1. basic types

There are two main gates into the reflect world: reflect.ValueOf and reflect.TypeOf. When we enter the reflect world, each variable can find a reflect.Value that corresponds to its own, through which we can read and write information of real-world variables. Here we mainly go through the methods to manipulate the value of each type of variable, so we mainly use reflect.ValueOf.

Go native basic types (non-compound types) mainly include:

Integer(int, int8, int16, int32(rune), int64, uint, uint8(byte), uint16, uint32, uint64)
Floating-point type(float32, float64)
Complex type(complex64, complex128)
Boolean type(bool)
String type (string)

How do we get the values of these type variables in the world of reflection, or how do we modify the values of these variables in the world of reflection? The following example can be used as a quick checklist for reading and writing Go basic type variables using reflect on a daily basis.

// github.com/bigwhite/experiments/blob/master/vars-in-reflect/basic/main.go

package main

import (
    "fmt"
    "reflect"
)

func main() {
    // 整型
    var i int = 11
    vi := reflect.ValueOf(i)                         // reflect Value of i
    fmt.Printf("i = [%d], vi = [%d]\n", i, vi.Int()) // i = [11], vi = [11]
    // vi.SetInt(11 + 100) // panic: reflect: reflect.Value.SetInt using unaddressable value

    vai := reflect.ValueOf(&i) // reflect Value of Address of i
    vi = vai.Elem()
    fmt.Printf("i = [%d], vi = [%d]\n", i, vi.Int()) // i = [11], vi = [11]
    vi.SetInt(11 + 100)
    fmt.Printf("after set, i = [%d]\n", i) // after set, i = [111]

    // 整型指针
    i = 11
    var pi *int = &i
    vpi := reflect.ValueOf(pi) // reflect Value of pi
    vi = vpi.Elem()
    vi.SetInt(11 + 100)
    fmt.Printf("after set, i = [%d]\n", i) // after set, i = [111]

    // 浮点型
    var f float64 = 3.1415

    vaf := reflect.ValueOf(&f)
    vf := vaf.Elem()
    fmt.Printf("f = [%f], vf = [%f]\n", f, vf.Float()) // f = [3.141500], vf = [3.141500]
    vf.SetFloat(100 + 3.1415)
    fmt.Printf("after set, f = [%f]\n", f) // after set, f = [103.141500]

    // 复数型
    var c = complex(5.1, 6.2)

    vac := reflect.ValueOf(&c)
    vc := vac.Elem()
    fmt.Printf("c = [%g], vc = [%g]\n", f, vc.Complex()) // c = [103.1415], vc = [(5.1+6.2i)]
    vc.SetComplex(complex(105.1, 106.2))
    fmt.Printf("after set, c = [%g]\n", c) // after set, c = [(105.1+106.2i)]

    // 布尔类型
    var b bool = true

    vab := reflect.ValueOf(&b)
    vb := vab.Elem()
    fmt.Printf("b = [%t], vb = [%t]\n", b, vb.Bool()) // b = [true], vb = [true]
    vb.SetBool(false)
    fmt.Printf("after set, b = [%t]\n", b) // after set, b = [false]

    // 字符串类型
    var s string = "hello, reflect"

    vas := reflect.ValueOf(&s)
    vs := vas.Elem()
    fmt.Printf("s = [%s], vs = [%s]\n", s, vs.String()) // s = [hello, reflect], vs = [hello, reflect]
    vs.SetString("bye, reflect")
    fmt.Printf("after set, s = [%s]\n", s) // after set, s = [bye, reflect]
}

We see that.

The native basic type variable enters the reflection world via reflect.ValueOf, and if the value of the original variable is eventually modified in the reflection world, then what is passed to ValueOf should not be the variable itself, but the address of the variable, except for the pointer type.
After entering the reflection world, use the Elem method of reflect.Value to get the real instance of Value that the pointer/address points to, and read the value of the variable through various “method sugars” provided by the Value type, such as reflect.Value.Int,reflect.Value.String,reflect.Value.Bool etc.
Similarly, in the reflection world, we set the value of the variable in question at runtime through the SetXXX series of methods of reflect.Value.

2. Composite types

As we have seen before, it is relatively easy to read and write variables of native basic types in the reflection world using the reflect package, so let’s take a look at reading and writing variables of composite type.

The composite types in Go include.

Arrays
slice
map
structure
channel

Unlike basic type variables, compound variables consist of both isomorphic and heteromorphic fields or elements, so how to read and write the value of a field or element in a compound type variable is what we need to consider. The following example can be used as a quick checklist for reading and writing field or element values in Go composite variables in the reflection world using reflect on a daily basis.

// github.com/bigwhite/experiments/blob/master/vars-in-reflect/composite/main.go

package main

import (
    "fmt"
    "reflect"
    "unsafe"
)

type Foo struct {
    Name string
    age  int
}

func main() {
    // 数组
    var a = [5]int{1, 2, 3, 4, 5}
    vaa := reflect.ValueOf(&a) // reflect Value of Address of arr
    va := vaa.Elem()
    va0 := va.Index(0)
    fmt.Printf("a0 = [%d], va0 = [%d]\n", a[0], va0.Int()) // a0 = [1], va0 = [1]
    va0.SetInt(100 + 1)
    fmt.Printf("after set, a0 = [%d]\n", a[0]) // after set, a0 = [101]

    // 切片
    var s = []int{11, 12, 13}
    vs := reflect.ValueOf(s)
    vs0 := vs.Index(0)
    fmt.Printf("s0 = [%d], vs0 = [%d]\n", s[0], vs0.Int()) // s0 = [11], vs0 = [11]
    vs0.SetInt(100 + 11)
    fmt.Printf("after set, s0 = [%d]\n", s[0]) // after set, s0 = [111]

    // map
    var m = map[int]string{
        1: "tom",
        2: "jerry",
        3: "lucy",
    }

    vm := reflect.ValueOf(m)
    vm_1_v := vm.MapIndex(reflect.ValueOf(1))                      // the reflect Value of the value of key 1
    fmt.Printf("m_1 = [%s], vm_1 = [%s]\n", m[1], vm_1_v.String()) // m_1 = [tom], vm_1 = [tom]
    vm.SetMapIndex(reflect.ValueOf(1), reflect.ValueOf("tony"))
    fmt.Printf("after set, m_1 = [%s]\n", m[1]) // after set, m_1 = [tony]

    // 为map m新增一组key-value
    vm.SetMapIndex(reflect.ValueOf(4), reflect.ValueOf("amy"))
    fmt.Printf("after set, m = [%#v]\n", m) // after set, m = [map[int]string{1:"tony", 2:"jerry", 3:"lucy", 4:"amy"}]

    // 结构体
    var f = Foo{
        Name: "lily",
        age:  16,
    }

    vaf := reflect.ValueOf(&f)
    vf := vaf.Elem()
    field1 := vf.FieldByName("Name")
    fmt.Printf("the Name of f = [%s]\n", field1.String()) // the Name of f = [lily]
    field2 := vf.FieldByName("age")
    fmt.Printf("the age of f = [%d]\n", field2.Int()) // the age of f = [16]

    field1.SetString("ally")
    // field2.SetInt(8) // panic: reflect: reflect.Value.SetInt using value obtained using unexported field
    nAge := reflect.NewAt(field2.Type(), unsafe.Pointer(field2.UnsafeAddr())).Elem()
    nAge.SetInt(8)
    fmt.Printf("after set, f is [%#v]\n", f) // after set, f is [main.Foo{Name:"ally", age:8}]

    // 接口
    var g = Foo{
        Name: "Jordan",
        age:  40,
    }

    // 接口底层动态类型为复合类型变量
    var i interface{} = &g
    vi := reflect.ValueOf(i)
    vg := vi.Elem()

    field1 = vg.FieldByName("Name")
    fmt.Printf("the Name of g = [%s]\n", field1.String()) // the Name of g = [Jordan]
    field2 = vg.FieldByName("age")
    fmt.Printf("the age of g = [%d]\n", field2.Int()) // the age of g = [40]

    nAge = reflect.NewAt(field2.Type(), unsafe.Pointer(field2.UnsafeAddr())).Elem()
    nAge.SetInt(50)
    fmt.Printf("after set, g is [%#v]\n", g) // after set, g is [main.Foo{Name:"Jordan", age:50}]

    // 接口底层动态类型为基本类型变量
    var n = 5
    i = &n
    vi = reflect.ValueOf(i).Elem()
    fmt.Printf("i = [%d], vi = [%d]\n", n, vi.Int()) // i = [5], vi = [5]
    vi.SetInt(10)
    fmt.Printf("after set, n is [%d]\n", n) // after set, n is [10]

    // channel
    var ch = make(chan int, 100)
    vch := reflect.ValueOf(ch)
    vch.Send(reflect.ValueOf(22))

    j := <-ch
    fmt.Printf("recv [%d] from channel\n", j) // recv [22] from channel

    ch <- 33
    vj, ok := vch.Recv()
    fmt.Printf("recv [%d] ok[%t]\n", vj.Int(), ok) // recv [33] ok[true]
}

From the above example, we can get some information as follows.

In the world of reflection, the reflect package provides corresponding methods for reading and writing elements or fields in composite types, such as Value.Index for array and slice elements, Value.MapIndex for map key-value, Field, FieldByName for structure fields, and Send and Recv for channel.
slices, maps and channels, because their underlying implementation is a pointer type structure, we can directly modify their internal elements in the reflection world using their corresponding Value in the reflection world.
For the non-exported fields in the structure (unexported field), we can read their values, but we cannot modify their values directly. In the above example, we have implemented the assignment of the value by the following unsafe means.

1
2

nAge = reflect.NewAt(field2.Type(), unsafe.Pointer(field2.UnsafeAddr())).Elem()
nAge.SetInt(50)

We create a new Value instance through reflect.NewAt, which represents a pointer to the field2 address. Then through the Elem method, we get the Value of the object pointed by that pointer Value: nAge, which is actually the field2 variable. The new value set by nAge will then also be reflected in the value of field2. This is similar to the function of vpi and vi in the basic type example above.

3. Get the value of the system resource descriptor

One of the functions of the reflect package is to get the value of some system resource descriptors that are encapsulated in the underlying system, such as socket descriptors and file descriptors.

a) File Descriptors

File provides the Fd method to get the value of the underlying os file descriptor. We can also use reflection to achieve the same functionality.

// github.com/bigwhite/experiments/blob/master/vars-in-reflect/system-resource/file_fd.go
package main

import (
    "fmt"
    "os"
    "reflect"
)

func fileFD(f *os.File) int {
    file := reflect.ValueOf(f).Elem().FieldByName("file").Elem()
    pfdVal := file.FieldByName("pfd")
    return int(pfdVal.FieldByName("Sysfd").Int())
}

func main() {
    fileName := os.Args[1]
    f, err := os.Open(fileName)
    if err != nil {
        panic(err)
    }

    defer f.Close()

    fmt.Printf("file descriptor is %d\n", f.Fd())
    fmt.Printf("file descriptor in reflect is %d\n", fileFD(f))
}

To execute the above example.

$go build file_fd.go
$./file_fd file_fd.go
file descriptor is 3
file descriptor in reflect is 3

We see that the fd value obtained by reflect is the same as the value obtained by the Fd method.

Here we can briefly analyze the implementation of the fileFD function based on the above understanding of reading and writing basic and compound type variables.

The definition of os.File is as follows.

// $GOROOT/src/os/types.go

type File struct {
        *file // os specific
}

To get the unexported pointer variable file by reflection, we use the following reflection statement.

`1`	`file := reflect.ValueOf(f).Elem().FieldByName("file").Elem()`

With the above Value instance file, we can proceed to reflect the os.file structure. os.file structure varies from os to os, taking the linux/mac unix as an example, the os.file structure is as follows.

// $GOROOT/src/os/file_unix.go

type file struct {
        pfd         poll.FD
        name        string
        dirinfo     *dirInfo // nil unless directory being read
        nonblock    bool     // whether we set nonblocking mode
        stdoutOrErr bool     // whether this is stdout or stderr
        appendMode  bool     // whether file is opened for appending
}

So we continue with the reflection.

`1`	`pfdVal := file.FieldByName("pfd")`

And the structure of poll.FD is as follows.

// $GOROOT/src/internal/poll/fd_unix.go

// field of a larger type representing a network connection or OS file.
type FD struct {
        // Lock sysfd and serialize access to Read and Write methods.
        fdmu fdMutex

        // System file descriptor. Immutable until Close.
        Sysfd int

        // I/O poller.
        pd pollDesc

        // Writev cache.
        iovecs *[]syscall.Iovec 

        // Semaphore signaled when file is closed.
        csema uint32

        // Non-zero if this file has been set to blocking mode.
        isBlocking uint32

        // Whether this is a streaming descriptor, as opposed to a
        // packet-based descriptor like a UDP socket. Immutable.
        IsStream bool

        // Whether a zero byte read indicates EOF. This is false for a
        // message based socket connection.
        ZeroReadIsEOF bool

        // Whether this is a file rather than a network socket.
        isFile bool
}

The Sysfd record is the value of the system file descriptor, so you can get the value of the file descriptor by using the following statement.

`1`	`return int(pfdVal.FieldByName("Sysfd").Int())`

b) socket descriptors

The socket descriptor is also a file descriptor, and Go does not provide an API to get the socket file descriptor directly in the standard library; we can only get it through reflection. See the following example.

// github.com/bigwhite/experiments/blob/master/vars-in-reflect/system-resource/socket_fd.go

package main

import (
    "fmt"
    "log"
    "net"
    "reflect"
)

func socketFD(conn net.Conn) int {
    tcpConn := reflect.ValueOf(conn).Elem().FieldByName("conn")
    fdVal := tcpConn.FieldByName("fd")
    pfdVal := fdVal.Elem().FieldByName("pfd")
    return int(pfdVal.FieldByName("Sysfd").Int())
}

func main() {

    ln, err := net.Listen("tcp", ":8080")
    if err != nil {
        panic(err)
    }

    for {
        conn, err := ln.Accept()
        if err != nil {
            if ne, ok := err.(net.Error); ok && ne.Temporary() {
                log.Printf("accept temp err: %v", ne)
                continue
            }

            log.Printf("accept err: %v", err)
            return
        }

        fmt.Printf("conn fd is [%d]\n", socketFD(conn))
    }
}

We see that the implementation of socketFD is somewhat similar to the implementation of fileFD, where we get the underlying Sysfd by reflecting from net.Conn step by step.

The real parameter passed to socketFD is essentially an instance of TCPConn, and we can get the value of this instance in the reflection world by reflect.ValueOf(conn).Elem().

// $GOROOT/src/net/tcpsock.go

type TCPConn struct {
        conn
}

Then get the Value of the field conn in the TCPConn structure in the reflection world by FieldByName(“conn”). net.conn structure is as follows.

// $GOROOT/src/net/net.go
type conn struct {
        fd *netFD
}

One of the netFDs is an os-related structure, taking linux/mac as an example, which has the following structure.

// $GOROOT/src/net/fd_posix.go

// Network file descriptor.
type netFD struct {
        pfd poll.FD

        // immutable until Close
        family      int
        sotype      int
        isConnected bool // handshake completed or use of association with peer
        net         string
        laddr       Addr
        raddr       Addr
}

We again see the poll.FD type field pfd, and further down the reflection is consistent with fileFD.

The source code covered in this article can be downloaded at here.

Table of Contents

1. basic types

2. Composite types

3. Get the value of the system resource descriptor

a) File Descriptors

b) socket descriptors