JSON (JavaScript Object Notation), a lightweight data interchange format1, holds almost a majority of the market share today. Although it lacks serialization and deserialization performance compared to more compact data interchange formats, JSON offers good readability and ease of use, and it is a very good choice for using JSON as a serialization format when extreme mechanical performance is not sought.

Design Principles

Almost all modern programming languages incorporate functions for handling JSON directly into the standard library, and the Go language is no exception. It provides standard JSON serialization and deserialization methods to the outside world via encoding/json, namely encoding/json.Marshal and encoding/json.Unmarshal, which are also the Marshal and encoding/json.Unmarshal, which are also the two most commonly used methods in the package.

The JSON serialization process in the Go language does not require any pre-implementation of the interface by the object being serialized, it takes the values in the structure or array through reflection and encodes them recursively in a tree structure. The standard library also decodes JSON based on the values passed in encoding/json.

The Go JSON library encoding and decoding process makes extensive use of reflection, and you will see a lot of reflection code in the second half of this section, so we won’t go into that. Here we will briefly introduce the interfaces and tags in the JSON standard library, which are among the few interfaces it provides for developers to influence the decoding and encoding process.

Interfaces

Marshaler and encoding/json.Unmarshaler interfaces are provided in the JSON standard library to affect the serialization and deserialization results of JSON, respectively.

1
2
3
4
5
6
7

type Marshaler interface {
	MarshalJSON() ([]byte, error)
}

type Unmarshaler interface {
	UnmarshalJSON([]byte) error
}

In the process of JSON serialization and deserialization, it will use reflection to determine whether the structure type implements the above interface, and if it does, it will give priority to the corresponding method for encoding and decoding operations. TextMarshaler and encoding.TextUnmarshaler.

1
2
3
4
5
6
7

type TextMarshaler interface {
	MarshalText() (text []byte, err error)
}

type TextUnmarshaler interface {
	UnmarshalText(text []byte) error
}

Once a JSON-related serialization method is found not to be implemented, the above two methods are called by the JSON standard library as candidates and participate in the coding and decoding process. In summary, we can implement these four methods on any type to customize the end result, with the latter two methods being more general in scope, but not being called by the JSON standard library in preference.

Tags

The Go language structure tag is also an interesting feature. By default, when we serialize and deserialize structures, the standard library assumes a one-to-one correspondence between field names and keys in JSON, however, Go language fields are generally camel nomenclature, and underscore nomenclature is relatively common in JSON, so it is a very convenient design to use the tag feature to directly establish a mapping between keys and fields.

The tag in JSON consists of two parts, as shown below, name and age are tag names, and all the strings after them are tag options, i.e. encoding/json.tagOptions, which establish a one-to-one correspondence between tag names and field names, and the tag options after them will also affect the coding and decoding process.

1
2
3
4

type Author struct {
    Name string `json:"name,omitempty"`
    Age  int32  `json:"age,string,omitempty"`
}

Two common tags are string and omitempty. The former indicates that the current integer or floating point number is represented by a string in JSON, while the other field, omitempty, ignores the corresponding key-value pair directly in the generated JSON when the field has a zero value, e.g., “age”: 0, “author”: “”, etc. The standard library will use encoding/json.parseTag to parse the tag as follows.

1
2
3
4
5
6

func parseTag(tag string) (string, tagOptions) {
	if idx := strings.Index(tag, ","); idx != -1 {
		return tag[:idx], tagOptions(tag[idx+1:])
	}
	return tag, tagOptions("")
}

From the implementation of this method, we can analyze what form the legal tags in the JSON standard library take: tag names and tag options are concatenated with , and the leading string is the tag name, followed by all the tag options.

Serialization

Marshal is the simplest serialization function available in the JSON standard library. It takes a value of type interface{} as an argument, which means that almost all Go language variables can be serialized by the JSON standard library. In order to provide such complex and generic functionality, it is common to use reflection in static languages, so let’s take a deeper look at its implementation.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

func Marshal(v interface{}) ([]byte, error) {
	e := newEncodeState()
	err := e.marshal(v, encOpts{escapeHTML: true})
	if err != nil {
		return nil, err
	}
	buf := append([]byte(nil), e.Bytes()...)
	encodeStatePool.Put(e)
	return buf, nil
}

The above method will call encoding/json.newEncodeState to get encoding/json.encodeState from the global pool of encoding states, which will be used for all subsequent serializations, and the structure will be put back into the pool for reuse after the encoding is done.

Following the complex call stack shown above, a series of serialization methods at the end get the reflected type of the object and call encoding/json.newTypeEncoder, a core encoding method that recursively finds the corresponding encoding method for all types, although its execution can be divided into two steps as follows.

  1. obtaining a user-defined encoding/json.Marshaler or encoding.TextMarshaler encoder.
  2. getting the JSON encoder built into the standard library for the base type.

In the first part of the method, we check whether the type of the current value can use the user-defined encoder, and here there are two different ways of determining this.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

func newTypeEncoder(t reflect.Type, allowAddr bool) encoderFunc {
	if t.Kind() != reflect.Ptr && allowAddr && reflect.PtrTo(t).Implements(marshalerType) {
		return newCondAddrEncoder(addrMarshalerEncoder, newTypeEncoder(t, false))
	}
	if t.Implements(marshalerType) {
		return marshalerEncoder
	}
	if t.Kind() != reflect.Ptr && allowAddr && reflect.PtrTo(t).Implements(textMarshalerType) {
		return newCondAddrEncoder(addrTextMarshalerEncoder, newTypeEncoder(t, false))
	}
	if t.Implements(textMarshalerType) {
		return textMarshalerEncoder
	}
	...
}

  1. If the current value is a value type, can take an address and the corresponding pointer type of the value type implements the encoding/json.Marshaler interface, call encoding/json.newCondAddrEncoder to get a conditional encoder, which will reselect a new encoder when encoding/json. addrMarshalerEncoder fails to reselect a new encoder.

  2. can be serialized directly using encoding/json.marshalerEncoder if the current type implements the encoding/json.Marshaler interface.

TextMarshaler, except that it first determines the encoding/json.Marshaler interface, which confirms what we speculated in the Design Principles section.

encoding/json.newTypeEncoder will get the corresponding encoder based on the reflected type of the incoming value, including encoders for basic types such as bool, int, float, etc. and complex types such as arrays, structures, slices, etc.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

func newTypeEncoder(t reflect.Type, allowAddr bool) encoderFunc {
	...
	switch t.Kind() {
	case reflect.Bool:
		return boolEncoder
	case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
		return intEncoder
	case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr:
		return uintEncoder
	case reflect.Float32:
		return float32Encoder
	case reflect.Float64:
		return float64Encoder
	case reflect.String:
		return stringEncoder
	case reflect.Interface:
		return interfaceEncoder
	case reflect.Struct:
		return newStructEncoder(t)
	case reflect.Map:
		return newMapEncoder(t)
	case reflect.Slice:
		return newSliceEncoder(t)
	case reflect.Array:
		return newArrayEncoder(t)
	case reflect.Ptr:
		return newPtrEncoder(t)
	default:
		return unsupportedTypeEncoder
	}
}

We won’t cover all the built-in type encoders here, but just a few of them to help you understand the overall design. First, let’s look at the boolean JSON encoder, which is a very simple implementation and not even much to cover:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

func boolEncoder(e *encodeState, v reflect.Value, opts encOpts) {
	if opts.quoted {
		e.WriteByte('"')
	}
	if v.Bool() {
		e.WriteString("true")
	} else {
		e.WriteString("false")
	}
	if opts.quoted {
		e.WriteByte('"')
	}
}

It will write a different string to the encoding state, i.e. true or false, depending on the current value, in addition to deciding whether to write double quotes around the boolean value depending on the encoding configuration", while all other encoders of basic types are similar.

Encoders for complex types have a relatively complex control structure, so we will take encoding/json.structEncoder for structures as an example to introduce their principle. encoding/json.newStructEncoder will call encoding/json.typeEncoder for all fields of the current structure. typeEncoder for all fields of the current structure and returns encoding/json.structEncoder.encode.

1
2
3
4

func newStructEncoder(t reflect.Type) encoderFunc {
	se := structEncoder{fields: cachedTypeFields(t)}
	return se.encode
}

We can see the result of structure serialization from the implementation of encoding/json.structEncoder.encode, which iterates through all the fields in the structure, and after writing the field name, it calls the encoding method of the field’s corresponding type to write the JSON corresponding to that field into the buffer:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

func (se structEncoder) encode(e *encodeState, v reflect.Value, opts encOpts) {
	next := byte('{')
FieldLoop:
	for i := range se.fields.list {
		f := &se.fields.list[i]

		fv := v
		for _, i := range f.index {
			if fv.Kind() == reflect.Ptr {
				if fv.IsNil() {
					continue FieldLoop
				}
				fv = fv.Elem()
			}
			fv = fv.Field(i)
		}

		if f.omitEmpty && isEmptyValue(fv) {
			continue
		}
		e.WriteByte(next)
		next = ','
		e.WriteString(f.nameNonEsc)
		opts.quoted = f.quoted
		f.encoder(e, fv, opts)
	}
	if next == '{' {
		e.WriteString("{}")
	} else {
		e.WriteByte('}')
	}
}

The implementation principle of encoders such as arrays and pointers is not much different from this method, as they all recursively call the encoding method holding the field using a similar strategy, which results in a tree structure as shown in the figure below.

After getting the encoder of the whole tree, encoding/json.encodeState.reflectValue is called from the root node and the serialization function of the whole tree is called in turn. The whole process of JSON serialization looks up the encoding methods of the types and subtypes and calls them, which makes use of a lot of reflection features to be generic enough.

Deserialization

Unmarshal handles the deserialization of JSON. Compared to serialization where the execution is deterministic, the process of deserialization is a gradual exploration process, so it is much more complex and the overhead is several times higher. Because of the limited expressiveness of the Go language, the use of deserialization is relatively cumbersome, so a variable needs to be passed in to help the standard library perform deserialization.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

func Unmarshal(data []byte, v interface{}) error {
	var d decodeState
	err := checkValid(data, &d.scan)
	if err != nil {
		return err
	}

	d.init(data)
	return d.unmarshal(v)
}

Before actually performing the deserialization, we will call encoding/json.checkValid to verify the legitimacy of the incoming JSON to ensure that no syntax errors are encountered during the deserialization process. unmarshal to start the deserialization.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

func (d *decodeState) unmarshal(v interface{}) error {
	rv := reflect.ValueOf(v)
	if rv.Kind() != reflect.Ptr || rv.IsNil() {
		return &InvalidUnmarshalError{reflect.TypeOf(v)}
	}
	d.scan.reset()
	d.scanWhile(scanSkipSpace)
	err := d.value(rv)
	if err != nil {
		return d.addErrorContext(err)
	}
	return d.savedError
}

If the value passed in is not a pointer or is a null pointer, the current method will return the error we often see encoding/json.InvalidUnmarshalError, which can be converted to “json: Unmarshal(non-pointer xxx)” using formatted output . The encoding/json.decodeState.value called by this method is the execution entry point for all deserialization processes.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

func (d *decodeState) value(v reflect.Value) error {
	switch d.opcode {
	default:
		panic(phasePanicMsg)
	case scanBeginArray:
		...
	case scanBeginLiteral:
		...
	case scanBeginObject:
		if v.IsValid() {
			if err := d.object(v); err != nil {
				return err
			}
		} else {
			d.skip()
		}
		d.scanNext()
	}
	return nil
}

This method, as the top-level deserialization method, can receive three different types of values, that is, arrays, literals and objects, all three types can be used as the top-level object of JSON, let’s first understand how the standard library parses the objects in JSON, the process will use encoding/json.decodeState.object for deserialization. which first calls encoding/json.indirect to find the non-pointer type corresponding to the current type: encoding/json.

1
2
3
4
5
6
7
8
9

func (d *decodeState) object(v reflect.Value) error {
	u, ut, pv := indirect(v, false)
	if u != nil {
		start := d.readIndex()
		d.skip()
		return u.UnmarshalJSON(d.data[start:d.off])
	}
	...
}

During the call to encoding/json.indirect, if the current value is of type **Type, it will check if the types **Type, *Type and Type in turn implement the encoding/json.Unmarshal or encoding. interface; if it does, the standard library will call UnmarshalJSON directly to complete the deserialization using a developer-defined method.

In other cases, we still fall back to the default logic for handling key-value pairs in objects, as shown in the code below, which calls encoding/json.decodeState.rescanLiteral to scan the JSON for keys and find the reflected value of the corresponding field in the structure, then continues to scan for symbols : and calls encoding/ json.decodeState.value to parse the corresponding value : and call encoding/json.decodeState.value to parse the corresponding value.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

func (d *decodeState) object(v reflect.Value) error {
	...
	v = pv
	t := v.Type()
	fields = cachedTypeFields(t)
	for {
		start := d.readIndex()
		d.rescanLiteral()
		item := d.data[start:d.readIndex()]
		key, _ := d.unquoteBytes(item)
		var subv reflect.Value
		var f *field
		if i, ok := fields.nameIndex[string(key)]; ok {
			f = &fields.list[i]
		}
		if f != nil {
			subv = v
			for _, i := range f.index {
				subv = subv.Field(i)
			}
		}

		if d.opcode != scanObjectKey {
			panic(phasePanicMsg)
		}
		d.scanWhile(scanSkipSpace)

		if err := d.value(subv); err != nil {
			return err
		}
		if d.opcode == scanEndObject {
			break
		}
	}
	return nil
}

When the above method calls encoding/json.decodeState.value, the method re-determines whether the value corresponding to the key is an object, an array or a literal, because arrays and objects are collection types, so the method scans them recursively. how they are processed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

func (d *decodeState) value(v reflect.Value) error {
	switch d.opcode {
	default:
		panic(phasePanicMsg)

	case scanBeginArray:
		...
	case scanBeginObject:
		...
	case scanBeginLiteral:
		start := d.readIndex()
		d.rescanLiteral()
		if v.IsValid() {
			if err := d.literalStore(d.data[start:d.readIndex()], v, false); err != nil {
				return err
			}
		}
	}
	return nil
}

The literal is scanned by encoding/json.decodeState.rescanLiteral, which sequentially scans the buffer for characters and slices the string according to the characters, somewhat like a compiler’s lexical analysis:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

func (d *decodeState) rescanLiteral() {
	data, i := d.data, d.off
Switch:
	switch data[i-1] {
	case '"': // string
		...
	case '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '-': // number
		...
	case 't': // true
		i += len("rue")
	case 'f': // false
		i += len("alse")
	case 'n': // null
		i += len("ull")
	}
	if i < len(data) {
		d.opcode = stateEndValue(&d.scan, data[i])
	} else {
		d.opcode = scanEnd
	}
	d.off = i + 1
}

Because JSON literals are actually only strings, numbers, booleans, and nulls, the implementation of this method is not particularly complicated, because after the method scans for the corresponding literal, it calls encoding/json.decodeState. SetInt, reflect.Value.SetFloat and reflect.Value.SetBool methods are called in the process.

Summary

JSON itself is a tree-like data structure that follows a top-down encoding and decoding process, whether serialized or deserialized, using a recursive approach to JSON objects. JSON as a standard library provides a very simple interface, and although its performance has been criticized by developers, it provides good generality as a framework. By analyzing the JSON library implementation, we can also learn from it various ways to use reflection.