In Go 1.19 development, string.SliceHeader and string.StringHeader went through a life-or-death struggle, and these two types were once marked as deprecated (deprecated), but these two types are often used in scenarios where slice of byte and string are efficiently interchanged, so if they are marked as deprecated, but there is no alternative, they are removed from the deprecation mark, if nothing else. They will also be marked as deprecated again in Go 1.20, if nothing else.

Optimization of byte slice and string conversion

Directly converting string(bytes) or []byte(str) will bring data duplication and poor performance, so in the pursuit of the ultimate performance scenario, we will use the “hack” way to achieve these two types of conversion, such as k8s using the following way.

1
2
3
4
5
6
7
8

// toBytes performs unholy acts to avoid allocations
func toBytes(s string) []byte {
    return *(*[]byte)(unsafe.Pointer(&s))
}
// toString performs unholy acts to avoid allocations
func toString(b []byte) string {
    return *(*string)(unsafe.Pointer(&b))
}

More often the following approach is used (rpcx also uses the following approach).

1
2
3
4
5
6
7
8

func SliceByteToString(b []byte) string {
    return *(*string)(unsafe.Pointer(&b))
}
func StringToSliceByte(s string) []byte {
    x := (*[2]uintptr)(unsafe.Pointer(&s))
    h := [3]uintptr{x[0], x[1], x[1]}
    return *(*[]byte)(unsafe.Pointer(&h))
}

Even the standard library uses this approach.

1
2
3
4
5
6
7
8

func Clone(s string) string {
    if len(s) == 0 {
        return ""
    }
    b := make([]byte, len(s))
    copy(b, s)
    return *(*string)(unsafe.Pointer(&b))
}

Since the slice of byte and string data structures are similar, we can use this ‘hack’ to force the conversion. These two types of data structures are defined in the reflect package.

1
2
3
4
5
6
7
8
9

type SliceHeader struct {
    Data uintptr
    Len  int
    Cap  int
}
type StringHeader struct {
    Data uintptr
    Len  int
}

Slice has one more Cap field than String, and their data is stored in an array, and the Data of both structures stores a pointer to this array.

The new way of Go 1.20

Many projects use the above approach for performance improvement, but this is achieved by unsafe, which is quite risky because after a strong turn, the slice may make some changes, resulting in the relevant data being overwritten or recycled, and there are often some unexpected problems. I made a similar mistake when using this approach for RedisProxy, and I thought it was a standard library error at the time.

Therefore, Go is going to deprecate these two types SliceHeader and StringHeader in 1.20 to avoid misuse.

In Go 1.12, several methods String, StringData, Slice and SliceData have been added as replacements to achieve this high performance conversion.

  • func Slice(ptr *ArbitraryType, len IntegerType) []ArbitraryType: Returns a Slice, whose underlying array starts at ptr, and whose length and capacity are both len
  • func SliceData(slice []ArbitraryType) *ArbitraryType: Returns a pointer to the underlying array
  • func String(ptr *byte, len IntegerType) string: Generate a string, the bottom array starts from ptr, length is len
  • func StringData(str string) *byte: Returns the array at the bottom of the string

These four methods look very low-level.

This commit was submitted by cuiweixie. Because it involves a very basic and low-level implementation, and because it is a method that is likely to be widely used, this commit was reviewed particularly carefully.

This change even alerted Rob Pike, who has been dormant for many months, to ask why there is only an implementation and not even a comment file: #54858. Of course, the reason is that this feature is still under development and review, but we can see that Rob Pike is very concerned about this change.

cuiweixie even modified the standard library to use these four methods from his commit unsafe.

Performance testing

Although cuiweixie’s commit has not been merged to the master branch yet, there are still some variables. But I found that using gotip works with these methods now. My understanding is that gotip is consistent with the master branch, isn’t it?

Anyway, let’s write a benchmark:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

var L = 1024 * 1024
var str = strings.Repeat("a", L)
var s = bytes.Repeat([]byte{'a'}, L)
var str2 string
var s2 []byte
func BenchmarkString2Slice(b *testing.B) {
    for i := 0; i < b.N; i++ {
        bt := []byte(str)
        if len(bt) != L {
            b.Fatal()
        }
    }
}
func BenchmarkString2SliceReflect(b *testing.B) {
    for i := 0; i < b.N; i++ {
        bt := *(*[]byte)(unsafe.Pointer(&str))
        if len(bt) != L {
            b.Fatal()
        }
    }
}
func BenchmarkString2SliceUnsafe(b *testing.B) {
    for i := 0; i < b.N; i++ {
        bt := unsafe.Slice(unsafe.StringData(str), len(str))
        if len(bt) != L {
            b.Fatal()
        }
    }
}
func BenchmarkSlice2String(b *testing.B) {
    for i := 0; i < b.N; i++ {
        ss := string(s)
        if len(ss) != L {
            b.Fatal()
        }
    }
}
func BenchmarkSlice2StringReflect(b *testing.B) {
    for i := 0; i < b.N; i++ {
        ss := *(*string)(unsafe.Pointer(&s))
        if len(ss) != L {
            b.Fatal()
        }
    }
}
func BenchmarkSlice2StringUnsafe(b *testing.B) {
    for i := 0; i < b.N; i++ {
        ss := unsafe.String(unsafe.SliceData(s), len(str))
        if len(ss) != L {
            b.Fatal()
        }
    }
}

The actual test results are as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

➜  strslice gotip test -benchmem  -bench .
goos: darwin
goarch: arm64
pkg: github.com/smallnest/study/strslice
BenchmarkString2Slice-8          	   18826	         63942 ns/op	 1048579 B/op	       1 allocs/op
BenchmarkString2SliceReflect-8   	1000000000	         0.6498 ns/op	       0 B/op	       0 allocs/op
BenchmarkString2SliceUnsafe-8    	1000000000	         0.8178 ns/op	       0 B/op	       0 allocs/op
BenchmarkSlice2String-8          	   18686	         65864 ns/op	 1048580 B/op	       1 allocs/op
BenchmarkSlice2StringReflect-8   	1000000000	         0.6488 ns/op	       0 B/op	       0 allocs/op
BenchmarkSlice2StringUnsafe-8    	1000000000	         0.9744 ns/op	       0 B/op	       0 allocs/op

As you can see, without the way of hacking, the time consumption of the two types of strong turn is very huge, if the way of reflect is used, the performance improvement is greatly improved.

If we use the latest unsafe package, the performance can also be greatly improved, although the time consumption than reflect slightly increased, can be ignored.

Reference