On March 15, 2022, Google released the much-anticipated Golang 1.18, which brings several major new features.
- a workspace to solve some of the problems associated with developing multiple repositories locally at the same time
- a Fuzzing Test that automatically detects code branches, generates random input, and checks to see if the code panics
- generic support that many developers have been waiting for.
This article will briefly describe these three features.
Go Workspace Mode
Realistic situation
Multi-repository development
In practice, we often modify multiple modules that have dependencies at the same time, for example, if we implement a requirement on a service module, we also need to make changes to a common module of the project team, the whole workflow will look like this.
As you can see, every time you modify the Common library, you need to push the code to the remote end, then modify the dependency of the local service repository, and then pull the Common code from the remote end via go mod tidy, which is a lot of trouble.
Some people may ask, In this case, can’t we just add a replace clause to go.mod in the service repository?
However, if you use replace in go.mod, you need to pay extra mental cost in maintenance, and if the go.mod with replace is pushed to the remote code base, others will be confused.
Multiple new repositories start development
Suppose at this point I am developing two new modules, as follows.
And MyService relies on Common.
During development, for various reasons, the code may not be pushed to the remote end immediately. So if I need to compile MyService locally, go build (or go mod tidy) will fail to automatically download the dependency because the Common library is not published to the codebase at all.
For the same reason as the “multiple repository simultaneous development” above, replace should not be added to MyService’s go.mod file either.
What is the Workspace Pattern
The Go workspace pattern first emerged from a proposal by Go developer Michael Matloob in April 2021 called “Multi-Module Workspaces in cmd/go”.
In this proposal, a new go.work file is created, and a series of local paths are specified in this file. The go modules under these local paths together form a workspace, and go commands can manipulate the go modules under these paths, and these go modules will be used first when compiling.
A workspace can be initialized and an empty go.work file generated with the following command.
|
|
The contents of the newly generated go.work file are as follows.
In the go.work file, the directory indicates the module directories in the workspace, and when compiling code, modules under the same workspace are used first.
In go.work, there is also support for using replace to specify the local code base, but in most cases it is better to add the path of the dependent local code base to the directory.
Recommended usage
Because go.work describes a local workspace, it cannot be committed to a remote repository. Although you can add this file to .gitignore, the most recommended approach is to use go.work in the upper level of the local repository.
For example, in the “multiple new repositories to start development” example above, let’s say that the local paths of my two repositories are as follows.
Then I can generate a go.work in the “/Users/bytedance/dev/my_new_project” directory with the following content.
You can also organize multiple directories into a workspace by placing go.work in the top-level directory, and since the top-level directory itself is not managed by git, you don’t have to worry about gitignore or anything like that, which is a relatively painless way to go.
Points to note when using
Currently (go 1.18) only go build makes judgments about go.work, and go mod tidy does not care about Go workspaces.
Go Fuzzing Test
Why Golang supports fuzzing tests
Since 1.18, Fuzzing Test has been added to Golang’s testing standard library as a part of language security, for the obvious reason that security is an essential and increasingly important consideration for programmers building software.
Golang has so far provided a number of features and tools for language security, such as enforcing explicit type conversions, disabling implicit type conversions, checking for out-of-bounds access to arrays and slices, hashing dependency packages with go.sum, and more.
As we enter the cloud-native era, Golang has become one of the head languages for cloud-native infrastructure and services. The security requirements of these systems are naturally self-evident. Especially for user input, it is one of the basic requirements for these systems not to be handled exceptionally, crash, or be manipulated by user input.
This requires our systems to be stable when handling any user input, but traditional quality assurance tools, such as Code Review, static analysis, manual testing, Unit Test, and so on. In the face of increasingly complex systems, it is naturally impossible to exhaust all possible combinations of inputs, especially some very obscure corner cases.
Fuzzy testing is one of the best practices in the industry to solve this problem, so it is not hard to understand why Golang chooses to support it.
What is fuzzy testing
Fuzzy testing is a way to test a program by automatically constructing some random data as input to the program through a data construction engine, supplemented by some initial data that the developer can provide. Fuzzy testing can help developers find hard-to-find errors in stability, logic, and even security, especially as the system under test becomes more complex.
Instead of relying on a data set defined by the development tester, fuzzy testing is usually implemented as a set of random data constructed by the data construction engine itself. The fuzzy tests provide this data as input to the program under test and monitor it for panic, assertion failure, infinite loops, or any other exceptions. The data generated by the data construction engine is called corpus, and fuzzy tests are also a means of continuous testing, because if there is no limit on the number of executions or the maximum time of execution, it will keep on executing.
Golang’s fuzzing tests are implemented in the compiler toolchain, so they use an entry generation technique called “coverage guided fuzzing”, which runs roughly as follows.
How Golang’s fuzzy tests are used
Golang’s fuzzy tests can simply be used directly when using them, or you can provide some initial corpus yourself.
The simplest practical example
The fuzzy test functions are also placed in xxx_test.go. To write the simplest fuzzy test example (with the obvious divide-by-0 error).
As you can see, similar to unit tests, the function names for fuzz tests are in FuzzXxx format and accept a testing.F
pointer object.
The fuzz test is then performed on the specified function using f.Fuzz
in the function. The first argument to the function being tested must be of type *testing.T
, and can be followed by any number of arguments of the basic type.
After writing, use the following command to start the fuzz test.
|
|
The fuzz test will continue by default, as long as the function being tested is not panic or error free. The “-fuzztime” option can be used to limit the duration of the fuzz test.
|
|
When you test the above code with fuzzy test, you will encounter a panic situation, and the fuzzy test will output the following message.
|
|
Of which.
|
|
This line indicates that the fuzzy test has saved the panic test input to this file, and tries to output the contents of this file at this time.
You can see the input that triggered the panic, and then we can check what is wrong with our code based on the input. Of course, this simple example is a deliberate write of a divide-by-0 error.
provides a custom corpus
Golang’s fuzz testing also allows developers to provide their own initial corpus, either through the “f.Add” method or by writing the corpus in the same format as the “Failing input” above, to You can also write the corpus in the same format as “Failing input” above, in “testdata/fuzz/FuzzXXX/custom corpus filename”.
Notes on use
Golang’s fuzzy tests currently only support these types of arguments for the function being tested.
According to the documentation of the standard library, more type support will be added later.
Go’s Generics
Golang finally added support for generics in 1.18. With generics, we can write some public library code like this.
Old code (reflection).
|
|
New code (generic).
Generics add three important new features to Golang: 1.
- support for using Type parameters when defining functions and types
- redefine interface as a “collection of types”.
- generic type support type derivation
The following is a brief explanation of each of these elements.
Type Parameters
The list of type parameters is very similar to the list of function parameters, except that it uses square brackets.
The above code defines a parameter type T for the Min function, which is very similar to template < typename T >
in C++, except that in Golang, you can specify the “constraints” that it needs to satisfy for this parameter type “. In this example, the “constraint” used is constraints.Ordered
The function can then be used as follows.
The process of specifying type parameters for a generic function is called “instantiation”, and the instantiated function can also be saved as a function object and used further.
Similarly, custom types support generic types.
As in the above code, the struct type supports its own member variables holding the same generic type as itself when using generics.
Type Sets
Let’s go a little deeper into the “constraints” mentioned in the above example. The “int”, “float64”, and “int64” in the above example are actually passed as “parameters” when instantiated. is passed to the “type parameter list”. That is, [T constraints.Ordered]
in the example above.
Just like passing a normal parameter requires verifying the type of the parameter, passing a type parameter requires verifying the type of the parameter being passed, to check if the type being passed meets the requirements.
For example, in the example above, when instantiating the Min function with the types “int”, “float64”, and “int64”, the compiler will check whether these parameters satisfy the constraint “constraints.Ordered”. Ordered” constraint, which describes the set of all types that can be compared using “<”, and is itself an interface.
In Go’s generics, a type constraint must be an interface, and the “traditional” Golang definition of an interface is “an interface that defines a collection of methods”, and any type that implements this collection of methods implements the interface. Any type that implements this set of methods implements this interface.
But here’s where the problem arises: the “<” comparison is clearly not a method (there are no C++ operator overloads in Go), and the constraints.Ordered describing the constraint is indeed an interface itself.
So starting with 1.18, Golang redefines an interface as “a collection of types”. As previously thought about interfaces, an interface can also be thought of as “a collection of all types that implement the set of methods of the interface”.
The two views are actually similar, but the latter is obviously more flexible, specifying a set of concrete types as an interface directly, even if those types don’t have any methods.
For example, in 1.18, an interface can be defined like this.
This definition means that int/bool/string can be used as MyInterface.
Ordered, which is actually defined as follows.
|
|
where the leading “~” symbol means “any underlying type is the type of the type that follows”, e.g.
|
|
The MyString defined in this way satisfies the “~string” type constraint.
Type Inference
Finally, type inference, which is common to all languages that support generics, is not absent. Type Inference allows users to call generic functions without specifying all type parameters. An example is the following function.
It can be used in the following way.
Generic functions can be used without specifying specific type arguments, or by specifying only some of the types to the left of the type argument list. The compiler will report an error when automatic type derivation fails.
Type derivation in Golang generics is divided into two main parts.
- function argument type derivation: the specific types corresponding to the type arguments are derived from the function’s input parameters.
- Constraint type derivation: Inferring the specific type of an unknown type parameter from the type parameter of a known specific type.
Both type derivations rely on a technique called “Type Unification”.
Type Unification
Type unification is the comparison of two types, which may themselves be a type parameter or may contain a type parameter.
The process of comparison is a comparison of the “structure” of the two types and requires that the two types being compared satisfy the following conditions:
- the “structure” of the two types must match after eliminating the type parameter
- the remaining specific types in the structure must be the same after the type parameters are removed
- if neither type parameter is included, then both types must be identical, or the underlying data types must be identical
By “structure”, we mean slice, map, function, etc. in the type definition, and any nesting between them.
When these conditions are met, the type uniformity comparison is successful and the compiler can further speculate on the type parameters, for example.
If we have two type parameters “T1” and “T2” at this time, then []map[int]bool
can match the following types.
As a counterexample, []map[int]bool
clearly does not match these types.
Function Argument Type Inference
Function Argument Type Inference, as the name implies, is a generalized function that is called without all type arguments being fully specified, and then the compiler infers the specific types of the type arguments based on the types of the actual function inputs, such as the Min function at the beginning of this article.
Like other languages that support generics, Golang’s function argument type derivation only supports “type arguments that can be derived from the input”, and if the type argument is used to mark the return type, then the type argument must be explicitly specified when it is used.
Functions like this, where part of the type parameter appears only in the return value (or only in the function body, not as an input or output parameter), cannot use function parameter type derivation, but must explicitly specify the type manually.
derived algorithm with examples
Using the Min function as an example, we can explain the process of deriving function parameter types.
Let’s look at the first scenario first.
|
|
At this point, both input parameters are untyped literal constants, so the first round of type unification is skipped, and the specific type of the input parameters is not determined. At this point, the compiler tries to use the default type int for both parameters, and since the type of both input parameters at the function definition is “T”, and both use the default type int, T is successfully inferred to be int.
Then look at the second case.
|
|
At this point, the second argument has an explicit type int64, so in the first round of type unification, T is inferred to be int64, and when trying to determine the type for the first argument “1” that was missed in the first round, T is successfully inferred to be int64 because “1” is a legal int64, T is successfully inferred as int64.
Let’s look at the third case.
|
|
At this point, the second argument has a clear type int64, so in the first round of type unification, T is inferred to be int64, and when trying to determine the type for the first argument “1.5”, which was missed in the first round, the type inference fails because “1.5” is not is not a legal int64 type value, the type derivation fails and the compiler reports an error.
Finally, look at the fourth case.
|
|
Similar to the first case, the first round of type unification is skipped and the specific types of the two input parameters are not determined, so the compiler starts trying to use the default types. The default types of the two parameters are int and float64, and since the same type parameter T can only be determined as one type in type derivation, type derivation fails at this point as well.
Constraints Type Inference
Constraint type inference is another powerful weapon of Golang generics, allowing the compiler to derive the specific type of a type parameter from another type parameter, and to preserve the caller’s type information by using type parameters.
Constraint type derivation can allow constraints to be specified for a type parameter using other type parameters. Such constraints are called “structured constraints”. Such constraints define the data structure that a type parameter must satisfy, e.g.
In the definition of this function, “~[]E” is a shorthand for the structured constraint on S. It is written in its entirety as “interface{~[]E}”, i.e., an interface defined as a collection of types, and which contains only one definition of “~[]E”, meaning “all types whose underlying data type is []E”.
Note that the set of types corresponding to a legal structured constraint should satisfy any of the following conditions: 1.
- the set of types contains only one type
- the underlying data types of all types in the type set are identical
In this example, the structured constraint used by S is a legal structured constraint because the underlying data types of all types that satisfy the constraint are []E.
The compiler attempts constraint type derivation when there are type parameters that cannot be determined by function parameter type derivation for specific types and the list of type parameters contains structured constraints.
Deduction algorithm with examples
Simple example
In conjunction with our example of the “DoubleSlice” function, let’s talk about the specific process of constraint type derivation.
In this call, the first thing performed is the normal function argument type derivation, and this step will result in a derivation like this
|
|
At this point the compiler finds that there is a type parameter E that has not been derived and that there is currently a type parameter S that uses structural constraints, so it starts constraining the type derivation.
The first thing to look for is a type parameter whose type derivation has been completed, in this case S, whose type has been derived as MySlice.
Then the actual type of S, “MySlice”, is unified with the structured constraint of S, “~[]E”, and since the underlying type of MySlice is []int, the structured match gives this match Result.
|
|
At this point all type parameters have been inferred and conform to their respective constraints, and type derivation is over.
A more complex example
Suppose there is a function such as
Then we call it like this.
|
|
The type derivation process generated at compile time is as follows, starting with the result of the function parameter type derivation.
|
|
Then using constrained type derivation for S, comparing []map[string]int
and ~[]M
, we get
|
|
Continuing with the constrained type derivation for M, comparing map[string]int
with ~map[K]V
, we get
The type derivation is now successfully completed.
Preserving Type Information with Constraint Type Inference
Another useful aspect of constraint type derivation is that it preserves the type information of the caller’s original arguments.
Using the “DoubleSlice” function from this section as an example, suppose we now implement a more “simple” version of it.
This version has only one type parameter, E. At this point, we call it as we did before.
The type derivation at this point is just the most basic type derivation of the function parameters, where the compiler does a direct structured comparison between MySlice and []E and concludes that the actual type of E is int.
The DoubleSliceSimple function returns []E, which is []int, not MySlice as passed in by the caller, whereas the previous DoubleSlice function, by defining a type parameter S with a structured constraint and using S to match the type of the input directly, and returning a value of type S, preserves The original argument type of the caller is preserved.
Limitations of using generics
There are still a number of limitations to Golang generics, some of the major ones include.
- member functions cannot use generics
- you cannot use a method that is not specified in the constraint definition, even if all the types in the type set implement the method
- you cannot use a member variable even if all types in the type set have the member
The following are examples of each.
member functions cannot use generics
In this example, the member function Method of MyStruct[T]
defines a function parameter T2 that belongs only to itself, however such an operation is currently not supported by the compiler (and most likely will not be supported in the future).
cannot use methods other than those defined by constraints
In this example, the two members of the MyConstraint collection, MyType1 and MyType2, cannot be called directly from a generic function, even though they both implement the .Method() function.
If they need to be called, MyConstraint should be rewritten in the following form.
Unable to use member variables
In this example, although MyType1 and MyType2 both contain a Name member and are both of type string, they cannot be used directly in a generic function in any way.
This is because the type constraint itself is an interface, and the definition of an interface can only contain a collection of types, and a list of member functions.
Summary
Golang 1.18 brings the above three very important new features, among them.
- workspace mode allows for a smoother workflow for local development.
- fuzzy tests can find some corner cases and improve the robustness of the code.
- generalization can make the code of some public libraries more elegant, avoiding the old way of having to use reflection for “generality”, which is not only difficult to write and read, but also increases the runtime overhead, because reflection is dynamic information at runtime, while generalization is static information at compile time.
This article also briefly covers these aspects and hopefully gives you a basic understanding of these new things in Golang.