There is an amazing thing about Go in the file IO scenario. When opening a file, instead of an interface, it returns a pointer to an os.File structure.

1
2
3

func Open(name string) (*File, error) {
    return OpenFile(name, O_RDONLY, 0)
}

This means that the concept of Go’s filesystem is directly related to the concept of the OS’s filesystem. You have to pass in a file path, and you have to actually go and open an OS file.

If you don’t use an interface, but instead strongly relate it to a specific type, it will lead to poor scalability. For example, if all the OS packages are used, then the operations are strongly bound to the OS file system.

Go’s designers have always been concerned about this, but there is nothing they can do about it. Because users are already using it, Go’s promise is to be forward-compatible, so directly modifying the original semantics and interface will not work.

What to do?

Go 1.16 gives us the answer. Go gives us io.FS. Go’s intention is to make another layer of FS abstraction at the level of its own language, so that it can be decoupled from the OS’s FS. io.FS can be any odd-shaped FS, as long as you implement the specified FS interface. The next step is to look at some of the core changes brought by Go 1.16.

Some people say that Go is already 1.19, why look at 1.16?

Because Go’s io/fs was introduced in Go 1.16. There is one major change in io.

What has changed about io in Go 1.16?

  • A new package for io/fs was added, abstracting an FS out.
  • The embed package uses this abstraction.
  • Regulates the contents of io/ioutil.

Let’s look at them one by one.

io.FS

Why does Go abstract FS?

As mentioned earlier, the concept of Go’s filesystem is directly related to the concept of OS’s filesystem. This creates an inconvenience for scalability. Most importantly, Go has identified a need for a different file system than OS, namely embed FS.

embed is a feature provided by Go to package files into a binary, which is also a file system-like requirement. But it is not a filesystem directly on the OS (the vfs thing).

So in Go 1.16 it came along. FS definition was introduced, and embed uses this layer of abstraction directly.

FS Interface

Go’s implementers are strong and recommend small interfaces. That is, minimalist, atomic interface semantics. You can see the strong power in the definition of io/fs.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

// File system interfaces
type FS interface {
    Open(name string) (File, error)
}

// File Interface
type File interface {
    Stat() (FileInfo, error)
    Read([]byte) (int, error)
    Close() error
}

This, in its simplest form, is FS. This is what the file system looks like in its simplest form, with just an Open method that returns a file.

In other words, Go understands the file system, as long as it can implement an Open method and return a File interface, the File only needs to implement the Stat, Read, Close methods.

Did you find that the OS FS already meets the conditions? So, Go’s FS can be OS’s FS, and naturally, it can be any other implementation.

Go extends the io.FS interface to add filesystem functionality. For example, adding ReadDir is a file system ReadDirFS with read directories.

1
2
3
4
5
6

type ReadDirFS interface {
    FS

    // ReadDir
    ReadDir(name string) ([]DirEntry, error)
}

Add a Glob method and you have a file system with path wildcard queries.

1
2
3
4

type GlobFS interface {
    FS
    Glob(pattern string) ([]string, error)
}

Add a Stat and it becomes a path-searching filesystem.

1
2
3
4
5


type StatFS interface {
    FS
    Stat(name string) (FileInfo, error)
}

The definition of these very classic file systems is already done by Go in io/fs.

How to use io.FS

Our goal is to implement an FS for Go, which is already defined in io.FS. We just need to write a structure that implements its methods, and then you can say it’s an FS.

There is a lot of room for imagination here, for example, it can be an OS FS, a memory FS, a hash FS, and so on. There are many examples on the web. But the standard library already has one of the best examples, and that is embed FS.

Let’s look at how embed implements an embedded filesystem. embed is implemented in the embed/embed.go file, which is very concise.

First, a structure FS is defined in the embed package, which will be the concrete implementation of io.FS.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

type FS struct {
    files *[]file
}

// Represents an embedded file
type file struct {
    name string
    data string  // The data of the file is all in memory
    hash [16]byte // truncated SHA256 hash
}

The FS structure inside embed only needs to implement the Open method.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

// Open
func (f FS) Open(name string) (fs.File, error) {
    // Find the file object by name match
    file := f.lookup(name)
    if file == nil {
        return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrNotExist}
    }
    // If it is a directory
    if file.IsDir() {
        return &openDir{file, f.readDir(name), 0}, nil
    }
    // Once found, it is wrapped in an openFile structure
    return &openFile{file, 0}, nil
}

Open above, in the case of a file, returns an openFile structure as a concrete implementation of the io.File interface.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

// A file implementation
type openFile struct {
    f *file // the file itself
    offset int64 // current read offset
}
func (f *openFile) Close() error               { return nil }
func (f *openFile) Stat() (fs.FileInfo, error) { return f.f, nil }
func (f *openFile) Read(b []byte) (int, error) {
    // Determine if the offset is as expected
    if f.offset >= int64(len(f.f.data)) {
        return 0, io.EOF
    }
    if f.offset < 0 {
        return 0, &fs.PathError{Op: "read", Path: f.f.name, Err: fs.ErrInvalid}
    }
    // Copy data from memory
    n := copy(b, f.f.data[f.offset:])
    f.offset += int64(n)
    return n, nil
}

As above, you only need to implement the Read, Stat, Close methods. This is a complete, Go-level implementation of FS.

You can use the embed file system as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

//go:embed hello.txt
var f embed.FS

func main() {
    // open
    file, err := f.Open("hello.txt")
    // ...
    // read
    n, err = file.Read(/*buffer*/)
}

In the above example, a hello.txt file in the current directory is packed into a binary file when it is compiled. You can read it out when the program starts.

Note: The variable f, the compiler will arrange to fill it. It has a value when the process is started.

Go 1.16 Other changes to IO

In addition to the io/fs and embed fs mentioned above, Go has made some more precise adjustments to the structure of the previous io to classify it. The stuff in the previous hodgepodge of io/ioutil has been split out. It has been moved to the corresponding io and os packages. For compatibility, the ioutil package was not removed directly, but imported. For example.

  • Discard is moved to the io library implementation
  • ReadAll has been moved to the io library implementation.
  • NopCloser has been moved to the io library implementation.
  • ReadFile moved to os library implementation
  • WriteFile moved to os library implementation

Basically ioutil package is hollowed out. go 1.16 is not deleted yet just for compatibility.

Summary

  1. Go’s io.FS interface is intended to be decoupled from the OS’s FS. This gives the programmer more room for imagination.
  2. embed FS has a typical FS interface, but it is not located directly on the OS file system. So it is ideal for the first practice with io.FS.
  3. FS to manage the files, so that it can be decoupled from the OS and easy to do unit testing.
  4. ioutil can be used sparingly, as its functionality has been moved to a more explicit package implementation.