RPC-based plug-in mechanism in Golang

Recently there is a need to implement the plug-in mechanism inside the project, the so-called plug-in means that the program can be extended without releasing a new version of the case, this plug-in mechanism is particularly widely used, common such as our browser extensions, Nginx extensions, PHP extensions and so on.

Inside the Go language, it comes with an official plugin extension mechanism, which I have described in a previous article.

But the official Go comes with this plug-in mechanism is very immature, more like a half-finished product, there are many problems, such as plug-ins can not be uninstalled, the respective version of the dependency must be completely consistent, which must be consistent with the version of the dependency requirements are very deadly, because in a large project, it is difficult to limit the plug-in developers to use the dependency library version, it is difficult to achieve uniform.

Today we will introduce a plug-in mechanism based on RPC implementation, and this approach is proven by large-scale practice, the availability is very high, it is worth trying.

1. Introduction

This is hashicorp open source project, Github: https://github.com/hashicorp/go-plugin

The official description is as follows.

go-plugin is a Go (golang) plugin system based on RPC. It is a plug-in system used by HashiCorp tools for over 4 years. Although originally created for Packer, it is also used by Terraform, Nomad, Vault, and Boundary. While the plug-in system is based on RPC, it is currently designed to work only on local [reliable] networks. Plugins on real networks are not supported and can cause unexpected behavior. The plug-in system has been used on millions of machines in many different projects and has proven to be durable enough for production use.

The introduction mentioned in a number of projects used, but I am not familiar with these, but there is an open source project many people should have heard of, the name is Grafana

Grafana is an open source monitoring visualization platform , which happens to be hashicorp project , currently also provides commercial services , its back end is written in Go , is also considered a very popular open source project in the Go language . Which uses the above-mentioned plug-in library, which Grafana panel inside the data source is the form of plug-ins, users can download and install the corresponding database plug-ins according to their needs.

However, in the Grafana project, this plug-in library is encapsulated in many layers, the code is very much, not so simple, let’s not see, first of all, first understand the functions provided by this plug-in library, according to the official documentation of this plug-in library has the following features.

1. plugins are implementations of Go interfaces: this makes writing and using plugins very natural. For the author of the plugin, he only needs to implement a Go interface; for the user of the plugin, he only needs to call a Go interface. go-plugin takes care of all the details of converting local calls to gRPC calls
2. cross-language support: plug-ins can be written based on any major language, the same can be consumed by any major language
3. support complex parameters, return values: go-plugin can handle interfaces, io.Reader/Writer and other complex types
4. two-way communication: in order to support complex parameters, the host process can send the interface implementation to the plug-in, and the plug-in can also call back to the host process
5. built-in logging system: any plug-in using the log standard library, will pass log information back to the host process. The host process will add the path of the plug-in binary file in front of these logs, and print the logs
6. protocol versioning: support a simple protocol versioning, adding the version number can be based on the old version of the protocol plug-in invalidation. Versions should be added when the interface signature changes
7. standard output / error synchronization: plug-ins run as child processes, these child processes are free to use the standard output / error, and the contents of the print will be automatically synchronized to the host process, the host process can be specified for the synchronization of the log io.Writer.
8. TTY Preservation: Plug-in sub-processes can be linked to the stdin file descriptor of the host process in order to require TTY software to work properly
9. host process upgrade: when the host process is upgraded, the plug-in sub-processes can continue to be allowed and automatically associated with the new host process after the upgrade
10. encrypted communication: gRPC channels can be encrypted
11. integrity checks: support for the plug-in binary file Checksum
12. the plug-in crashed, will not cause the host process to crash
13. easy to install: you only need to put the plug-in into a directory that the host process can access

These features look very rich and powerful, however, according to my understanding, the actual application still needs to do some packaging, from the official demos, does not reflect these features.

2. Example

There are several examples inside the official repository, let’s look at the simplest demo first

1. Plugin definition

First of all, let’s look at the greeter_interface.go file. The first part we can understand for the plug-in to define the interface to be implemented, constrain the behavior of the plug-in.

1
2
3
4
5

// Greeter is the interface that we're exposing as a plugin.
type Greeter interface {
    Greet() string
}

Then define a Greeter that implements this plug-in interface, here it goes through RPC, so an rpc client is needed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

// Here is an implementation that talks over RPC
type GreeterRPC struct{ client *rpc.Client }

func (g *GreeterRPC) Greet() string {
    var resp string
    err := g.client.Call("Plugin.Greet", new(interface{}), &resp)
    if err != nil {
        // You usually want your interfaces to return errors. If they don't,
        // there isn't much other choice here.
        panic(err)
    }
    return resp
}

Immediately afterwards, an RPCServer was defined to wrap it up again.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

// Here is the RPC server that GreeterRPC talks to, conforming to
// the requirements of net/rpc
type GreeterRPCServer struct {
    // This is the real implementation
    Impl Greeter
}

func (s *GreeterRPCServer) Greet(args interface{}, resp *string) error {
    *resp = s.Impl.Greet()
    return nil
}

Finally, the last is the implementation of the plug-in, mainly the 2 methods Server and Client.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

type GreeterPlugin struct {
    // Impl Injection
    Impl Greeter
}

func (p *GreeterPlugin) Server(*plugin.MuxBroker) (interface{}, error) {
    return &GreeterRPCServer{Impl: p.Impl}, nil
}

func (GreeterPlugin) Client(b *plugin.MuxBroker, c *rpc.Client) (interface{}, error) {
    return &GreeterRPC{client: c}, nil
}

Definition of the RPC Plugin interface.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

// Plugin is the interface that is implemented to serve/connect to an
// inteface implementation.
type Plugin interface {
    // Server should return the RPC server compatible struct to serve
    // the methods that the Client calls over net/rpc.
    Server(*MuxBroker) (interface{}, error)

    // Client returns an interface implementation for the plugin you're
    // serving that communicates to the server end of the plugin.
    Client(*MuxBroker, *rpc.Client) (interface{}, error)
}

Ultimately, under layers of packaging, this file defines the framework of a plugin and the methods to be implemented, but an implementation is still missing, which is inside the greeter_impl.go file.

2. Plug-in implementation

First define an object to implement Greet method, this is relatively simple, here use a library inside the logger library.

1
2
3
4
5
6
7
8
9

// Here is a real implementation of Greeter
type GreeterHello struct {
    logger hclog.Logger
}

func (g *GreeterHello) Greet() string {
    g.logger.Debug("message from GreeterHello.Greet")
    return "Hello!"
}

Then there is the content inside main, the operation of this piece is simply to set some parameters, start an RPC service, and wait for the arrival of the request.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

func main() {
    logger := hclog.New(&hclog.LoggerOptions{
        Level:      hclog.Trace,
        Output:     os.Stderr,
        JSONFormat: true,
    })

    greeter := &GreeterHello{
        logger: logger,
    }
    // pluginMap is the map of plugins we can dispense.
    var pluginMap = map[string]plugin.Plugin{
        "greeter": &example.GreeterPlugin{Impl: greeter},
    }

    logger.Debug("message from plugin", "foo", "bar")

    plugin.Serve(&plugin.ServeConfig{
        HandshakeConfig: plugin.HandshakeConfig{
            ProtocolVersion:  1,
            MagicCookieKey:   "BASIC_PLUGIN",
            MagicCookieValue: "hello",
        },
        Plugins: pluginMap,
    })
}

Finally, don’t forget to compile the plug-in. The compilation of the plug-in is actually no different from a normal Go program and will result in a binary file, which will be used later.

1

go build -o ./plugin/greeter ./plugin/greeter_impl.go

3. Using the plug-in

The code for using plug-ins is relatively simple as well, you just need to New a Client, set the relevant parameters, and then initiate the call.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

func main() {
    // We're a host! Start by launching the plugin process.
    client := plugin.NewClient(&plugin.ClientConfig{
        HandshakeConfig: plugin.HandshakeConfig{
            ProtocolVersion:  1,
            MagicCookieKey:   "BASIC_PLUGIN",
            MagicCookieValue: "hello",
        },
        Plugins: map[string]plugin.Plugin{
            "greeter": &example.GreeterPlugin{},
        },
        Cmd: exec.Command("./plugin/greeter"),
        Logger: hclog.New(&hclog.LoggerOptions{
            Name:   "plugin",
            Output: os.Stdout,
            Level:  hclog.Debug,
        }),
    })
    defer client.Kill()

    // Connect via RPC
    rpcClient, err := client.Client()
    if err != nil {
        log.Fatal(err)
    }

    // Request the plugin
    raw, err := rpcClient.Dispense("greeter")
    if err != nil {
        log.Fatal(err)
    }

    // We should have a Greeter now! This feels like a normal interface
    // implementation but is in fact over an RPC connection.
    greeter := raw.(example.Greeter)
    fmt.Println(greeter.Greet())
}

What needs to be noted here is a handshakeConfig inside the configuration to be consistent with the plug-in implementation inside, in addition to setting the location of the binary executable.

Second, Plugins is a map which stores the mapping relationship between the plugin name and the plugin definition.

3. My doubts

From the above demo, this library’s plugin system is essentially a local RPC call, although the performance may be a little lower, after all, the local network also has overhead, but it does not have some of the problems of the official plugin mechanism.

But I did not see from this demo how to achieve multiple plug-in coexistence, as well as plug-in updates and other functions, in fact, in my subsequent research I found that the library does not achieve these functions.

If you want to implement these features you may have to implement them yourself, Grafana project in the use of this library has done a lot of packaging.

4. Principle

First, let’s look at the plug-in implementation piece, mainly the plugin.Serve method, which requires passing in a plug-in configuration.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81

func Serve(opts *ServeConfig) {
    // 一些配置
    ...

    // Register a listener so we can accept a connection
    listener, err := serverListener()
    if err != nil {
        logger.Error("plugin init error", "error", err)
        return
    }

    // Close the listener on return. We wrap this in a func() on purpose
    // because the "listener" reference may change to TLS.
    defer func() {
        listener.Close()
    }()
    
    // TLS的配置
    ...

    // Create the channel to tell us when we're done
    doneCh := make(chan struct{})
    
    // Build the server type
    var server ServerProtocol
    switch protoType {
    case ProtocolNetRPC:
        // If we have a TLS configuration then we wrap the listener
        // ourselves and do it at that level.
        if tlsConfig != nil {
            listener = tls.NewListener(listener, tlsConfig)
        }

        // Create the RPC server to dispense
        server = &RPCServer{
            Plugins: pluginSet,
            Stdout:  stdout_r,
            Stderr:  stderr_r,
            DoneCh:  doneCh,
        }

    case ProtocolGRPC:
        // Create the gRPC server
        server = &GRPCServer{
            Plugins: pluginSet,
            Server:  opts.GRPCServer,
            TLS:     tlsConfig,
            Stdout:  stdout_r,
            Stderr:  stderr_r,
            DoneCh:  doneCh,
            logger:  logger,
        }
    default:
        panic("unknown server protocol: " + protoType)
    }

    // Initialize the servers
    if err := server.Init(); err != nil {
        logger.Error("protocol init", "error", err)
        return
    }
    ...

    // Accept connections and wait for completion
    go server.Serve(listener)

    ctx := context.Background()
    if opts.Test != nil && opts.Test.Context != nil {
        ctx = opts.Test.Context
    }
    select {
    case <-ctx.Done():
        listener.Close()
        if s, ok := server.(*GRPCServer); ok {
            s.Stop()
        }
        // Wait for the server itself to shut down
        <-doneCh
    case <-doneCh:
    }
}

There is a lot of code, only the core code is shown here, in fact, is doing one thing is to initialize and start the RPC service, ready to accept requests.

More code in the plug-in use of this block, first we New a Client, the Client is to maintain the plug-in, and is a plug-in a Client, so if you want to achieve multiple plug-in coexistence, you can go to the implementation of a plug-in and Client mapping relationship can be.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

type Client struct {
    config            *ClientConfig
    exited            bool
    l                 sync.Mutex
    address           net.Addr
    process           *os.Process
    client            ClientProtocol
    protocol          Protocol
    logger            hclog.Logger
    doneCtx           context.Context
    ctxCancel         context.CancelFunc
    negotiatedVersion int

    // clientWaitGroup is used to manage the lifecycle of the plugin management
    // goroutines.
    clientWaitGroup sync.WaitGroup

    // stderrWaitGroup is used to prevent the command's Wait() function from
    // being called before we've finished reading from the stderr pipe.
    stderrWaitGroup sync.WaitGroup

    // processKilled is used for testing only, to flag when the process was
    // forcefully killed.
    processKilled bool
}

In the main inside when we finished NewClient, in turn called the Client and Dispense2 methods, the 2 methods are very important.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

// Client returns the protocol client for this connection.
// Subsequent calls to this will return the same client.
func (c *Client) Client() (ClientProtocol, error) {
    _, err := c.Start()
    if err != nil {
        return nil, err
    }

    c.l.Lock()
    defer c.l.Unlock()

    if c.client != nil {
        return c.client, nil
    }

    switch c.protocol {
    case ProtocolNetRPC:
        c.client, err = newRPCClient(c)

    case ProtocolGRPC:
        c.client, err = newGRPCClient(c.doneCtx, c)

    default:
        return nil, fmt.Errorf("unknown server protocol: %s", c.protocol)
    }

    if err != nil {
        c.client = nil
        return nil, err
    }

    return c.client, nil
}

Start this method does a lot of things, simply put, is based on the configuration of the cmd inside, that is, we compile the plug-in to get the binary executable file, start the plug-in rpc service.

Then, depending on the protocol, start the RPC service or GRPC service, get a real available Client, which is equivalent to the channel has been opened, the next is to launch the request.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

func (c *RPCClient) Dispense(name string) (interface{}, error) {
    p, ok := c.plugins[name]
    if !ok {
        return nil, fmt.Errorf("unknown plugin type: %s", name)
    }

    var id uint32
    if err := c.control.Call(
        "Dispenser.Dispense", name, &id); err != nil {
        return nil, err
    }

    conn, err := c.broker.Dial(id)
    if err != nil {
        return nil, err
    }

    return p.Client(c.broker, rpc.NewClient(conn))
}

Dispense method is based on the name of the plug-in to get the corresponding plug-in object, and then wrapped a layer to get a Client object, remember the initial definition of the plug-in when the Client?

1
2
3

func (GreeterPlugin) Client(b *plugin.MuxBroker, c *rpc.Client) (interface{}, error) {
    return &GreeterRPC{client: c}, nil
}

Finally, assert and call the plugin’s method, which is when the RPC request is actually initiated and the return result is obtained.

1
2

greeter := raw.(example.Greeter)
greeter.Greet()

Finally, assert and call the methods of the plugin, this is when the RPC request is really launched and get the return result One thing I feel particularly strange, from the point of view of this plugin implementation, Dispense this step is more like subdividing the plugins inside the plugins, because in my understanding, a binary file is a plugin, a plugin has only one implementation.

But obviously, this library does not think so, it believes that a plug-in file can be implemented inside multiple plug-ins, so it adds a Plugins to store the mapping relationship of plug-ins, which means you can implement multiple interfaces inside a plug-in.

This implementation is also equivalent to a constraint, and in fact inside the Grafana project, it declares the types of plugins that can be supported in this way. As a plugin, you can implement some of the interfaces according to your needs.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

// getPluginSet returns list of plugins supported
func getPluginSet() goplugin.PluginSet {
    return goplugin.PluginSet{
        "diagnostics": &grpcplugin.DiagnosticsGRPCPlugin{},
        "resource":    &grpcplugin.ResourceGRPCPlugin{},
        "data":        &grpcplugin.DataGRPCPlugin{},
        "stream":      &grpcplugin.StreamGRPCPlugin{},
        "renderer":    &pluginextensionv2.RendererGRPCPlugin{},
    }
}

5. Summary

After writing so much, for this library, I personally feel that the usability is very high, after the actual test of production. But only the core functionality, lack of encapsulation, for the use of plug-in system, at least the following functions should be achieved.

• Management of multiple plug-ins.
• automatic and manual update of plug-ins.
• plug-in health check, because the plug-in is essentially an rpc service, may hang, hang after what to do?

If you really want to use this library , it really needs to spend a lot of effort , there is a good place is that we can refer to the Hashicorp family of other open source projects to improve the code , in fact, I also wonder if the library can be slightly encapsulated to provide a simple and easy to use interface.