In a recent project I had to implement a way to separate data mapping functionality from the basic persistence logic.
Whereas the storage should be identical, the mapping logic differs depending on the input data. The mapping logic should be expandable by other developers without access to the sources of the application or the need to recompile the whole application again.
So I invested some time in looking into the different options within the Go ecosystem: How can I implement a fitting plugin system.
I started with the obvious choices: the Hashicorp Plugin Model as well as the built-in plugins package.
I finally chose Javascript as a plugin DSL. The following post will demonstrate some of the implementation I did. All sources can be found in the corresponding GitHub repository.

Hashicorp Plugins

If you search for “Golang Plugin”, you will most certainly find the Hashicorp go-plugin project (hashicorp/go-plugin - Golang plugin system over RPC.).

Although this function is widely used in most of the hashicorp products to implement extesibility, it comes with some overhead. To use this library, you basically have to have several binaries: One for the main “hosting” application and one for each plugin you want to run. The data exchange between the host and the plugin happens via RPC.

I created a small example with this library.

The contract between the host and the plugin is a specific interface. In this hello-world example it looks like this:

type Hello interface {
    Greet() string
}

Calling the plugin looks identical to calling an external binary, because it is just that:

client := plugin.NewClient(&plugin.ClientConfig{
    HandshakeConfig: handshakeConfig,
    Plugins:         pluginMap,
    Cmd:             exec.Command("./plugin/hello"),
})
defer client.Kill()

The object handshakeConfig only contains a struct that ensures that the current plugin is used in the correct version:

var handshakeConfig = plugin.HandshakeConfig{
    ProtocolVersion:  1,
    MagicCookieKey:   "BASIC_PLUGIN",
    MagicCookieValue: "hello",
}

Although it looks like a Go-only thing, the README states that other languages are supported, too:

Cross-language support. Plugins can be written (and consumed) by almost every major language. This library supports serving plugins via gRPC. gRPC-based plugins enable plugins to be written in any language.

As you can see, there is quite a tight coupling between the host and the plugin.
In addition, the go-plugin module also expects some specific skills from plugin developers (like the Golang Interface and RPC).
So the question is if it is possible to use something else to have plugins with less tight coupling, less overhead, and less complexity.

Go plugins Module

Of course, there is also the plugins module, which is an integral part of Go.

In theory, you don’t need to update the “hosting” binary, becaus there is no direct connection between the application and a plugin.

The plugin just exports itself as a type. Every plugin function is then attached to that type.

package main

import "fmt"

type greeting string

func (g greeting) Greet() {
    fmt.Println("Hello World!")
}

// Greeter - exported
var Greeter greeting

In the main function, you only need an interface of the plugin’s methods and resolve the specific plugin via the complete path to the compiled library file.

type Greeter interface {
    Greet()
}

plug, err := plugin.Open("plugin/implementation.so")
if err != nil {
    log.Println(err)
}

You then resolve the symbol, cast it to the interface type, and run the Greet() method afterwards.

symGreeter, err := plug.Lookup("Greeter")
if err != nil {
    log.Println(err)
}

var greeter Greeter
greeter, ok := symGreeter.(Greeter)
if !ok {
    log.Println("unexpected type from module symbol")
}

greeter.Greet()

This example is based on a blog post from Domenico Luciani. This method has less overhead than the one from Hashicorp (e.g. no RPCs), but also comes with some constraints:

Using a Different Approach

Since JavaScript is well known to most developers, I think that plugins written in JavaScript are a good fit. After a quick search, I found the project called Otto VM, that essentially is a JavaScript interpreter written in Go. I created some small examples to demonstrate how this module is used in different use cases.

A simple hello world with Otto looks like this:

package main

import "github.com/robertkrimen/otto"

func main() {

    vm := otto.New()
    vm.Run(`
        console.log("Hello World!");
    `)
}

As you can see, the runtime is able to interpret and run plain JavaScript (as of now, it is limited to ECMAScript 5).
The interesting part of Otto is, that you can extend the JS API with external Go functions and also exchange data in both directions.

For that purpose, Otto has a simple type mapping between JS and Go (and vice versa):

Export will attempt to convert the value to a Go representation and return it via an interface{} kind. Export returns an error, but it will always be nil. It is present for backwards compatibility. If a reasonable conversion is not possible, then the original value is returned.

undefined   -> nil
null        -> nil
boolean     -> bool
number      -> A number type (int, float32, uint64, ...)
string      -> string
Array       -> []interface{}
Object      -> map[string]interface{}

Adding functions

Let’s say we don’t want to use the console.log(…) statement, but want to have messages from JS logged via the default Go logging function. For that, we create a small wrapper function and map it to a JavaScript statement:

package main

import (
    "log"
    "github.com/robertkrimen/otto"
)

func main() {
    vm := otto.New()

    err := vm.Set("log", logJS)
    if err != nil {
        panic(err)
    }

    vm.Run(`
        console.log("logging with JS!");
        log("logging with Golang!");
    `)
}

func logJS(content string) {
    log.Println(content)
}

Data Exchange

Injecting data from our Go program into JavaScript or receiving a result from a JavaScript function is also quite simple:

package main

import (
    "log"
    "github.com/robertkrimen/otto"
)

func main() {
    vm := otto.New()

    // jsData contains the result of `date`
    jsDate, err := vm.Run(`
    (function(){
        date = new Date();
        return date;
    })();
    `)

    if err != nil {
        panic(err)
    }
    log.Printf("jsDate: %s", jsDate)

    dataMap := make(map[string]interface{})
    dataMap["foo"] = "bar"
    dataMap["one"] = "1"
    dataMap["two"] = "2"

    err = vm.Set("dataMap", dataMap)
    if err != nil {
        panic(err)
    }

    value, err := vm.Run(`
    (function(){
        var keys = [];
        for(k in dataMap) {
            console.log(k + ": " + dataMap[k]);
            keys.push(k);
        }
        return keys;
    })();
    `)

    keys, err := value.Export()
    if err != nil {
        panic(err)
    }

    keyArray := keys.([]string)
    log.Printf("keys: %s", keyArray)
}

In typical Go manner, the call vm.Run(…) has two return types, a value and an error object. If we are interested in a string representation, we can use the plain value object.
Otherwise, we first have to invoke the …Export() method to get the correct type mapping. This is shown in the second example.
Here we extract the keys of a Go map into our JavaScript code and extract the resulting array afterwards.

That’s it for the basics. But for a real plugin environment, we need some additional parts.

Real Plugin: API Clients

First of all, plugin code and application code should be separate. For that reason, we define our own basic plugin structure.

A plugin is a folder with one info.json file (for the plugin metadata) and one or more JavaScript files:

├── plugin
    ├── client.js
    └── info.json

In our next example, we want to implement an application for different APIs. A plugin can either call the Twitter API, another plugin calls a (random) HTTP Service.
Since every data exchange happens via our Application, we can implement some kind of service whitelisting (so that e.g. the HTTP plugin is not allowed to call the Twitter API).
We also offer a plugin the option to request the injection of specific environment variables (e.g. the credentials for the twitter API).
Both constraints are defined in the info.json file. This file can also have other values, such as plugin version and name. It is meant as the contract between the application (and the enduser) and the plugin (and the plugin developer).

If the application loads the plugin for the first time, it might present the user a confirmation dialog with all the traits that a plugin requests for usage.
This is a similar approach to the confirmation dialog on mobile devices (Android/iOS) that a new app triggers during the first startup.

In our example, the info.json file of the Twitter plugin looks like this:

{
    "whitelist": [
        "https://api.twitter.com"
    ],
    "env_variables": [
        "TWITTER_API_KEY",
        "TWITTER_API_SECRET_KEY",
        "TWITTER_ACCESS_TOKEN",
        "TWITTER_ACCESS_TOKEN_SECRET"
    ]
}

The plugin wants to connect to the URI https://api.twitter.com and will use the env variables defined in env_variables (e.g. in our case the credentials for the twitter API).

The other file is the plugin implementation itself client.js:

(function(){
    request = {
        "oauth1": {
            "consumerKey": env("TWITTER_API_KEY"),
            "consumerSecret": env("TWITTER_API_SECRET_KEY"),
            "accessToken": env("TWITTER_ACCESS_TOKEN"),
            "accessSecret": env("TWITTER_ACCESS_TOKEN_SECRET")
        },
        "host":"https://api.twitter.com/1.1/search/tweets.json?q=from%3Atwitterdev&result_type=mixed&count=2"
    }
    response = GET(request);
    var body = response["body"];
    for(i in body["statuses"]) {
        var status = body["statuses"][i];
        console.log(status["created_at"] + " @"+status["user"]["screen_name"] + ": " + status["text"]);
    }
})();

This call is from the basic example of the Twitter developer documentation. The interesting thing is, that the plugin developer does not have to implement the oauth1 request by herself. The api will automatically use the correct flow if the values for a specific key (e.g. oauth1) are defined within the request object. If you want to know more, you can have a look into the specific implementation.

Running the plugin presents us the expected result:

connections $ go run main.go

running Plugin twitter-plugin:

2020/03/06 16:06:56 info: map[env_variables:[TWITTER_API_KEY TWITTER_API_SECRET_KEY TWITTER_ACCESS_TOKEN TWITTER_ACCESS_TOKEN_SECRET] whitelist:[https://api.twitter.com]]

2020/03/06 16:06:56 Wed Feb 26 17:32:52 +0000 2020 @TwitterDev: 📄 Finally, @dara_tobi, the developer behind @QuotedReplies, built an app that automatically hides replies that are… https://t.co/m1VLNwCcS8
2020/03/06 16:06:56 Wed Feb 26 17:32:51 +0000 2020 @TwitterDev: In November, we gave people the ability to hide replies to their Tweets. Starting today, we’re opening this feature… https://t.co/aN8kan0Lsw

The second example is about calling an URL that is blocked. The info.json is:

{
    "whitelist": [
        "http://example.com"
    ]
}

The plugin implementation tries to call another URI:

(function(){
    request = {
        "host":"https://google.com"
    }
    response = GET(request);
    LOG("error response: " + JSON.stringify(response));

    request = {
        "host":"http://example.com/api"
    }
    response = GET(request);
    var body = response["body"];
    LOG("response: " + body);
})();

The result is as expected:

running Plugin basic-plugin:

2020/03/06 16:06:56 info: map[whitelist:[http://example.com]]

2020/03/06 16:06:56 error response: {"error":"accessing https://google.com is blocked"}
2020/03/06 16:06:56 response: <!doctype html>
<html>
<head>
    <title>Example Domain</title>
…
</html>

Again, you can have a look at the implementation.

Trigger Plugins by Events

Sometimes you just don’t want to call a plugin directly, but have an internal event triggering the plugin.

The next example will demonstrate exactly this. We have two plugins:

For the demonstration purpose, this example uses a Data Generator, that creates random events of the types CREATE, READ, UPDATE and DELETE. Either for a user or an object.

In the specific info.json file, the plugin registers itself as a listener for specific events:

{
    "events" : [
        "create",
        "update"
    ]
}

The setup of the event notification is done during application startup but can also be done during runtime.

var listeners map[string][]string
var scripts map[string]string

func init() {
    scripts = make(map[string]string)
    listeners = make(map[string][]string)
    for _, event := range []string{"create", "read", "update", "delete"} {
        listeners[event] = make([]string, 0)
    }
    err := loadPlugins()
    if err != nil {
        panic(err)
    }
}

func loadPlugins() error {
    files, err := ioutil.ReadDir(".")
    if err != nil {
        return err
    }

    for _, file := range files {
        if file.IsDir() {
            info, err := utils.ReadJSON(path.Join(file.Name(), "info.json"))
            if err != nil {
                return err
            }

            script, err := utils.ReadFile(path.Join(file.Name(), "script.js"))
            if err != nil {
                return err
            }
            scripts[file.Name()] = script
            events := info["events"].([]interface{})
            for _, eventEntry := range events {
                event := eventEntry.(string)
                listeners[event] = append(listeners[event], file.Name())
                log.Printf("register %s for event %s", file.Name(), event)
            }
        }
    }
    return nil
}

After that, you have two maps:

There is also a notifier method, that triggers the matching script based on the event type:

func notifyListener(data map[string]string) error {
    event := data["event"]
    if listenerScripts, ok := listeners[event]; ok {
        vm := otto.New()
        for _, name := range listenerScripts {
            log.Printf("notify %s about %s event", name, event)
            err := vm.Set("data", data)
            if err != nil {
                return err
            }
            _, err = vm.Run(scripts[name])
            if err != nil {
                return err
            }
        }
    }
    return nil
}

This notifier is triggered if the save method is called via the creator plugin.

func saveMethod(data map[string]string) error {
    log.Printf("saving some data %s", data)
    return notifyListener(data)
}

So the userUpdate plugin is only executed if an create or update events is triggered. The plugin itself then implements further filtering to only run for user types:

(function(){
    console.log("event: " + JSON.stringify(data));
    if(data["type"] === "user") {
        console.log("Ok, I will also update the User DB!");
    }
})();

The output is, again, as expected:

data: {"event":"delete","id":"nmaaHjXSoB7bmKpBDFUHqU","type":"user"}
2020/03/06 16:25:54 saving some data map[event:delete id:nmaaHjXSoB7bmKpBDFUHqU type:user]

data: {"event":"update","id":"KhVQFehukZdaM9kVQHdCr3","type":"object"}
2020/03/06 16:25:54 saving some data map[event:update id:KhVQFehukZdaM9kVQHdCr3 type:object]
2020/03/06 16:25:54 notify userUpdater about update event
event: {"event":"update","id":"KhVQFehukZdaM9kVQHdCr3","type":"object"}

data: {"event":"create","id":"KyPpkkVCYtyLFy3cfGuxd9","type":"user"}
2020/03/06 16:25:54 saving some data map[event:create id:KyPpkkVCYtyLFy3cfGuxd9 type:user]
2020/03/06 16:25:54 notify userUpdater about create event
event: {"event":"create","id":"KyPpkkVCYtyLFy3cfGuxd9","type":"user"}
Ok, I will also update the User DB!

data: {"event":"delete","id":"sAZhYCCL3kkvhF9w3iwQZg","type":"object"}
2020/03/06 16:25:54 saving some data map[event:delete id:sAZhYCCL3kkvhF9w3iwQZg type:object]

Drawbacks

The current implementation of the JavaScript interpreter has some drawbacks.
Debugging of plugins can be quite tedious. Fortunately, one of the recent contributions was the integration of a Debugging hook. Now you can at least see, where in your AST Parsing a problem occured.

Another big issue can be the limiting support of JavaScript. As previously mentioned, at the moment the Interpreter only understands ECMAScript v5.
This means that you cannot use the most modern JS Packages. At some point in your project, you have to decide if the limited functionality is sufficient to fulfill the use cases.

On the other hand, Otto comes with a full-blown AST Parser. There is already one attempt to implement JSX Parsing with Otto.

Summary

As you can see, it is possible to implement very different use cases with this straightforward technology.
It is quite easy to run foreign code in a sandbox-like environment. You can also implement auditing functionality, since you have a well defined dataflow.
You can also enhance the plugin loading mechanism with additional security checks, like signature verification of plugins.

TAGS