gRPC Plugin framework (#1461)

* Add New Storage Plugin Framework

Still a work in progress. Related to issue #422.
Leverages Hashicorp's go-plugin.
New set of protos for the use of plugin maintainers.
Missing tests. Still need to create a plugin binary for the memory backend.

Signed-off-by: Olivier Boucher <[email protected]>

Added Memory GRPC Plugin. Renamed packages (#422)

Renamed packages from "external" to "grpc".
Changed storage type to "grpc-plugin".
Added cmd/memory-grpc-plugin that uses the memory backend as a grpc plugin.
Added go-plugin dependency to Glide.
Added task to generate plugin protos to Makefile.

Signed-off-by: Olivier Boucher <[email protected]>

Refactored gRPC storage plugin protos

Now makes use of protos from model/proto/model.proto and removed mapping functions.
Plugin invoker calls the plugin binary directly instead of relying on sh.

Signed-off-by: Olivier Boucher <[email protected]>

Refactored gRPC plugin protos to use defined models

Removed mapping code.
Removed used of sh, calling plugin binary directly.

Signed-off-by: Olivier Boucher <[email protected]>

Removed .editorconfig file

Signed-off-by: Olivier Boucher <[email protected]>

Added support for gRPC plugin configuration

Configuration file is passed down to the plugin through the --config argument.
It is the plugin's responsability to parse the file and initialize itself.

Signed-off-by: Olivier Boucher <[email protected]>

Added benchmark for noop SpanWriter vs gRPC based noop plugin

Fixed issues related to proto changes.
Deleted memory-grpc-plugin since it was no longer relevant.

Signed-off-by: Olivier Boucher <[email protected]>

Updated gRPC plugin & protos to implement latest SpanReader

Added dependencies to Gopkg.lock since Glide -> dep transition happened recently.

Signed-off-by: Olivier Boucher <[email protected]>

Fixed proto generation introduced when switching to Dep

Changed json tags to match when TraceID was not a proto.

Signed-off-by: Olivier Boucher <[email protected]>

proto fix

Fixed issue introduced moving TraceID to model.proto

Added proper annotations to storage.proto instead of exposing TraceID in model.proto.
Tests are back to green.

Signed-off-by: Olivier Boucher <[email protected]>

Added tests for gRPC storage factory and options

Moved DependencyLink to model.proto since it had to be used and the existing struct did not implement any Marshaler/Unmarshaler methods.

Changed storage configuration to allow testing via mocks.

Added DependencyReader interface to the StoragePlugin interface.

Signed-off-by: Olivier Boucher <[email protected]>

* Remove old generated storage file

Signed-off-by: Charles Dixon <[email protected]>

* Update plugin grpc interface/implementation

Signed-off-by: Charles Dixon <[email protected]>

* Fix gosec by adding an exclude comment

Signed-off-by: Charles Dixon <[email protected]>

* Run make fmt

Signed-off-by: Charles Dixon <[email protected]>

* Add comments to uncommented functions

Signed-off-by: Charles Dixon <[email protected]>

* Split grpc server and client into separate files

Signed-off-by: Charles Dixon <[email protected]>

* Use errors.wrap rather than fmt for errors

Signed-off-by: Charles Dixon <[email protected]>

* Remove unused plugin file

Signed-off-by: Charles Dixon <[email protected]>

* Add copyright header to grpc server file

Signed-off-by: Charles Dixon <[email protected]>

* Update headers to updated license on uncommitted files

Signed-off-by: Charles Dixon <[email protected]>

* Move grpc config to plugin/storage

Signed-off-by: Charles Dixon <[email protected]>

* Add empty test files to fix build

Signed-off-by: Yuri Shkuro <[email protected]>

* Move grpc config empty_test file

Signed-off-by: Charles Dixon <[email protected]>

* Change fmt for error to errors.Wrap

Signed-off-by: Charles Dixon <[email protected]>

* Use single-line for comprehensions where possible

Signed-off-by: Charles Dixon <[email protected]>

* Don't accumulate spans before sending

Signed-off-by: Charles Dixon <[email protected]>

* Add function to serve the plugin

Signed-off-by: Charles Dixon <[email protected]>

* Small grpc client refactor

Signed-off-by: Charles Dixon <[email protected]>

* Update grpc interface and move noop plugin to examples

Signed-off-by: Charles Dixon <[email protected]>

* Add missed fix for factory

Signed-off-by: Charles Dixon <[email protected]>

* Rename memory-store example to memstore-plugin

Signed-off-by: Charles Dixon <[email protected]>

* Refactor Serve to call ServeWithGRPCServer

Signed-off-by: Charles Dixon <[email protected]>

* Refactor to make some structs private

Signed-off-by: Charles Dixon <[email protected]>

* Change factory_test to use mocks

Signed-off-by: Charles Dixon <[email protected]>

* Add storage grpc integration test

Signed-off-by: Charles Dixon <[email protected]>

* Create mocks for storage_v1 and add grpc client tests

Signed-off-by: Charles Dixon <[email protected]>

* Add grpc server tests

Signed-off-by: Charles Dixon <[email protected]>

* Run make fmt

Signed-off-by: Charles Dixon <[email protected]>

* Strip unnecessary code from test

Signed-off-by: Charles Dixon <[email protected]>

Gopkg.toml

@@ -157,3 +157,11 @@ required = [
 [[constraint]]
   name = "github.com/rs/cors"
   version = "1.3.0"
+
+[[constraint]]
+  name = "github.com/hashicorp/go-plugin"
+  version = "1.0.0"
+
+[[constraint]]
+  name = "github.com/hashicorp/go-hclog"
+  version = "0.8.0"

examples/memstore-plugin/empty_test.go

@@ -0,0 +1,15 @@
+// Copyright (c) 2018 The Jaeger Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main

examples/memstore-plugin/main.go

@@ -0,0 +1,65 @@
+// Copyright (c) 2018 The Jaeger Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+	"flag"
+	"path"
+	"strings"
+
+	"github.com/spf13/viper"
+
+	"github.com/jaegertracing/jaeger/plugin/storage/grpc"
+	"github.com/jaegertracing/jaeger/storage/dependencystore"
+	"github.com/jaegertracing/jaeger/plugin/storage/memory"
+	"github.com/jaegertracing/jaeger/storage/spanstore"
+)
+
+var configPath string
+
+func main() {
+	flag.StringVar(&configPath, "config", "", "A path to the plugin's configuration file")
+	flag.Parse()
+
+	if configPath != "" {
+		viper.SetConfigFile(path.Base(configPath))
+		viper.AddConfigPath(path.Dir(configPath))
+	}
+
+	v := viper.New()
+	v.AutomaticEnv()
+	v.SetEnvKeyReplacer(strings.NewReplacer("-", "_", ".", "_"))
+
+	opts := memory.Options{}
+	opts.InitFromViper(v)
+
+	grpc.Serve(&memoryStore{store: memory.NewStore()})
+}
+
+type memoryStore struct {
+	store *memory.Store
+}
+
+func (ns *memoryStore) DependencyReader() dependencystore.Reader {
+	return ns.store
+}
+
+func (ns *memoryStore) SpanReader() spanstore.Reader {
+	return ns.store
+}
+
+func (ns *memoryStore) SpanWriter() spanstore.Writer {
+	return ns.store
+}

plugin/storage/factory.go

@@ -26,6 +26,7 @@ import (
 	"github.com/jaegertracing/jaeger/plugin/storage/badger"
 	"github.com/jaegertracing/jaeger/plugin/storage/cassandra"
 	"github.com/jaegertracing/jaeger/plugin/storage/es"
+	"github.com/jaegertracing/jaeger/plugin/storage/grpc"
 	"github.com/jaegertracing/jaeger/plugin/storage/kafka"
 	"github.com/jaegertracing/jaeger/plugin/storage/memory"
 	"github.com/jaegertracing/jaeger/storage"
@@ -38,6 +39,7 @@ const (
 	elasticsearchStorageType = "elasticsearch"
 	memoryStorageType        = "memory"
 	kafkaStorageType         = "kafka"
+	grpcPluginStorageType    = "grpc-plugin"
 	badgerStorageType        = "badger"
 	downsamplingRatio        = "downsampling.ratio"
 	downsamplingHashSalt     = "downsampling.hashsalt"
@@ -48,7 +50,7 @@ const (
 	defaultDownsamplingHashSalt = ""
 )
 
-var allStorageTypes = []string{cassandraStorageType, elasticsearchStorageType, memoryStorageType, kafkaStorageType, badgerStorageType}
+var allStorageTypes = []string{cassandraStorageType, elasticsearchStorageType, memoryStorageType, kafkaStorageType, badgerStorageType, grpcPluginStorageType}
 
 // Factory implements storage.Factory interface as a meta-factory for storage components.
 type Factory struct {
@@ -90,6 +92,8 @@ func (f *Factory) getFactoryOfType(factoryType string) (storage.Factory, error)
 		return kafka.NewFactory(), nil
 	case badgerStorageType:
 		return badger.NewFactory(), nil
+	case grpcPluginStorageType:
+		return grpc.NewFactory(), nil
 	default:
 		return nil, fmt.Errorf("unknown storage type %s. Valid types are %v", factoryType, allStorageTypes)
 	}

plugin/storage/grpc/config/config.go

@@ -0,0 +1,76 @@
+// Copyright (c) 2019 The Jaeger Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package config
+
+import (
+	"fmt"
+	"os/exec"
+	"runtime"
+
+	"github.com/hashicorp/go-hclog"
+	"github.com/hashicorp/go-plugin"
+
+	"github.com/jaegertracing/jaeger/plugin/storage/grpc/shared"
+)
+
+// Configuration describes the options to customize the storage behavior
+type Configuration struct {
+	PluginBinary            string `yaml:"binary"`
+	PluginConfigurationFile string `yaml:"configuration-file"`
+}
+
+// Build instantiates a StoragePlugin
+func (c *Configuration) Build() (shared.StoragePlugin, error) {
+	// #nosec G204
+	cmd := exec.Command(c.PluginBinary, "--config", c.PluginConfigurationFile)
+
+	client := plugin.NewClient(&plugin.ClientConfig{
+		HandshakeConfig: shared.Handshake,
+		VersionedPlugins: map[int]plugin.PluginSet{
+			1: shared.PluginMap,
+		},
+		Cmd:              cmd,
+		AllowedProtocols: []plugin.Protocol{plugin.ProtocolGRPC},
+		Logger: hclog.New(&hclog.LoggerOptions{
+			Level: hclog.Warn,
+		}),
+	})
+
+	runtime.SetFinalizer(client, func(c *plugin.Client) {
+		c.Kill()
+	})
+
+	rpcClient, err := client.Client()
+	if err != nil {
+		return nil, fmt.Errorf("error attempting to connect to plugin rpc client: %s", err)
+	}
+
+	raw, err := rpcClient.Dispense(shared.StoragePluginIdentifier)
+	if err != nil {
+		return nil, fmt.Errorf("unable to retrieve storage plugin instance: %s", err)
+	}
+
+	storagePlugin, ok := raw.(shared.StoragePlugin)
+	if !ok {
+		return nil, fmt.Errorf("unexpected type for plugin \"%s\"", shared.StoragePluginIdentifier)
+	}
+
+	return storagePlugin, nil
+}
+
+// PluginBuilder is used to create storage plugins
+type PluginBuilder interface {
+	Build() (shared.StoragePlugin, error)
+}

plugin/storage/grpc/config/empty_test.go

@@ -0,0 +1,15 @@
+// Copyright (c) 2018 The Jaeger Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package config