Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implementation of the amazon kinesis data streams fluent bit plugin #1

Merged
merged 12 commits into from
Oct 17, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Binaries for programs and plugins
*.exe
*.exe~
*.dll
*.so
*.dylib

# Test binary, built with `go test -c`
*.test

# build output dir
bin

# Output of the go coverage tool, specifically when used with LiteIDE
*.out
44 changes: 44 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

ROOT := $(shell pwd)

all: build

SCRIPT_PATH := $(ROOT)/scripts/:${PATH}
SOURCES := $(shell find . -name '*.go')
PLUGIN_BINARY := ./bin/kinesis.so

.PHONY: release
release:
hencrice marked this conversation as resolved.
Show resolved Hide resolved
mkdir -p ./bin
go build -buildmode c-shared -o ./bin/kinesis.so ./
@echo "Built Amazon Kinesis Data Streams Fluent Bit Plugin"

.PHONY: build
build: $(PLUGIN_BINARY) release

$(PLUGIN_BINARY): $(SOURCES)
PATH=${PATH} golint ./kinesis

.PHONY: generate
generate: $(SOURCES)
PATH=$(SCRIPT_PATH) go generate ./...

.PHONY: test
test:
go test -timeout=120s -v -cover ./...

.PHONY: clean
clean:
rm -rf ./bin/*
52 changes: 47 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,51 @@
## Fluent Bit Plugin for Kinesis Streams
## Fluent Bit Plugin for Amazon Kinesis Data Streams

A Fluent Bit output plugin for Kinesis Streams
A Fluent Bit output plugin for Amazon Kinesis Data Streams.

# Under Development
#### Security disclosures

## License
If you think you’ve found a potential security issue, please do not post it in the Issues. Instead, please follow the instructions [here](https://aws.amazon.com/security/vulnerability-reporting/) or email AWS security directly at [[email protected]](mailto:[email protected]).

This library is licensed under the Apache 2.0 License.
### Plugin Options

* `region`: The region which your Kinesis Data Stream is in.
* `stream`: The name of the Kinesis Data Stream that you want log records sent to.
* `partition_key`: A partition key is used to group data by shard within a stream. A Kinesis Data Stream uses the partition key that is associated with each data record to determine which shard a given data record belongs to. For example, if your logs come from Docker containers, you can use container_id as the partition key, and the logs will be grouped and stored on different shards depending upon the id of the container they were generated from. As the data within a shard are coarsely ordered, you will get all your logs from one container in one shard roughly in order. If you don't set a partition key or put an invalid one, a random key will be generated, and the logs will be directed to random shards. If the partition key is invalid, the plugin will print an warning message.
* `data_keys`: By default, the whole log record will be sent to Kinesis. If you specify key name(s) with this option, then only those keys and values will be sent to Kinesis. For example, if you are using the Fluentd Docker log driver, you can specify `data_keys log` and only the log message will be sent to Kinesis. If you specify multiple keys, they should be comma delimited.
* `role_arn`: ARN of an IAM role to assume (for cross account access).
* `endpoint`: Specify a custom endpoint for the Kinesis Streams API.
* `append_newline`: If you set append_newline as true, a newline will be addded after each log record.

### Permissions

The plugin requires `kinesis:PutRecords` permissions.

### Credentials

This plugin uses the AWS SDK for Go, and uses its [default credential provider chain](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html). If you are using the plugin on Amazon EC2 or Amazon ECS, the plugin will use your EC2 instance role or ECS Task role permissions. The plugin can also retrieve credentials from a (shared credentials file)[https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html], or from the standard `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN` environment variables.

### Environment Variables

* `FLB_LOG_LEVEL`: Set the log level for the plugin. Valid values are: `debug`, `info`, and `error` (case insensitive). Default is `info`. **Note**: Setting log level in the Fluent Bit Configuration file using the Service key will not affect the plugin log level (because the plugin is external).
* `SEND_FAILURE_TIMEOUT`: Allows you to configure a timeout if the plugin can not send logs to Kinesis Streams. The timeout is specified as a [Golang duration](https://golang.org/pkg/time/#ParseDuration), for example: `5m30s`. If the plugin has failed to make any progress for the given period of time, then it will exit and kill Fluent Bit. This is useful in scenarios where you want your logging solution to fail fast if it has been misconfigured (i.e. network or credentials have not been set up to allow it to send to Kinesis Streams).

### Fluent Bit Versions

This plugin has been tested with Fluent Bit 1.2.0+. It may not work with older Fluent Bit versions. We recommend using the latest version of Fluent Bit as it will contain the newest features and bug fixes.

### Example Fluent Bit Config File

```
[INPUT]
Name forward
Listen 0.0.0.0
Port 24224
[OUTPUT]
Name kinesis
Match *
region us-west-2
stream my-kinesis-stream-name
partition_key container_id
append_newline true
```
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also add the Docker Hub and ECR links, same as in the Firehose readme: https://github.com/aws/amazon-kinesis-firehose-for-fluent-bit#docker-hub

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will update it onec our images are avaiable on Docker Hub and ECR.

156 changes: 156 additions & 0 deletions fluent-bit-kinesis.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,156 @@
// Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License"). You may
// not use this file except in compliance with the License. A copy of the
// License is located at
//
// http://aws.amazon.com/apache2.0/
//
// or in the "license" file accompanying this file. This file is distributed
// on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
// express or implied. See the License for the specific language governing
// permissions and limitations under the License.

package main

import (
"fmt"
"C"
"unsafe"
"strings"

"github.com/aws/amazon-kinesis-streams-for-fluent-bit/kinesis"
"github.com/aws/amazon-kinesis-firehose-for-fluent-bit/plugins"
"github.com/fluent/fluent-bit-go/output"
"github.com/sirupsen/logrus"
)

var (
pluginInstances []*kinesis.OutputPlugin
)

func addPluginInstance(ctx unsafe.Pointer) error {
hencrice marked this conversation as resolved.
Show resolved Hide resolved
pluginID := len(pluginInstances)
hencrice marked this conversation as resolved.
Show resolved Hide resolved
output.FLBPluginSetContext(ctx, pluginID)
hencrice marked this conversation as resolved.
Show resolved Hide resolved
instance, err := newKinesisOutput(ctx, pluginID)
if err != nil {
return err
}

pluginInstances = append(pluginInstances, instance)
return nil
}

func getPluginInstance(ctx unsafe.Pointer) *kinesis.OutputPlugin {
pluginID := output.FLBPluginGetContext(ctx).(int)
return pluginInstances[pluginID]
}


func newKinesisOutput(ctx unsafe.Pointer, pluginID int) (*kinesis.OutputPlugin, error) {
stream := output.FLBPluginConfigKey(ctx, "stream")
logrus.Infof("[kinesis %d] plugin parameter stream = '%s'", pluginID, stream)
region := output.FLBPluginConfigKey(ctx, "region")
logrus.Infof("[kinesis %d] plugin parameter region = '%s'", pluginID, region)
dataKeys := output.FLBPluginConfigKey(ctx, "data_keys")
logrus.Infof("[kinesis %d] plugin parameter data_keys = '%s'", pluginID, dataKeys)
partitionKey := output.FLBPluginConfigKey(ctx, "partition_key")
logrus.Infof("[kinesis %d] plugin parameter partition_key = '%s'", pluginID, partitionKey)
roleARN := output.FLBPluginConfigKey(ctx, "role_arn")
logrus.Infof("[kinesis %d] plugin parameter role_arn = '%s'", pluginID, roleARN)
endpoint := output.FLBPluginConfigKey(ctx, "endpoint")
logrus.Infof("[kinesis %d] plugin parameter endpoint = '%s'", pluginID, endpoint)
appendNewline := output.FLBPluginConfigKey(ctx, "append_newline")
logrus.Infof("[kinesis %d] plugin parameter append_newline = %s", pluginID, appendNewline)

if stream == "" || region == "" {
return nil, fmt.Errorf("[kinesis %d] stream and region are required configuration parameters", pluginID)
}

if partitionKey == "log" {
return nil, fmt.Errorf("[kinesis %d] 'log' cannot be set as the partition key", pluginID)
}

if partitionKey == "" {
logrus.Infof("[kinesis %d] no partition key provided. A random one will be generated.", pluginID)
}

appendNL := false
if strings.ToLower(appendNewline) == "true" {
appendNL = true
}
return kinesis.NewOutputPlugin(region, stream, dataKeys, partitionKey, roleARN, endpoint, appendNL, pluginID)
}


// The "export" comments have syntactic meaning
// This is how the compiler knows a function should be callable from the C code

//export FLBPluginRegister
func FLBPluginRegister(ctx unsafe.Pointer) int {
return output.FLBPluginRegister(ctx, "kinesis", "Amazon Kinesis Data Streams Fluent Bit Plugin.")
}

//export FLBPluginInit
func FLBPluginInit(ctx unsafe.Pointer) int {
plugins.SetupLogger()
logrus.Debugf("[kinesis] Debug log level test successful")
err := addPluginInstance(ctx)
if err != nil {
logrus.Errorf("[kinesis] Failed to initialize plugin: %v\n", err)
return output.FLB_ERROR
}
return output.FLB_OK
}

//export FLBPluginFlushCtx
func FLBPluginFlushCtx(ctx, data unsafe.Pointer, length C.int, tag *C.char) int {
var count int
var ret int
var record map[interface{}]interface{}

// Create Fluent Bit decoder
dec := output.NewDecoder(data, int(length))

kinesisOutput := getPluginInstance(ctx)
fluentTag := C.GoString(tag)
logrus.Debugf("[kinesis %d] Found logs with tag: %s\n", kinesisOutput.PluginID, fluentTag)

for {
//Extract Record
ret, _, record = output.GetRecord(dec)
if ret != 0 {
break
}

retCode := kinesisOutput.AddRecord(record)
if retCode != output.FLB_OK {
return retCode
}
count++
}
err := kinesisOutput.Flush()
if err != nil {
logrus.Errorf("[kinesis %d] %v\n", kinesisOutput.PluginID, err)
return output.FLB_ERROR
}
logrus.Debugf("[kinesis %d] Processed %d events with tag %s\n", kinesisOutput.PluginID, count, fluentTag)

return output.FLB_OK
}

//export FLBPluginExit
func FLBPluginExit() int {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will fluentbit guarantee that all the plugin instances have a chance to flush? Asking because I'm not sure whether this method should be as simple as just returning ok or it should implement some form of graceful shutdown.

Copy link
Contributor Author

@hossain-rayhan hossain-rayhan Oct 10, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now, before the final exit, calling Flush() for all the instances as the final try. Also, printing the error message if it fails for any plugin instance.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hossain-rayhan Do you want to create issues on the other fluent bit plugin repos to request the same change. All of them should have this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@PettitWesley Good thought. I will create issues for the other fluent-bit plugins.

// Before final exit, call Flush() for all the instances of the Output Plugin
for i := range pluginInstances {
err := pluginInstances[i].Flush()
if err != nil {
logrus.Errorf("[kinesis %d] %v\n", pluginInstances[i].PluginID, err)
}
}

return output.FLB_OK
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should probably return FLB_ERROR if any of the flushes failed, so that Fluent Bit can record the failure.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure but earlier we decied only to print the error messages in case of failure. And, finally return FLB_OK from the FLBPluginExit() as the default behaviour.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 We should clearly define the behavior and be explicit in the doc.

}

func main() {
hencrice marked this conversation as resolved.
Show resolved Hide resolved
}
13 changes: 13 additions & 0 deletions go.mod
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
module github.com/aws/amazon-kinesis-streams-for-fluent-bit

go 1.12

require (
github.com/aws/amazon-kinesis-firehose-for-fluent-bit v0.0.0-20190911230620-0883cb76f511
github.com/aws/aws-sdk-go v1.25.1
github.com/fluent/fluent-bit-go v0.0.0-20190925192703-ea13c021720c
github.com/golang/mock v1.3.1
github.com/json-iterator/go v1.1.7
github.com/sirupsen/logrus v1.4.2
github.com/stretchr/testify v1.3.0
)
47 changes: 47 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
github.com/aws/amazon-kinesis-firehose-for-fluent-bit v0.0.0-20190911230620-0883cb76f511 h1:h4ta9iM29D9zSiLCmr7ybUY5qcv2QgP+GVjipJ1Kx/4=
github.com/aws/amazon-kinesis-firehose-for-fluent-bit v0.0.0-20190911230620-0883cb76f511/go.mod h1:jZjLd+hsaK0oNVotfqbMK9gUsIhXvYEl7Z3tdZTYa54=
github.com/aws/aws-sdk-go v1.20.6/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN924inxo=
github.com/aws/aws-sdk-go v1.25.1 h1:d7zDXFT2Tgq/yw7Wku49+lKisE8Xc85erb+8PlE/Shk=
github.com/aws/aws-sdk-go v1.25.1/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN924inxo=
github.com/cenkalti/backoff v2.1.1+incompatible h1:tKJnvO2kl0zmb/jA5UKAt4VoEVw1qxKWjE/Bpp46npY=
github.com/cenkalti/backoff v2.1.1+incompatible/go.mod h1:90ReRw6GdpyfrHakVjL/QHaoyV4aDUVVkXQJJJ3NXXM=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/fluent/fluent-bit-go v0.0.0-20190614024040-c017a8579953/go.mod h1:WQX+afhrekY9rGK+WT4xvKSlzmia9gDoLYu4GGYGASQ=
github.com/fluent/fluent-bit-go v0.0.0-20190925192703-ea13c021720c h1:QwbffUs/+ptC4kTFPEN9Ej2latTq3bZJ5HO/OwPXYMs=
github.com/fluent/fluent-bit-go v0.0.0-20190925192703-ea13c021720c/go.mod h1:WQX+afhrekY9rGK+WT4xvKSlzmia9gDoLYu4GGYGASQ=
github.com/golang/mock v1.3.1 h1:qGJ6qTW+x6xX/my+8YUVl4WNpX9B7+/l2tRsHGZ7f2s=
github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y=
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af h1:pmfjZENx5imkbgOkpRUYLnmbU7UEFbjtDA2hxJ1ichM=
github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=
github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
github.com/json-iterator/go v1.1.7 h1:KfgG9LzI+pYjr4xvmz/5H4FXjokeP+rlHLhv3iH62Fo=
github.com/json-iterator/go v1.1.7/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI=
github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/sirupsen/logrus v1.4.2 h1:SPIRibHv4MatM3XXNO2BJeFLZwZ2LvZgfQ5+UNI2im4=
github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/ugorji/go v1.1.4 h1:j4s+tAvLfL3bZyefP2SEWmhBzmuIlH/eqNuPdFPgngw=
github.com/ugorji/go v1.1.4/go.mod h1:uQMGLiO92mf5W77hV/PUCpI3pbzQx3CRekS0kk+RGrc=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190422165155-953cdadca894 h1:Cz4ceDQGXuKRnVBDTS23GTn/pU5OE2C0WrNTOYK1Uuc=
golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/tools v0.0.0-20190425150028-36563e24a262/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
16 changes: 16 additions & 0 deletions kinesis/generate_mock.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
// Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License"). You may
// not use this file except in compliance with the License. A copy of the
// License is located at
//
// http://aws.amazon.com/apache2.0/
//
// or in the "license" file accompanying this file. This file is distributed
// on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
// express or implied. See the License for the specific language governing
// permissions and limitations under the License.

package kinesis

//go:generate mockgen.sh github.com/aws/amazon-kinesis-streams-for-fluent-bit/kinesis PutRecordsClient mock_kinesis/mock.go
Loading