Skip to content

Latest commit

 

History

History
195 lines (138 loc) · 6.86 KB

README.md

File metadata and controls

195 lines (138 loc) · 6.86 KB
Raw

Redis Output Plugin For fluentd

Fluentd output plugin to upload/publish event data to Redis storage.

Requirements

fluent-plugin-redis-store fluentd ruby
>= 0.2.0 >= v0.14.15 >= 2.1
< 0.2.0 >= v0.12.0 >= 1.9

Background

This is a forked project from fluent-plugin-redis-store.

Release Process

  1. Bump version in gemspec file
  2. gem build fluent-plugin-ddl-redis-store.gemspec
  3. gem push <.gem file> (with the gem file generated in the previous step)
    Ex: gem push fluent-plugin-ddl-redis-store-0.3.0.gem

The push command needs to be authenticated with RubyGems.org.

Latest release: https://rubygems.org/gems/fluent-plugin-ddl-redis-store

Features

Supported Redis commands

Currently the plugin supports following Redis commands:

  • set by string type (of the plugin)
  • lpush/rpush by list type
  • sadd by set type
  • zadd by zset type
  • publish by publish type

Supported value format

key string for Redis storage

Redis commands require key and value.
For key, the plugin supports either way;

  1. Specify a fixed key.
    You can do this simply using key option in td-agent configuration file.

    type redis_store
key userdata

  2. Lookup a key string in every event data by a lookup path.
    If event data have structured data like

    { "user": { "name": "Kei" } }

    and you want to use each name of user, you can use key_path option.

    type redis_store
key_path user.name

    With the above data, Kei will be a key.

In addition, key_prefix and key_suffix are useful in some cases. Both are available either key and key_path

type redis_store
key_path user.name
key_prefix ouruser.
key_suffix .accesslog

With the previous data, key will be outuser.Kei.accesslog.

value data for Redis storage

To determine what value in every event data to be srtored, you have two options;

  1. Store extracted data in event data, by a lookup path with value_path option.
    It works like key_path.
  2. Store whole data.
    This is default behavior. To do it, simply omit value_path option.

Installation

fluent-gem install fluent-plugin-redis-store

# or if you are using td-agent:
td-agent-gem install fluent-plugin-redis-store

Configuration

Redis connection

Key Type Required? Default Description
host string Optional 127.0.0.1 host name of Redis server
port int Optional 6379 port number of Redis server
password string Optional password for Redis connection
path string Optional To connect via Unix socket, try '/tmp/redis.sock'
db int Optional 0 DB number of Redis
timeout float Optional 5.0 connection timeout in seconds

common options for storages

Key Type Default Description
key string Fixed key used to store(publish) in Redis
key_path string path to lookup for key in the event data
key_prefix string prefix of key
key_suffix string suffix of key
value_path string (whole event data) path to lookup for value in the event data
store_type string zset string/list/set/zset/publish
format_type string plain format type for value (plain/json/msgpack)
key_expire int -1 If set, the key will be expired in specified seconds
flush_interval time 1 Time interval which events will be flushed to Redis

Note: either key or key_path is required.

string storage specific options

Key Type Default Description
type string Fixed key used to store(publish) in Redis
No more options than common options.

list storage specific options

Key Type Default Description
order string asc asc: rpush, desc: lpush

set storage specific options

No more options than common options.

zset storage specific options

Key Type Default Description
score_path string (time of log event) path to lookup for score in the event data
collision_policy string (nil) Only update elements that already exist (XX) or add a new element (NX)
value_expire int value expiration in seconds

If value_expire is set, the plugin assumes that the score in the SortedSet is based on timestamp and it deletes expired members every after new event data arrives.

publish storage specific options

No more options than common options.

Contributors

moaikids
HANAI tohru
Mohit Khanna
yamada-shinji
Heitor de Souza
Hiroshi Hatake
Gabriel Bordeaux

Copyright

Copyright (c) 2013 moaikids
Copyright (c) 2014 HANAI Tohru

License

Apache License, Version 2.0