Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add standard chunking format #914

Merged
merged 5 commits into from
Apr 26, 2016
Merged

Add standard chunking format #914

merged 5 commits into from
Apr 26, 2016

Commits on Apr 22, 2016

  1. Event streams should return # of records 

    * to enable limits by # of records for buffer chunks
* to be able to visualize data flow in fluentd
    @tagomoris
    tagomoris committed Apr 22, 2016
    Configuration menu
    Copy the full SHA
    88b372e View commit details
    Browse the repository at this point in the history

  2. add API to append(concat) whole event stream to chunks at once 

    * it improves performance
* it's needed to implement output plugins like ObjectBufferedOutput of v0.12
    @tagomoris
    tagomoris committed Apr 22, 2016
    Configuration menu
    Copy the full SHA
    6596c9a View commit details
    Browse the repository at this point in the history

  3. add a variation of output plugin, without #format method implementation 

    * This type of output provides the standard format of binary representation of buffer chunk payloads
  * It is actually equal to results of EventStream#to_msgpack_stream
  * We can provide some tools to read-and-rescue buffer chunks in failure situations for such chunks
* This output style is actually same with ObjectBufferedOutput of v0.12
    @tagomoris
    tagomoris committed Apr 22, 2016
    Configuration menu
    Copy the full SHA
    8e56525 View commit details
    Browse the repository at this point in the history

  4. add mixin to iterate events from chunks as event_stream

    @tagomoris
    tagomoris committed Apr 22, 2016
    Configuration menu
    Copy the full SHA
    c814a86 View commit details
    Browse the repository at this point in the history

  5. add a method not to iterate/re-pack content

    @tagomoris
    tagomoris committed Apr 22, 2016
    Configuration menu
    Copy the full SHA
    2845bad View commit details
    Browse the repository at this point in the history