Any Flutned Forward Protocol Specification?

[kawanet]

Is there any of documentation which describes a protocol between out_forward and in_forward plugins?

http://docs.fluentd.org/articles/in_forward describes some part of the protocol but not completed. PackedForward is used by out_forward but not documented, for instance.

Protocol

This plugin accepts both JSON or MessagePack messages and automatically detects 
which is used. Internally, Fluent uses MessagePack as it is more efficient than JSON.
The time value is a platform specific integer and is based on the output of Ruby’s 
Time.now.to_i function. On Linux, BSD and MAC systems, this is the number of 
seconds since 1970.

Multiple messages may be sent in the same connection.

stream:
  message...

message:
  [tag, time, record]
  or
  [tag, [[time,record], [time,record], ...]]

example:
  ["myapp.access", 1308466941, {"a":1}]["myapp.messages", 1308466942, {"b":2}]
  ["myapp.access", [[1308466941, {"a":1}], [1308466942, {"b":2}]]]

After diving into in_forward.rb, I've tried to write a snippet about the protocol specification.

https://gist.github.com/kawanet/078e274952638fd53150

It also mentions EventTime ext format of type 0 discussed on issue #653 as well as plain old Integer time. I hope some protocol experts could give a comment on it.

[repeatedly]

Thanks for writing this up.
Fluentd is now growing so we need such spec for new clients and eco-system.
I will check it your gist later.

[kawanet]

Multiple messages may be sent in the same connection.

I'm sorry but I've been missing the streaming feature. I've added it as well as Grammar section below.

Connection ::= <<Request>>*

Request ::= Message | Forward | PackedForward | nil

Message ::= [ Tag, Time, Record, Option? ]

Forward ::= [ Tag, MultiEventStream, Option? ]

MultiEventStream ::= [ Event* ]

PackedForward ::= [ Tag, MessagePackEventStream, Option? ]

MessagePackEventStream ::= <<Event>>*

Event ::= [ Time, Record ]

Tag ::= string

Time ::= integer | EventTime

Record ::= object

Option ::= object

[kawanet]

I'd like to ask experts about some points below, for instance.

(1)

Note: In addition to the three carrier modes, in_forward plugin also accepts JSON representation of a single event record for convenience. It detect it by the first byte of the request.

in_forward accepts JSON representation as well as msgpack representation. I like the convenience. However, I'm not sure that it should be a part of the protocol specification. To make it clear and simple, the specification should define msgpack stream as the standard, in my opinion. out_forward doesn't use it. I've mentioned JSON as an optional, then.

---

(2)

Client SHOULD send a MessagePackEventStream as msgpack bin format as it's a binary representation.
Client MAY send a MessagePackEventStream as msgpack str format by compatibility reasons.
Server MUST accept both formats of bin and str.

I guess there is a simple reason that the msgpack specification had no bin format, at the moment of the first fluentd shipping.

I've confirmed that in_forward accepts bin as well as str format. The specification should define that bin is the standard format, in my opinion, as it is a binary but not a string. str is a variant for compatibility. Some other platforms, including JavaScript, understand the difference between binary and string. Both works on Ruby though

---

(3)

I found only one usage of the option parameter. It's chunk which is used for health checking. Is there any other usages of the option parameter?

{"chunk": "p8n9gmxTQVC8/nh2wlKKeQ=="}

[repeatedly]

(1)

This is for some environment which doesn't have msgpack library, e.g. newest IoT devices.
"JSON as an optional" is correct.

(2)

Right. Some msgpack libraries don't support latest msgpack format yet so in_forward should support bin and str.
bin is better than str as an event stream format.

(3)

One is compression support: #637
Another one is adding the number of events for event stream metrics.

[tagomoris]

Oh, I missed this issue. I'll make progress about this in these few weeks.

[tagomoris]

I created a Forward Protocol Specification page on Wiki.
https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification

• based on @kawanet 's draft https://gist.github.com/kawanet/078e274952638fd53150
• added overview of that document
• added protocol versions, and add v0 versioning on current specification
• added a definition about event, and fixed some event records into events

I did some changes before committing first version, but almost changes are available with this link:
https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification/_compare/1521c584ec1457bb1a9193ce6309b7219a15c73e...bda64194209c7afce4c37b3f5eaa7034cd02f868

@repeatedly @kawanet Could you review contents?

[kawanet]

gorgeous!

[tagomoris]

I added some fixes for English, and added " v0" suffix for page name.
@repeatedly I'll remove This page is under construction. if you cannot find any problems.

[repeatedly]

It seems good.

[tagomoris]

OK. I removed "under construction" and added a section for Changes.
https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification-v0