PROPOSE: add format raw

[sonots]

I propose to add format raw.

Currently, even if users do not require to parse log lines, we have to write configuration like

<source>
    type tail
    format /^(?<message>.*)$/
    ....
</source>

This requires to parse log lines by a regular expression, and its overhead is not trivial.

       user     system      total        real
asis:    0.080000   0.000000   0.080000 (  0.081076)
regexp:  7.590000   0.000000   7.590000 (  7.592521)

cf. http://blog.1q77.com/2013/01/fluent-agent-lite-%E3%81%A8-in_tail/

This plugin enables to write configuration like

<source>
    type tail
    format raw
    message_key message
    ....
</source>

(where message_key is set to message as default) and does not require to parse log lines by a regular expression.

There is a plugin fluent-plugin-tail-asis which does the same thing with this patch, and downloaded 2354 times as in 2013.09.02 http://fluentd.org/plugin/. It tells us that some users are already using this feature, and I propose to have this feature in the fluentd itself.

[tagomoris]

Structured raw logs come here!

[nikushi]

👍

[SpringMT]

👍

[yancya]

👍

[futoase]

👍

[repeatedly]

format raw seems mismatch name.
For me, raw represents 'No transformation and emit logs directly' like splunk.

I want more clear name...

[futoase]

🌻

[frsyuki]

proposal: "format message"

It still creates a structured record and its key name is "msssage". raw doesn't imply this behavior.
Apart from the name, I think it's good idea to include this REGEXP. GJ!

[adriancdperu]

👍 totally up for this one. sometimes you just dont want to parse the stuff..

[studio3104]

👍

[kenjiskywalker]

👍

[sonots]

proposal: "format none"

I think that the value of format option should describe the name of a format.
format message lets me feel like "what was the message format?".

In contrast, I think format none clearly describes as its format is none (although I felt the same thing for format raw too ...)

How do you think?

[kentaro]

👍 > format none

or

How about format plain?

[y-ken]

I feel format plain is the best one.

[choplin]

proposal: format whole_line

This indicates directly what will comes in records.

[studio3104]

👍 > format plain

[sonots]

@repeatedly it seems format plain is the most popular. what do you think? If it's okay, I will modify the source code.

[tagomoris]

All of you can use fluent-agent-lite for raw-plain-whole-message record.
https://github.com/tagomoris/fluent-agent-lite

[kentaro]

@tagomoris Anybody knows your work

[yteraoka]

👍 > format none

[repeatedly]

format none or format plain are candidate.

For me, plain seems to be similar to raw.
So 👍 for format none.

I wait a day for decision.

[sonots]

@repeatedly renamed raw to none!!

[y-ken]

format none is also good 👍

[repeatedly]

none wins!

[sonots]

thanks!!