filter_record_transformer does not support UTF-8 string in fluent.conf

[okkez]

• fluentd v0.14.17 and master HEAD
• Debian GNU/Linux and Ruby 2.4.1

See also y-ken/fluent-plugin-twilio#8 (comment) and following comments in the PR.

How to reproduce

<source>
  @type forward
  @label @NOTIFY
</source>

<label @NOTIFY>
  <filter>
    @type record_transformer
    <record>
      message こんにちは. ${record["name"]} has made a order of ${record["item"]} just now.
    </record>
  </filter>
  <match dummy.log>
    @type stdout
  </match>
</label>

and run following command.

$ echo '{"name": "太郎", "message": "ほげほげ", "item": "寿司" }' | env LANG=C bundle exec fluent-cat dummy.log

And got following error:

2017-06-05 12:14:33 +0900 [warn]: #0 dump an error event: error_class=Encoding::CompatibilityError error="incompatible character encodings: ASCII-8BIT and UTF-8" tag="dummy.log" time=#<Fluent::EventTime:0x0055965471a558 @sec=1496632473, @nsec=421845491> record={"name"=>"\u592A\u90CE", "message"=>"\u307B\u3052\u307B\u3052", "item"=>"\u5BFF\u53F8"}

My investigation

This line does not consider encoding.

• Fluentd reads all files as ASCII-8BIT https://github.com/fluent/fluentd/blob/master/lib/fluent/supervisor.rb#L561
• String value will encode as UTF-8 by https://github.com/fluent/fluentd/blob/master/lib/fluent/config/types.rb#L67 and https://github.com/fluent/fluentd/blob/master/lib/fluent/config/types.rb#L88
• filter_record_transformer parses fluent.conf by its own code https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin/filter_record_transformer.rb#L48

Solution A

Fluentd reads fluent.conf as UTF-8.
On the other hand, other files should be treated as ASCII-8BIT I think.

• Pros
  • We don't need to fix filter_record_transformer
  • We don't need to fix other plugins for same issue
• Cons
  • Probably there is large side effect

Solution B

Fix filter_record_transformer to consider encoding.

• Pros
  • No side effect
• Cons
  • We need to fix other plugins in near future

[y-ken]

Another users has faced to related encoding problem with some plugins.

• datacounter
  http://inokara.hateblo.jp/entry/2015/06/24/010004

• grep
  https://blue21neo.blogspot.jp/2016/03/centos6fluentd-filtergrep.html

[repeatedly]

It works with 0.12.
So v0.14 breaks the compatibility and it should be fixed.
@okkez Could you write a patch?

[okkez]

OK, I will create patch according to solution A.

[y-ken]

Thank you!