Failed to create a sub plugin instance after the plugin failed to start

[yoheimuta]

I want to ask how to avoid the problem.

I'm not sure to be appropriate to issue the problem in this place, because this is involved with other system and plugins like buf_file, fluent-plugin-s3 and out_copy.

But it would be helpful if there is a workaround against this issue.

DETAILED DESCRIPTION:

out_forest logged error messages like below after the instance went into lb.

2015-05-08 19:30:00 +0900 [error]: plugin/out_forest.rb:145:rescue in plant: failed to configure/start sub output copy: Net::ReadTimeout
2015-05-08 19:30:00 +0900 [error]: plugin/out_forest.rb:146:rescue in plant: /opt/td-agent/embedded/lib/ruby/2.1.0/net/protocol.rb:158:in `rescue in rbuf_fill'
/opt/td-agent/embedded/lib/ruby/2.1.0/net/protocol.rb:152:in `rbuf_fill'
/opt/td-agent/embedded/lib/ruby/2.1.0/net/protocol.rb:134:in `readuntil'
/opt/td-agent/embedded/lib/ruby/2.1.0/net/protocol.rb:144:in `readline'
/opt/td-agent/embedded/lib/ruby/2.1.0/net/http/response.rb:39:in `read_status_line'
/opt/td-agent/embedded/lib/ruby/2.1.0/net/http/response.rb:28:in `read_new'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/aws-sdk-v1-1.63.0/lib/aws/core/http/patch.rb:29:in `block in new_transport_request'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/aws-sdk-v1-1.63.0/lib/aws/core/http/patch.rb:26:in `catch'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/aws-sdk-v1-1.63.0/lib/aws/core/http/patch.rb:26:in `new_transport_request'
/opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:1378:in `request'
/opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:1371:in `block in request'
/opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:853:in `start'
/opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:1369:in `request'
...
...
...
2015-05-08 20:16:51 +0900 [error]: plugin/out_forest.rb:140:rescue in plant: failed to configure sub output copy: Other 's3' plugin already use same buffer_path: type = s3, buffe
r_path = /tmp/td-agent/s3-buffer/media_auth_log.ip-10-0-1-49
2015-05-08 20:16:51 +0900 [error]: plugin/out_forest.rb:141:rescue in plant: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/buf_file.rb:97:in `c
onfigure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:197:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:488:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-mixin-config-placeholders-0.3.0/lib/fluent/mixin/config_placeholders.rb:105:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-s3-0.5.7/lib/fluent/plugin/out_s3.rb:42:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/out_copy.rb:43:in `block in configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/out_copy.rb:34:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/out_copy.rb:34:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:132:in `block in plant'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:128:in `synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:128:in `plant'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:168:in `emit'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/event_router.rb:88:in `emit_stream'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/event_router.rb:79:in `emit'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_forward.rb:164:in `on_message'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_forward.rb:238:in `call'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_forward.rb:238:in `block in on_read_msgpack'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_forward.rb:237:in `feed_each'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_forward.rb:237:in `on_read_msgpack'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/io.rb:123:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/io.rb:186:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_forward.rb:91:in `run'
2015-05-08 20:16:51 +0900 [error]: plugin/out_forest.rb:142:rescue in plant: Cannot output messages with tag 'media_auth_log.ip-10-0-1-49'
...
# same errors (= failed to configure) were repeated until the instance went out from lb.

DIRECT ROOT CAUSE:

1. plant method is invoked when sub plugin is not created and a message comes.
   • First, configure method of out_copy plugin is invoked in this method.
   • Second, buffer_path is registered into buffer_paths array when sub plugin of copy plugin is a type of buffered.
   • Finally, start method of out_copy plugin is failed by any errors like troubles of network connection and plant method is also failed to create sub plugin.
2. Again, plant method is invoked when a second message comes because sub plugin is not created yet.
   • First, configure method of out_copy plugin is invoked again in this method.
   • Second, the configure method is failed because the buffer_path has already been registered into buffer_paths array.
   • This error is happened when a subsequent message comes until td-agent is restarted.

REPRODUCED MANUAL:

td-agent.conf

$ td-agent -c td-agent.conf
2015-05-11 10:40:21 +0900 [info]: reading config file path="td-agent.conf"
2015-05-11 10:40:21 +0900 [info]: starting fluentd-0.12.7
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-mixin-config-placeholders' version '0.3.0'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-amplifier-filter' version '0.1.6'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-bigquery' version '0.2.6'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-buffer-lightening' version '0.0.2'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-cloudwatch' version '1.2.9'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-config-expander' version '0.1.5'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-datacounter' version '0.4.3'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-file-sprintf' version '0.0.6'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-filter' version '0.0.3'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-flowcounter' version '0.2.1'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-forest' version '0.3.0'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-format' version '0.0.1'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-grep' version '0.3.3'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-mackerel' version '0.0.8'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-map' version '0.0.4'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-mongo' version '0.7.8'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-norikra-patched-7' version '0.2.2'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-ping-message' version '0.1.2'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-record-modifier' version '0.2.0'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-retag' version '0.0.1'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-rewrite' version '0.0.12'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.1'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.4.1'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-s3' version '0.5.7'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-sampling-filter' version '0.1.3'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-slack' version '0.2.1'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-suppress' version '0.0.6'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-td' version '0.10.26'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.0'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-webhdfs' version '0.4.1'
2015-05-11 10:40:21 +0900 [info]: gem 'fluentd' version '0.12.7'
2015-05-11 10:40:21 +0900 [info]: gem 'fluentd' version '0.10.61'
2015-05-11 10:40:21 +0900 [info]: using configuration file: <ROOT>
  <source>
    type http
    port 24225
  </source>
  <match ad.*>
    type forest
    subtype copy
    <template>
      <store>
        type s3
        aws_key_id AK***
        aws_sec_key ***
        s3_bucket log
        path 20150511-fluent-plugin-forest/
        buffer_type file
        buffer_chunk_limit 8m
        buffer_queue_limit 16
        buffer_path /tmp/20150511-fluent-plugin-forest/s3-buffer-${tag}
        flush_interval 1m
        time_slice_format %Y%m%d/%H/%Y%m%d%H_${tag}
        format json
        include_time_key false
        include_tag_key false
      </store>
      <store>
        type file
        path /tmp/20150511-fluent-plugin-forest/${tag_parts[1]}.log
        time_slice_format %Y%m%d%H
        time_slice_wait 1m
        buffer_type file
        buffer_path /tmp/20150511-fluent-plugin-forest/buffer-${tag}
        buffer_chunk_limit 64m
        compress gzip
      </store>
    </template>
  </match>
</ROOT>
2015-05-11 10:40:21 +0900 [info]: adding match pattern="ad.*" type="forest"
2015-05-11 10:40:21 +0900 [info]: adding source type="http"

I reproduced the situation by stopping any outbound network connections.

$ ping google.jp
PING google.jp (173.194.126.216) 56(84) bytes of data.
^C
--- google.jp ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 1999ms

I sent a log to td-agent.

$ curl -X POST -d 'json={"action":"login","user":2}' http://localhost:24225/ad.s3

And, out_copy was failed to send log to s3 via out_s3 due to open timeout.

2015-05-11 10:47:45 +0900 [error]: failed to configure/start sub output copy: can't call S3 API. Please check your aws_key_id / aws_sec_key or s3_region configuration. error = #<
Net::OpenTimeout: execution expired>
2015-05-11 10:47:45 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-s3-0.5.7/lib/fluent/plugin/out_s3.rb:152:in `rescue in check_apikeys'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-s3-0.5.7/lib/fluent/plugin/out_s3.rb:148:in `check_apikeys'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-s3-0.5.7/lib/fluent/plugin/out_s3.rb:93:in `start'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/out_copy.rb:50:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/out_copy.rb:49:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/out_copy.rb:49:in `start'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:133:in `block in plant'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:128:in `synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:128:in `plant'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:168:in `emit'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/event_router.rb:88:in `emit_stream'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/event_router.rb:79:in `emit'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:167:in `on_request'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:334:in `call'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:334:in `on_message_complete'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:236:in `<<'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:236:in `on_read'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/io.rb:123:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/io.rb:186:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:113:in `run'
2015-05-11 10:47:45 +0900 [error]: Cannot output messages with tag 'ad.s3'

After that, I restored outbound network connection.

$ ping google.jp
PING google.jp (173.194.126.215) 56(84) bytes of data.
64 bytes from nrt04s07-in-f23.1e100.net (173.194.126.215): icmp_seq=1 ttl=58 time=1.84 ms
64 bytes from nrt04s07-in-f23.1e100.net (173.194.126.215): icmp_seq=2 ttl=58 time=1.95 ms
64 bytes from nrt04s07-in-f23.1e100.net (173.194.126.215): icmp_seq=3 ttl=58 time=1.86 ms
^C
--- google.jp ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 1.846/1.889/1.959/0.061 ms

I sent a log to td-agent.

$ curl -X POST -d 'json={"action":"login","user":2}' http://localhost:24225/ad.s3

out_copy was failed to be configure.

2015-05-11 10:49:12 +0900 [error]: failed to configure sub output copy: Other 's3' plugin already use same buffer_path: type = s3, buffer_path = /tmp/20150511-fluent-plugin-forest/s3-buffer-ad.s3
2015-05-11 10:49:12 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/buf_file.rb:97:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:197:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:488:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-mixin-config-placeholders-0.3.0/lib/fluent/mixin/config_placeholders.rb:105:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-s3-0.5.7/lib/fluent/plugin/out_s3.rb:42:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/out_copy.rb:43:in `block in configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/out_copy.rb:34:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/out_copy.rb:34:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:132:in `block in plant'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:128:in `synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:128:in `plant'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:168:in `emit'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/event_router.rb:88:in `emit_stream'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/event_router.rb:79:in `emit'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:167:in `on_request'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:334:in `call'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:334:in `on_message_complete'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:236:in `<<'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:236:in `on_read'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/io.rb:123:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/io.rb:186:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:113:in `run'
2015-05-11 10:49:12 +0900 [error]: Cannot output messages with tag 'ad.s3'

This error was repeated against every log.

2015-05-11 10:59:02 +0900 [error]: failed to configure sub output copy: Other 's3' plugin already use same buffer_path: type = s3, buffer_path = /tmp/20150511-fluent-plugin-forest/s3-buffer-ad.s3
2015-05-11 10:59:02 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/buf_file.rb:97:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:197:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:488:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-mixin-config-placeholders-0.3.0/lib/fluent/mixin/config_placeholders.rb:105:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-s3-0.5.7/lib/fluent/plugin/out_s3.rb:42:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/out_copy.rb:43:in `block in configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/out_copy.rb:34:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/out_copy.rb:34:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:132:in `block in plant'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:128:in `synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:128:in `plant'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:168:in `emit'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/event_router.rb:88:in `emit_stream'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/event_router.rb:79:in `emit'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:167:in `on_request'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:334:in `call'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:334:in `on_message_complete'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:236:in `<<'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:236:in `on_read'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/io.rb:123:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/io.rb:186:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:113:in `run'
2015-05-11 10:59:02 +0900 [error]: Cannot output messages with tag 'ad.s3'

Thank you for reading a long post.

[tagomoris]

Thanks for report!
But this cannot be solved in simple way...