Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fluentd-0.12.33 log_level=trace NoMethodError undefined method `write' for nil:NilClass #191

Closed
brycied00d opened this issue Apr 15, 2017 · 3 comments
Closed

fluentd-0.12.33 log_level=trace NoMethodError undefined method `write' for nil:NilClass #191

brycied00d opened this issue Apr 15, 2017 · 3 comments
Assignees
@repeatedly

Comments

@brycied00d
Copy link

When we upgraded from 0.12.32 to 0.12.33 of fluentd, the fluent-plugin-s3 was no longer able to start, throwing a NoMethodError exception during check_apikeys. Downgrading back to 0.12.32 resolves the issue. Removing log_level trace, or setting it to a lesser level (debug) also resolve the issue.
Evidently, there has been some change or regression by 0.12.33 that makes this plugin fail with log_level set to trace.

I am able to reproduce this on 3 different machines.
Running fluentd with --dry-run does not raise this exception.
Version 0.12.34 still exhibits this issue.

Looking at the changes between 0.12.32 and 0.12.33, it appears to be caused by this issue fluent/fluentd#1478 and commit fluent/fluentd@6fc58dd
As for what aspect of the aws-sdk is attempting to call #write or #<< and why #write is undefined remains a mystery to me.

2017-04-15 12:09:17 -0400 [info]: plugins/logging.rb:51:log: [Aws::S3::Client 0 0.035408 0 retries] list_objects(prefix:"<the prefix>/",bucket:"<our bucket>",encoding_type:"url") NoMethodError undefined method `write' for nil:NilClass

2017-04-15 12:09:17 -0400 [error]: fluent/engine.rb:208:rescue in run: unexpected error error_class=RuntimeError error=#<RuntimeError: can't call S3 API. Please check your aws_key_id / aws_sec_key or s3_region configuration. error = #<NoMethodError: undefined method `write' for nil:NilClass>>
  2017-04-15 12:09:17 -0400 [error]: fluent/supervisor.rb:162:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-s3-0.8.2/lib/fluent/plugin/out_s3.rb:350:in `rescue in check_apikeys'
  2017-04-15 12:09:17 -0400 [error]: fluent/supervisor.rb:162:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-s3-0.8.2/lib/fluent/plugin/out_s3.rb:346:in `check_apikeys'
  2017-04-15 12:09:17 -0400 [error]: fluent/supervisor.rb:162:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-s3-0.8.2/lib/fluent/plugin/out_s3.rb:190:in `start'
  2017-04-15 12:09:17 -0400 [error]: fluent/supervisor.rb:162:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.33/lib/fluent/plugin/out_copy.rb:55:in `block in start'
  2017-04-15 12:09:17 -0400 [error]: fluent/supervisor.rb:162:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.33/lib/fluent/plugin/out_copy.rb:54:in `each'
  2017-04-15 12:09:17 -0400 [error]: fluent/supervisor.rb:162:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.33/lib/fluent/plugin/out_copy.rb:54:in `start'
  2017-04-15 12:09:17 -0400 [error]: fluent/supervisor.rb:162:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-td-monitoring-0.2.2/lib/fluent/plugin/out_td_counter.rb:44:in `start'
  2017-04-15 12:09:17 -0400 [error]: fluent/supervisor.rb:162:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.33/lib/fluent/agent.rb:71:in `block in start'
  2017-04-15 12:09:17 -0400 [error]: fluent/supervisor.rb:162:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.33/lib/fluent/agent.rb:70:in `each'
  2017-04-15 12:09:17 -0400 [error]: fluent/supervisor.rb:162:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.33/lib/fluent/agent.rb:70:in `start'
  2017-04-15 12:09:17 -0400 [error]: fluent/supervisor.rb:162:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.33/lib/fluent/root_agent.rb:108:in `start'
  2017-04-15 12:09:17 -0400 [error]: fluent/supervisor.rb:162:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.33/lib/fluent/engine.rb:237:in `start'
  2017-04-15 12:09:17 -0400 [error]: fluent/supervisor.rb:162:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.33/lib/fluent/engine.rb:187:in `run'
  2017-04-15 12:09:17 -0400 [error]: fluent/supervisor.rb:162:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.33/lib/fluent/supervisor.rb:570:in `run_engine'
  2017-04-15 12:09:17 -0400 [error]: fluent/supervisor.rb:162:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.33/lib/fluent/supervisor.rb:162:in `block in start'
  2017-04-15 12:09:17 -0400 [error]: fluent/supervisor.rb:162:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.33/lib/fluent/supervisor.rb:366:in `call'
  2017-04-15 12:09:17 -0400 [error]: fluent/supervisor.rb:162:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.33/lib/fluent/supervisor.rb:366:in `main_process'
  2017-04-15 12:09:17 -0400 [error]: fluent/supervisor.rb:162:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.33/lib/fluent/supervisor.rb:339:in `block in supervise'
  2017-04-15 12:09:17 -0400 [error]: fluent/supervisor.rb:162:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.33/lib/fluent/supervisor.rb:338:in `fork'
  2017-04-15 12:09:17 -0400 [error]: fluent/supervisor.rb:162:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.33/lib/fluent/supervisor.rb:338:in `supervise'
  2017-04-15 12:09:17 -0400 [error]: fluent/supervisor.rb:162:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.33/lib/fluent/supervisor.rb:156:in `start'
  2017-04-15 12:09:17 -0400 [error]: fluent/supervisor.rb:162:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.33/lib/fluent/command/fluentd.rb:173:in `<top (required)>'
  2017-04-15 12:09:17 -0400 [error]: fluent/supervisor.rb:162:block in start: /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:69:in `require'
  2017-04-15 12:09:17 -0400 [error]: fluent/supervisor.rb:162:block in start: /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:69:in `require'
  2017-04-15 12:09:17 -0400 [error]: fluent/supervisor.rb:162:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.33/bin/fluentd:5:in `<top (required)>'
  2017-04-15 12:09:17 -0400 [error]: fluent/supervisor.rb:162:block in start: /opt/td-agent/embedded/bin/fluentd:23:in `load'
  2017-04-15 12:09:17 -0400 [error]: fluent/supervisor.rb:162:block in start: /opt/td-agent/embedded/bin/fluentd:23:in `<top (required)>'
  2017-04-15 12:09:17 -0400 [error]: fluent/supervisor.rb:162:block in start: /usr/sbin/td-agent:7:in `load'
  2017-04-15 12:09:17 -0400 [error]: fluent/supervisor.rb:162:block in start: /usr/sbin/td-agent:7:in `<main>'
2017-04-15 12:09:17 -0400 [info]: fluent/engine.rb:211:run: shutting down fluentd
2017-04-15 12:09:17 -0400 [info]: fluent/supervisor.rb:354:supervise: process finished code=0

Gem versions:

/opt/td-agent/embedded/bin/gem list | egrep '(fluentd|s3|aws)'
aws-sdk (2.6.42, 2.5.5)
aws-sdk-core (2.6.42, 2.5.5)
aws-sdk-resources (2.6.42, 2.5.5)
aws-sdk-v1 (1.66.0)
aws-sigv4 (1.0.0)
fluent-plugin-s3 (0.8.2, 0.8.0, 0.7.0)
fluentd (0.12.32)
fluentd-ui (0.4.3)

Configuration section:

  <store>
    @type s3
    log_level trace
    aws_key_id xxxxxx
    aws_sec_key xxxxxx
    s3_bucket xxxxxx
    s3_region us-west-2
    use_server_side_encryption AES256
    auto_create_bucket false
    compute_checksums true
    path xxxxxx/
    format json
    include_time_key 
    buffer_path /var/tmp/td-agent/s3/
    time_slice_format year=%Y/month=%m/day=%d/hour=%H/xxxxxx
    s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
    time_slice_wait 10m
    store_as gzip
    utc 
  </store>
@repeatedly repeatedly self-assigned this May 8, 2017
@repeatedly
Copy link
Member

I confirmed it. Now investigating...

@repeatedly
Copy link
Member

Okay. We need to add << to delete list in PluginLogger.

@repeatedly repeatedly mentioned this issue May 8, 2017
Merged
@repeatedly
Copy link
Member

Send a patch to fluentd.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@repeatedly repeatedly

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@repeatedly @brycied00d