Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ignore non buffer chunk files when resumed #1124

Merged
merged 4 commits into from
Jul 28, 2016

Conversation

tagomoris
Copy link
Member

This fixes #1098.

buf_file in v0.14 loads all files which matches with glob for buffer file path as queued. That behavior is to read files as much as possible when upgrading Fluentd.
But there is a bug to read files as queued, which is written by out_file in the same directory with buffer chunk files.

This change is to fix not to load files which doesn't have a part like (b|q)[0-9a-f]+, which shows that file is a buffer chunk.

note about #1098

#1098 is a result of combination of this bug and append true, which does read data from buffer chunk file (this file) and also does write these data to output file (this file) - it is infinite loop of read & write.

The minimum steps to reproduce it is:

  1. execute command: echo "foobar" > temperature.\ruby -e 'puts Time.now.strftime("%Y%m%d")'.log
  2. execute fluentd with configuration below:
  3. use ctrl-z to stop Fluentd process without shutdown sequence (shutdown calls flushing)
# no <source> needed
<match demo.temperature>
  @type file
  path temperature
  flush_interval 1s
  append true
</match>

@repeatedly
Copy link
Member

Test failed by introducing unknown

@tagomoris
Copy link
Member Author

@repeatedly Oops. I fixed that test (and added another data for pattern newly tested).

@repeatedly repeatedly mentioned this pull request Jul 26, 2016
@tagomoris tagomoris merged commit 57660ee into master Jul 28, 2016
@tagomoris tagomoris deleted the ignore-non-buffer-chunk-files branch July 28, 2016 03:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Fluentd crashes
2 participants