Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: Fix random test failures at test/plugin/test_output.rb#L926 #1890

Merged

Conversation

fujimotos
Copy link
Member

This patch fixes the test failures around invoke_slow_flush_log_thread_test() which
have been occurring at Travis CI.

For actual failing test runs, see job#4160.1, job#4169.4 and job#4154.5

Problem

I can reproduce this failure reliably by inserting sleep 3 into test_output.rb:

def invoke_slow_flush_log_threshold_test(i)
  ...
  i.start
  i.after_start

  sleep 3

  t = event_time()
  i.emit_events('tag', Fluent::ArrayEventStream.new([ [t, {"key" => "value1"}],
                                              [t, {"key" => "value2"}] ]))
  i.force_flush

  waiting(4) { Thread.pass until i.test_finished? }

So it is a multi-threading bug. In particular, if the buffer-flushing thread interrupts the
test execution at the code point, it will accidentally turn on the @test_finished flag
and breaks the precondition for the test case. This is the exact cause of the test failure.

Solution

This patch resolves this issue by setting a sufficiently long value to flush_interval
(30 sec) so that it won't mess with the test execution.

This is basically the same bug that has been fixed by 29d2f7e.

What's wrong with those test cases is that they naively assumes
background threads (for flushing buffers) never interrupts the test
execution. Any sort of delay in test execution would break the
assumption and result in an undefined behaviour.

This patch fixes this test failure by taking additional care to
suppress background flush attempts.
@repeatedly repeatedly merged commit 284a65e into fluent:master Mar 15, 2018
@repeatedly
Copy link
Member

Thanks!

@fujimotos fujimotos deleted the sf/fix-random-failure-in-test-output branch July 5, 2018 02:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants