How nightly fuzzer tests work

[rui-mo]

There were some issues on spark functions discovered from nightly fuzzer tests https://github.com/facebookincubator/velox/issues?q=is%3Aopen+is%3Aissue+label%3Aspark-functions. As we have some idle test clusters, is it possible for us to run them in our nightly tests to pre-discover more issues?
We notice fuzzer tests are triggered for each pull request, but seems some issues are only reported in nightly fuzzer tests. If possible, we would like to know how these tests are triggered, and what are the factors should be considered to let the fuzzer tests find more issues.
Thanks for your advice!

[rui-mo]

cc @zhouyuan

[zhouyuan]

Thank you Rui, I tried to enable the Expression and Agg fuzz tests in Gluten Jenkins env, each with 10 minutes. However the job usually will fail with unexpected results. This seems not expected so I disabled those, and only check the C++ unit tests for now. It would be great if there is some guide from Velox developers on how to run those fuzz tests, and how to collect/report the results/issues.

CC: @mbasmanova

Thanks, -yuan

[mbasmanova]

@zhouyuan

However the job usually will fail with unexpected results.

Can you share the details of these failures? These could indicate bugs in functions or expression evaluation engine.

[mbasmanova]

@zhouyuan Here is documentation about the fuzzers: https://facebookincubator.github.io/velox/develop/testing/fuzzer.html#how-to-run

[zhouyuan]

@mbasmanova
Thank you, I'm curious how is the fuzz test results on your env, will it always successful to run or will it also fail frequently?
I'm following Velox circle CI job to do the setup, with 10 minutes run:

  # agg fuzzer
 #           mkdir -p /tmp/aggregate_fuzzer_repro/
 #           rm -rfv /tmp/aggregate_fuzzer_repro/*
 #           chmod -R 777 /tmp/aggregate_fuzzer_repro
 #          velox/exec/tests/velox_aggregation_fuzzer_test \
 #               --seed 11245 \
 #               --duration_sec 360 \
 #               --logtostderr=1 \
 #               --minloglevel=0 \
 #               --repro_persist_path=/tmp/aggregate_fuzzer_repro \
 #           && echo -e "\n\nAggregation fuzzer run finished successfully."

Here's one failure log I captured a few days ago. I will try to re-enable the checks here.

Unexpected results
E1129 02:43:04.849676 3180739 Exceptions.h:69] Line: ../../velox/exec/tests/utils/AggregationFuzzer.cpp:1289, Function:testPlan, Expression: assertEqualResults({expected.result}, {actual.result}) Logically equivalent plans produced different results, Source: RUNTIME, ErrorCode: INVALID_STATE
I1129 02:43:04.932868 3180739 AggregationFuzzer.cpp:646] Persisted aggregation plans to /tmp/aggregate_fuzzer_repro/velox_aggregationVerifier_b7ruoM/plan_nodes
terminate called after throwing an instance of 'facebook::velox::VeloxRuntimeError'
  what():  Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Logically equivalent plans produced different results
Retriable: False
Expression: assertEqualResults({expected.result}, {actual.result})
Function: testPlan
File: ../../velox/exec/tests/utils/AggregationFuzzer.cpp
Line: 1289
Stack trace:
# 0  facebook::velox::VeloxException::VeloxException(char const*, unsigned long, char const*, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, bool, facebook::velox::VeloxException::Type, std::basic_string_view<char, std::char_traits<char> >)
# 1  void facebook::velox::detail::veloxCheckFail<facebook::velox::VeloxRuntimeError, char const*>(facebook::velox::detail::VeloxCheckFailArgs const&, char const*)
# 2  facebook::velox::exec::test::(anonymous namespace)::AggregationFuzzer::testPlan(facebook::velox::exec::test::(anonymous namespace)::AggregationFuzzer::PlanWithSplits const&, bool, bool, bool, std::vector<std::shared_ptr<facebook::velox::exec::test::ResultVerifier>, std::allocator<std::shared_ptr<facebook::velox::exec::test::ResultVerifier> > > const&, facebook::velox::test::ResultOrError const&, int)
# 3  facebook::velox::exec::test::(anonymous namespace)::AggregationFuzzer::testPlans(std::vector<facebook::velox::exec::test::(anonymous namespace)::AggregationFuzzer::PlanWithSplits, std::allocator<facebook::velox::exec::test::(anonymous namespace)::AggregationFuzzer::PlanWithSplits> > const&, bool, std::vector<std::shared_ptr<facebook::velox::exec::test::ResultVerifier>, std::allocator<std::shared_ptr<facebook::velox::exec::test::ResultVerifier> > > const&, facebook::velox::test::ResultOrError const&, int)
# 4  facebook::velox::exec::test::(anonymous namespace)::AggregationFuzzer::verifyAggregation(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::shared_ptr<facebook::velox::RowVector>, std::allocator<std::shared_ptr<facebook::velox::RowVector> > > const&, bool, std::vector<std::shared_ptr<facebook::velox::exec::test::ResultVerifier>, std::allocator<std::shared_ptr<facebook::velox::exec::test::ResultVerifier> > > const&)
# 5  facebook::velox::exec::test::aggregateFuzzer(std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<std::shared_ptr<facebook::velox::exec::AggregateFunctionSignature>, std::allocator<std::shared_ptr<facebook::velox::exec::AggregateFunctionSignature> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<std::shared_ptr<facebook::velox::exec::AggregateFunctionSignature>, std::allocator<std::shared_ptr<facebook::velox::exec::AggregateFunctionSignature> > > > > >, unsigned long, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<facebook::velox::exec::test::ResultVerifier>, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<facebook::velox::exec::test::ResultVerifier> > > > const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<facebook::velox::exec::test::InputGenerator>, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<facebook::velox::exec::test::InputGenerator> > > > const&, facebook::velox::VectorFuzzer::Options::TimestampPrecision, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__
# 6  facebook::velox::exec::test::AggregationFuzzerRunner::run(unsigned long, std::unique_ptr<facebook::velox::exec::test::ReferenceQueryRunner, std::default_delete<facebook::velox::exec::test::ReferenceQueryRunner> >, facebook::velox::exec::test::AggregationFuzzerRunner::Options const&)
# 7  main
# 8  __libc_start_main
# 9  _start

[mbasmanova]

@zhouyuan

I'm curious how is the fuzz test results on your env, will it always successful to run or will it also fail frequently?

It does fail quite frequently. We create fuzzer-found issues, investigate and fix. See https://github.com/facebookincubator/velox/issues?q=is%3Aissue+is%3Aopen+label%3Afuzzer-found

Reason: Logically equivalent plans produced different results

This suggests a bug in either aggregate function or Aggregation operator. Need investigating. The logs should include the plan so you can see which function was tested. Also, see https://facebookincubator.github.io/velox/develop/testing/fuzzer.html#how-to-run for how to reproduce fuzzer failures.

[zhouyuan]

@mbasmanova much appreciate for your insights! Will follow your advices, try to re-enable the fuzz tests in our Jenkins and report back the issues/bugs if identified

thanks, -yuan