StatsThreadLocalStoreTestNoFixture.MemoryWithoutTlsFakeSymbolTable flaky?

[alyssawilk]

Josh, could you take a look? I don't think this failure is related to the PR it failed on ( #8281) so I think it's a random flake.

https://circleci.com/gh/envoyproxy/envoy/278531?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

[ RUN ] StatsThreadLocalStoreTestNoFixture.MemoryWithoutTlsFakeSymbolTable
test/common/stats/thread_local_store_test.cc:898: Failure
Expected equality of these values:
memory_test.consumedBytes()
Which is: 15268288
15268336
Stack trace:
0x528d66: Envoy::Stats::StatsThreadLocalStoreTestNoFixture_MemoryWithoutTlsFakeSymbolTable_Test::TestBody()
0xa94ea4: testing::internal::HandleExceptionsInMethodIfSupported<>()
0xa94dcc: testing::Test::Run()
0xa95e1f: testing::TestInfo::Run()
... Google Test internal frames ...

[ FAILED ] StatsThreadLocalStoreTestNoFixture.MemoryWithoutTlsFakeSymbolTable (787 ms)

[zuercher]

Saw this on a docs-only PR as well (#8285).

[jmarantz]

It had been stable I think since June. I'd love to know what possible sources of non-determinism there might be that could affect this fairly narrow test.

I'm imagining it might be something else recent that got checked in.

[jmarantz]

and in fact the unexpected behavior is that it consumed 336-288 = 48 fewer bytes than the golden value.

[jmarantz]

I have a theory; there are actually real threads happening in this test, although they have nothing to do with the test.

TestRunner::RunTests() instantiates ProcessWide which runs grpc_init() which ultimately calls pthread_create(). In fact it create 3 threads. I suspect there's some allocation going on in there in parallel to the test body, adding some (rare) flakiness.

[jmarantz]

Possible cause: #8196 -- maybe that new version of gRPC made threads active in the background, which is obviously harmful to memory tests.

[jmarantz]

Running

bazel test --cache_test_results=no --compilation_mode=opt \
  --test_env=ENVOY_MEMORY_TEST_EXACT=true \ 
  --cache_test_results=no //test/common/stats:thread_local_store_test \
  --runs_per_test=100

fails 7/100 times. If I comment out the calls to grpc_init() and grpc_shutdown() in source/exe/process_wide.cc, I see 0 failures, even with --runs_per_test=1000

[jmarantz]

@htuch WDYT about calling grpc_init/grpc_shutdown from the ctor/dtor of GoogleAsyncClientImpl ?

That would keep it more constrained and avoid having it run threads over the top of unrelated tests?