[Flaky Test] Testintegration getting timed out

[samiura]

Description

Flaky test:
https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/5966235030/job/16185463868?pr=26086#step:5:609
https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/5966235030/job/16186427803

FAIL	github.com/open-telemetry/opentelemetry-collector-contrib/receiver/flinkmetricsreceiver	360.040s
?   	github.com/open-telemetry/opentelemetry-collector-contrib/receiver/flinkmetricsreceiver/internal/mocks	[no test files]
?   	github.com/open-telemetry/opentelemetry-collector-contrib/receiver/flinkmetricsreceiver/internal/models	[no test files]
ok  	github.com/open-telemetry/opentelemetry-collector-contrib/receiver/flinkmetricsreceiver/internal/metadata	0.044s [no tests to run]
FAIL
make[2]: *** [../../Makefile.Common:116: mod-integration-test] Error 1
make[1]: *** [Makefile:174: receiver/flinkmetricsreceiver] Error 2
make[2]: Leaving directory '/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/flinkmetricsreceiver'
make: *** [Makefile:76: integration-test] Error 2
make[1]: Leaving directory '/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib'
Error: Process completed with exit code 2.

2023/08/24 16:57:17 🐳 Creating container for image flink:1.17.0
2023/08/24 16:57:18 🐳 Creating container for image flink:1.17.0
2023/08/24 16:57:19 🐳 Creating container for image flink:1.17.0
2023/08/24 16:57:20 🐳 Creating container for image flink:1.17.0
2023/08/24 16:57:21 🐳 Creating container for image flink:1.17.0
2023/08/24 16:57:22 🐳 Creating container for image flink:1.17.0
panic: test timed out after 6m0s
running tests:
	TestIntegration (6m0s)

goroutine 8485 [running]:
testing.(*M).startAlarm.func1()
	/opt/hostedtoolcache/go/1.20.7/x64/src/testing/testing.go:2241 +0x219
created by time.goFunc
	/opt/hostedtoolcache/go/1.20.7/x64/src/time/sleep.go:176 +0x48

[github-actions]

Pinging code owners for receiver/flinkmetrics: @JonathanWamsley @djaglowski. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[nslaughter]

This issue came up in workflows for #24774.

In one case, noted what I thought were a few key points for summary...

• containers start and then way later
• after 3:30 trying to create a container
• testcontainers starts unwinding on the 6 minute timeout

Also want to throw out there that on a follow-up run trying to work through this I saw the same pattern with kafkametricsreceiver. To me that meant it might be JVM related or just big memory use related, but not convinced that it's otherwise related to flinkmetrics receiver in particular.

Hoping to come up with some more concrete causes and changes from there.

[djaglowski]

Thanks @nslaughter.

kafkametricsreceiver and flinkmetrics tests both rely on scraperinttest. This package was created in response to instability problems in complex tests, primarily those which depend on containers. It helps by adding retry logic around failure points that are not directly the purpose of the test (e.g. container failing to start is part of test setup, not test validation, so retry). Obviously scraperinttest is not perfect but it's been a major improvement upon what we formerly had.

If you are looking into this, it's likely a good idea to look for a solution that can be built into the package, so that this type of failure can be prevented broadly.

[dmitryax]

Run into this couple times in https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/6003150441/job/16281731431?pr=26125

I think we should skip the test for now

[dmitryax]

Submitted #26194

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/flinkmetrics: @JonathanWamsley @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[hughesjj]

I'll note that I've had issues with NamedContainer and NetworkRequest and am diving into it

[djaglowski]

I've proposed removing this test in #30218

[djaglowski]

Closed by #30218