Fix flaky docker celery tests

[owais]

Celery docker tests randomly fail with the following output:

=================================== FAILURES ===================================
_________________________ test_custom_tracer_provider __________________________

celery_app = <Celery celery.tests at 0x7f88b2d3cbe0>
memory_exporter = <opentelemetry.sdk.trace.export.in_memory_span_exporter.InMemorySpanExporter object at 0x7f88b8305790>

    def test_custom_tracer_provider(celery_app, memory_exporter):
        @celery_app.task
        def fn_task():
            return 42
    
        resource = resources.Resource.create({})
        tracer_provider = TracerProvider(resource=resource)
        span_processor = export.SimpleSpanProcessor(memory_exporter)
        tracer_provider.add_span_processor(span_processor)
    
        trace_api.set_tracer_provider(tracer_provider)
    
        CeleryInstrumentor().uninstrument()
        CeleryInstrumentor().instrument(tracer_provider=tracer_provider)
    
        fn_task.delay()
    
        spans_list = memory_exporter.get_finished_spans()
>       assert len(spans_list) == 1
E       assert 2 == 1
E         +2
E         -1

celery/test_celery_functional.py:579: AssertionError

We need to investigate this and fix th test so it always passes.

[owais]

We didn't fix the problem completely. The fix did reduce the chance of this failing by a huge factor it seems but I did see it fail at least once (compared to ~50% of builds) since this was merged.

[lzchen]

https://docs.pytest.org/en/6.2.x/flaky.html