Integration test for databroker quite unstable

[SebastianSchildt]

CI often fails integration tests. Restarting failed runs solves it most of the time.

https://github.com/eclipse/kuksa.val/actions/runs/4817567878/jobs/8612176448

It may be a timing issue, maybe it can be made more robust? This is typical output

Status: Downloaded newer image for ttl.sh/kuksa.val/kuksa-databroker-3fe7cccf5dc0c3ad7a2f18483272077143dd4c86:latest
============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.3.1, pluggy-1.0.0 -- /home/runner/work/kuksa.val/kuksa.val/.venv/bin/python3
cachedir: .pytest_cache
rootdir: /home/runner/work/kuksa.val/kuksa.val
plugins: asyncio-0.21.0, ordering-0.6
asyncio: mode=strict
collecting ... collected 3 items

kuksa_databroker/integration_test/test_databroker.py::test_databroker_connection FAILED [ 33%]
kuksa_databroker/integration_test/test_databroker.py::test_feeder_metadata_registered PASSED [ 66%]
kuksa_databroker/integration_test/test_databroker.py::test_events PASSED [100%]

=================================== FAILURES ===================================
__________________________ test_databroker_connection __________________________

    @pytest.mark.asyncio
    async def test_databroker_connection() -> None:
        logger.info("Connecting to VehicleDataBroker {}".format(DATABROKER_ADDRESS))
        helper = Databroker(DATABROKER_ADDRESS)
>       await helper.get_metadata()

kuksa_databroker/integration_test/test_databroker.py:42: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
kuksa_databroker/integration_test/helper.py:100: in get_metadata
    response = await self._broker_stub.GetMetadata(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_AioCall of RPC that terminated with:
	status = failed to connect to all addresses; last error: UNAVAILABLE: ipv4:127...NAVAILABLE: ipv4:127.0.0.1:55555: Socket closed {created_time:"2023-04-28T17:00:44.185987833+00:00", grpc_status:14}"
>

    def __await__(self) -> ResponseType:
        """Wait till the ongoing RPC request finishes."""
        try:
            response = yield from self._call_response
        except asyncio.CancelledError:
            # Even if we caught all other CancelledError, there is still
            # this corner case. If the application cancels immediately after
            # the Call object is created, we will observe this
            # `CancelledError`.
            if not self.cancelled():
                self.cancel()
            raise
    
        # NOTE(lidiz) If we raise RpcError in the task, and users doesn't
        # 'await' on it. AsyncIO will log 'Task exception was never retrieved'.
        # Instead, if we move the exception raising here, the spam stops.
        # Unfortunately, there can only be one 'yield from' in '__await__'. So,
        # we need to access the private instance variable.
        if response is cygrpc.EOF:
            if self._cython_call.is_locally_cancelled():
                raise asyncio.CancelledError()
            else:
>               raise _create_rpc_error(self._cython_call._initial_metadata,
                                        self._cython_call._status)
E               grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
E               	status = StatusCode.UNAVAILABLE
E               	details = "failed to connect to all addresses; last error: UNAVAILABLE: ipv4:127.0.0.1:55555: Socket closed"
E               	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNAVAILABLE: ipv4:127.0.0.1:55555: Socket closed {created_time:"2023-04-28T17:00:44.185987833+00:00", grpc_status:14}"
E               >

.venv/lib/python3.10/site-packages/grpc/aio/_call.py:290: AioRpcError
=========================== short test summary info ============================
FAILED kuksa_databroker/integration_test/test_databroker.py::test_databroker_connection - grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNAVAILABLE: ipv4:127.0.0.1:55555: Socket closed"
	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNAVAILABLE: ipv4:127.0.0.1:55555: Socket closed {created_time:"2023-04-28T17:00:44.185987833+00:00", grpc_status:14}"
>
========================= 1 failed, 2 passed in 4.05s ==========================
Stopping databroker container
0ab81235a5d0b122602cc7030740ab9095d8135f1aa5be4eb855ab54b321f5dc
Error: Process completed with exit code 1.

[SebastianSchildt]

Another failed run https://github.com/eclipse/kuksa.val/actions/runs/4926284003/jobs/8804623073

[SebastianSchildt]

maybe a timeout after

kuksa.val/kuksa_databroker/integration_test/run.sh

Line 30 in 2656eea

docker run -d -v ${VSS_DATA_DIR}:/data -p 55555:55555 --rm ${DATABROKER_IMAGE} --metadata data/vss-core/vss_release_2.1.json

might help (we can check the container is running, but does not mean db is really up yet). Or the test itself needs to retry a couple of times, have not checked