[CI] minio fails due to port clashes

[costin]

Looks like minio is failing again due to clashes with already opened port.
This seems to have been already handled through
#32701 and #32208 but it still occurred in master.

For example:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/1411/console

Unable to start the server: Port :36076 is already in use.

It doesn't seem like the port is hardcoded:

elasticsearch/plugins/repository-s3/build.gradle

Line 205 in 7394892

ServerSocket serverSocket = new ServerSocket(0, 1, InetAddress.getByName(minioAddress))

Maybe it's a race condition when gradle runs in parallel?

[elasticmachine]

Pinging @elastic/es-core-infra

[alpar-t]

We don't yet run this part in parallel.
It's odd that we hit a race condition here, but the code as it's written would theoretically allow for it.
We could use the testfixtures plugin https://github.com/elastic/elasticsearch/blob/7394892b4c36f44d5addb2abd2dbca7453cb2d79/buildSrc/src/main/java/org/elasticsearch/gradle/testfixtures/TestFixturesPlugin.java
to start minio in a docker container. That would take care of port allocation and mappings.

Right now, that would imply creating a minio-fixture simmilar to smb-fixture which already uses the plugin, but we could also update the plugin to allow defining and using the fixture within the same Gradle project.

[costin]

Looks like another failure on the 6.x line:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-unix-compatibility/os=oraclelinux/195/console

 Port 60925 for Minio process is already taken
07:22:48 java.net.BindException: Address already in use (Bind failed)

[original-brownbear]

@atorok went for a slightly less exciting solution for now in #37747 for the time being. I'd be all for using Docker here though in general :)

[original-brownbear]

Reopening this as it's still failing with the current solution, I'm on it fixing it for good in #37783 now though.

[matriv]

New occurrence for :plugins:repository-s3:startMinio: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.5+release-tests/386/console

> Task :plugins:repository-s3:startMinio FAILED
Task ':plugins:repository-s3:startMinio' is not up-to-date because:
  Task has not declared any outputs despite executing actions.
�7ERROR �8�[6CUnable to start the server: Port :60920 is already in use.
�7      �8�[6C> Please ensure no other program uses the same address/port.
:plugins:repository-s3:startMinio (Thread[Task worker for ':',5,main]) completed. Took 0.034 secs.

[original-brownbear]

Fix is incoming for 6.x, master is fine already, will back port this today or tomorrow :)

[original-brownbear]

Fixed in 6.x via 4b6678f