[CI] Many multijob-unix-compatibility CI runs fail due to absence of Docker

[droberts195]

Many multijob-unix-compatibility builds are failing with the error:

* What went wrong:
Docker (checked [/usr/bin/docker,/usr/local/bin/docker]) is required to run the following tasks: 
 :distribution:docker:buildDockerImage
 :distribution:docker:buildOssDockerImage
you can address this by attending to the reported issue, removing the offending tasks from being executed, or by passing -Dbuild.docker=false

Some examples are:

• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=oraclelinux-7/27/console
• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=centos-6/27/console
• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=oraclelinux-6/27/console
• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=amazon/27/console

Those same 4 OSs fail like this every 8 hours. Either Docker needs to be added to the OS images used by CI or maybe the multijob-unix-compatibility jobs should explicitly exclude building the Docker distribution?

[elasticmachine]

Pinging @elastic/es-core-infra

[droberts195]

This also affects 7.x, for example https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+multijob-unix-compatibility/os=oraclelinux-7/28/console

But not 7.2 or earlier.

[alpar-t]

Thanks for opening the issue @droberts195 this works as designed, we should bail when we try to build the docker images and no docker is available ( we have technical reasons for the not having it on the workers that don't have it btw ).
The test tasks that depend on docker are auto disabled when docker is not present, but not the image build itself. So check doesn't trigger the image build, but we have build in CI which depends on assemble and thus triggers the image builds.

The fix for this one is to switch to using check instead of build in CI for the support matrix specific tests.

[bizybot]

Reopening this issue based on the discussions in core infra channel.
Looks like we need to do something more here, as we think the fixture does not start and then causing failures in the cluster start. This needs more investigation.

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=oraclelinux-7/37/console

> Configure project :test:fixtures:krb5kdc-fixture
Tests for :test:fixtures:krb5kdc-fixture require docker-compose at /usr/local/bin/docker-compose or /usr/bin/docker-compose but none could be found so these will be skipped

and then

* What went wrong:
Execution failed for task ':x-pack:qa:kerberos-tests:integTestCluster#wait'.
> Failed to locate seed node transport file [/var/lib/jenkins/workspace/elastic+elasticsearch+master+multijob-unix-compatibility/os/oraclelinux-7/x-pack/qa/kerberos-tests/build/cluster/integTestCluster node0/elasticsearch-8.0.0-SNAPSHOT/logs/transport.ports]: timed out waiting for it to be created after 40 seconds

[alpar-t]

The problem here is that the build fixture task is explicitly enabled, overriding the configuration done in the plugin. The plugin disables it because no docker is available but the build scripts turns it back on.

[alpar-t]

Since this is a simple one liner change that's not covered by PR checks I pushed 83182d6 directly to fix it.