Remove usages of ClusterFormationTasks and replace with test clusters

[mark-vieira]

This PR replaces all usages of ClusterFormationTasks with the new test clusters plugin since the former was removed in elastic/elasticsearch#47572.

Please note that this does depend on elastic/elasticsearch#47837 and builds will not start passing until that is merged and a new build-tools JAR is published.

I'm still seeing some test failures for the spark project integration tests. Any ideas @jbaiera?
https://gradle-enterprise.elastic.co/s/3bkbpjyn647ie

They all seem to be the same type of error:

org.elasticsearch.hadoop.rest.EsHadoopRemoteException: no handler found for uri [/defaulturiquerynopush_skiphandledspark-test-null-data-test-2/_doc/1/_source] and method [GET]

[mark-vieira]

Looks like the same "null value" tests are failing in master even with ClusterFormationTasks so I suspect they are unrelated to the refactoring.
https://gradle-enterprise.elastic.co/s/ppurjbiq66tz4/tests/failed

[jbaiera]

When I run ./gradlew clean build locally I'm getting:

FAILURE: Build failed with an exception.

* Where:
Build file '/Users/james.baiera/Documents/source/elasticsearch-hadoop/hive/build.gradle' line: 2

* What went wrong:
A problem occurred evaluating project ':elasticsearch-hadoop-hive'.
> Failed to apply plugin [class 'org.elasticsearch.gradle.DistributionDownloadPlugin']
   > Cannot get property 'bwcVersions' on extra properties extension as it does not exist

I don't have gradle scans set up yet but I can get one up if needed.

[mark-vieira]

Sounds like the changes from elastic/elasticsearch#47837 haven't gone through yet.

Sounds like we might have a bit of a chicken/egg thing here. The stack release builds won't pass due to hadoop failing and hadoop wont' build due to the new build-tools jar not being published. We might have to discuss this with the release folks as I think publishing is an atomic operation, we either publish everything after a successful build or nothing.

[jbaiera]

Discussed off thread: I'll switch to using a snapshot of the current build tools from a local repo location. Should unblock the testing for this.

[jbaiera]

I've been able to verify that things are running well with the new test cluster plugin. LGTM!

[colings86]

@mark-vieira Please remember to label all PRs with change type, area and version labels. This is important for the release notes process as well as letting people looking at the PR know quickly where it is targetting

[colings86]

I've added some labels I think are appropriate but please check and amend as necessary

[mark-vieira]

@colings86 Thanks for doing that. Those labels look good to me.