Smarter copying of the rest specs and tests

[jakelandis]

This PR addresses the unnecessary copying of the rest specs and allows
for better semantics for which specs and tests are copied. By default
the rest specs will get copied if the project applies
elasticsearch.standalone-rest-test or esplugin and the project
has rest tests or you configure the custom extension restResources.

This PR also removes the need for dozens of places where the x-pack
specs were copied by supporting copying of the x-pack rest specs too.

The plugin/task introduced here can also copy the rest tests to the
local project through a similar configuration.

The new plugin/task allows a user to minimize the surface area of
which rest specs are copied. Per project can be configured to include
only a subset of the specs (or tests). Configuring a project to only
copy the specs when actually needed should help with build cache hit
rates since we can better define what is actually in use.
However, project level optimizations for build cache hit rates are
not included with this PR.

Also, with this PR you can no longer use the includePackaged flag on
integTest task.

The following items are included in this PR:

• new plugin: elasticsearch.rest-resources
• new tasks: CopyRestApiTask and CopyRestTestsTask - performs the copy
• new extension 'restResources'

restResources {
  restApi {
    includeCore 'foo' , 'bar' //will include the core specs that start with foo and bar
    includeXpack 'baz' //will include x-pack specs that start with baz
  }
  restTests {
    includeCore 'foo', 'bar' //will include the core tests that start with foo and bar
    includeXpack 'baz' //will include the x-pack tests that start with baz
  }
}

[jakelandis]

@mark-vieira - would you mind taking a look at the direction of this PR. If you agree with the direction, I will add some tests, update some docs, and do some minor clean up.

Of note that I am unsure about

• using a custom extension to configure how the plugin behaves, which requires deferring until projects.evaluated.
• the fact that you can apply the plugin and it can do nothing (if there are no tests and you don't set alwaysCopySpec to true)
• dropping some support around executing the tests from the jar (as far as i can tell we don't ever do it, and i can't think of a reason why a plugin dev would (however, they still could since there are no changes to the packaging))

However, I think this provides a nice compromise to keeping it to just work (for plugin devs) and separating the concerns.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Build)

[jakelandis]

@elasticmachine update branch

[jakelandis]

Hold off on reviewing ... I am re-visiting lazy properties and trying pushing some of the conditional logic out of the plugin and into a task at execution time. The basic idea is the same (configuration via a custom extension applied via a plugin by default and conditionally copying a subset of files), but I think lazy properties is what this PR needs to push some of the work out of the configuration phase and avoid the projects.evaluated hack.

[mark-vieira]

Hold off on reviewing ... I am re-visiting lazy properties and trying pushing some of the conditional logic out of the plugin and into a task at execution time.

I was just going to say this. For what you want the preferred pattern is to have the extension configure lazy properties that are "wired" up to the appropriate tasks. Essentially, since we don't need to know any of this stuff until those tasks actually execute, we are free to mutate those properties up until the point of execution.

[mark-vieira]

I'm wondering if everywhere we see REST we should actually use YAML. If you are not using are YAML testing framework, is anything of this stuff useful? That is, for the Java-based REST tests, do we need any of this stuff? If not, let's be more specific since we already overload the phrase "rest tests" to an enormous degree, let's be as specific as possible with naming conventions here.

[jakelandis]

I ended up with two different extensions, copyRestApiSpecs and copyYamlTests. I am abit adverse to referring to the RestApiSpec with YAML, since they are actually defined in JSON, and have purposes outside of the YAML RestTests. But I agree rest tests are overloaded and am flexible on the naming.

[mark-vieira]

I am abit adverse to referring to the RestApiSpec with YAML, since they are actually defined in JSON, and have purposes outside of the YAML RestTests.

What else to we expect build consumers to need with these spec outside the scope of YAML testing? Are there instances of projects consuming rest specs but not doing YAML testing?

[jakelandis]

All of the client's (except HLRC) use the spec to drive code generation or language specific testing, however not sure if they integrate with Gradle. The HLRC tests does some validation outside of the RestTests framework, and docs also use them for testing (but in a slightly different way). There could also be additional future usages of the spec outside the testing framework.

[jakelandis]

@mark-vieira thanks for you help so far! I have updated the PR to use lazy properties and a proper task that is configured via the plugin. I still have some more testing (manual and writing unit) to do, but I think this iteration is more Gradle correct.

[mark-vieira]

I've left some bigger feedback items and questions. There are also a number of nit picky polish items that I think would be simpler for me to just do myself, either later, or in another PR.

[mark-vieira]

This should live in the plugin not in the task.

[jakelandis]

Actually, thinking about this a bit more, I think this needs to stay in the task for a couple reasons:

1. This method checks against the source dir for tests committed to GH, but also checks the output directory for tests copied over by prior gradle tasks. This is why the spec task depends on the test task. Since this evaluation is done in the task at execution time, I believe this is a reliable way to ensure that any source or copied tests exist.

2. We can not actually skip the creation of the task based on the output of this. The spec copy task needs to execute if the project has no rest tests, but also you requested to include some tests, and the IIUC the extension property isn't guaranteed to be resolved at configuration time. e.g. since we can not read the property at configuration time we will always need to create the task in case there are no tests at all, but you need the spec (docs and HLRC are examples of this).

[ywelsch]

The backport PR seems to have been merged. I'm therefore removing the backport pending label here. Please shout if this is incorrect