Move x-content implementation to a separate classloader

[rjernst]

x-content is a relatively well contained abstraction for parsing and generating json-like content. It is backed by Jackson, but consumers throughout Elasticsearch use the x-content interfaces and don't generally care about how Jackson is used. However, because x-content is a dependency of server, Jackson is also a dependency of server. Since all server dependencies must be exactly matched by plugins (and ES modules), upgrading Jackson is difficult since it must be done in lock step across the codebase.

This change isolates the Jackson implementation of x-content parsers and generators to a separate classloader. The code is loaded dynamically upon accessing any x-content functionality.

The x-content implementation is embedded inside the x-content jar, as a hidden set of resource files. These are loaded through a special classloader created to initialize the XContentProvider through service loader. One caveat to this approach is that IDEs will no longer trigger building the x-content implementation when it changes. However, running any test from the command line, or running a full Build in IntelliJ will trigger the directory to be built.

Most of the changes here are logistical, just moving methods and classes across to the implementation project. One common change to code using x-content is exception handling. This code relied on knowledge of Jackson exception types. That code now uses special exception types from x-content. Another change needed in a few places was now including jackson-core in plugins, since server no longer provides it.

The hope is that this change can be used as a model for future similar isolation of server dependencies.

[rjernst]

@mark-vieira @nik9000 I think this is ready for another round of review. The implementation has changed so that the implementation jars are now exploded and embedded in the x-content jar in a hidden location. This alleviates the need to add anything additional to the classpath, or have any new artifacts published.

@nik9000 Thanks for looking at Eclipse. It's about exactly what we anticipated. I'm not sure we should do anything. The hack you came up with is quite fragile and long term Eclipse is going to have more of these problems as we move to modules. An alternative hack would be to put the genereated resources directory on the classpath manually within the project. It's not much worse than what IntelliJ is doing automatically by reading the gradle model, but it still requires running any cli command that will trigger building the directory. I know it isn't what you want to hear, but I think the benefit of this change outweighs complexity added for the few developers using Eclipse.

[mark-vieira]

This works well and LGTM other than a few comments. Thanks for iterating on this. I think this implementation is definitely better and more sustainable going forward.

[mark-vieira]

Do we still need to publish this given we are effectively embedding it in the x-content jar?

[mark-vieira]

We should just set this on the project convention. That is, add archivesBaseName = "x-content-impl" to the root of the build script

[mark-vieira]

I think we want to use the elasticsearch.java plugin here since build implicitly applies the publish plugin which I don't believe we want.

[mark-vieira]

Why are we disabling this?

[mark-vieira]

I feel like this should have a more descriptive name. Also confusing given we'll have a MANIFEST.TXT and MANIFEST.MF in the JAR.

[rjernst]

I actually named it manifest since it sort of parallels the jar manifest. I'm not tied to the name, though. I just don't have a better one. Do you have a suggestion?

[mark-vieira]

Something with "embedded" in the name?

[rjernst]

I pushed "LISTING.TXT", wdyt?

[mark-vieira]

Works for me.

[breskeby]

We should apply a custom attribute for the providerImpl configuration and on the registered transformation to
ensure we only apply the transformation for dependencies resolved from the providerImpl configuration.

E.g.

configurations {
  providerImpl {
    canBeConsumed = false
    attributes.attribute(isProviderImpl, Boolean.TRUE)
  }
}

...
...

dependencies {
  registerTransform(
    UnzipTransform.class, transformSpec -> {
    transformSpec.getFrom()
            .attribute(ArtifactTypeDefinition.ARTIFACT_TYPE_ATTRIBUTE, ArtifactTypeDefinition.JAR_TYPE)
            .attribute(isProviderImpl, true)

    transformSpec.getTo()
            .attribute(ArtifactTypeDefinition.ARTIFACT_TYPE_ATTRIBUTE, ArtifactTypeDefinition.DIRECTORY_TYPE)
            .attribute(isProviderImpl, true)

    transformSpec.parameters(parameters -> {
      parameters.includeArtifactName.set(true)
    })

  })

...
}

This still results in the original raised circular dependency issue but with a clearer output:

Circular dependency between the following tasks:
:libs:elasticsearch-x-content:classes
\--- :libs:elasticsearch-x-content:generateProviderImpl
     +--- :libs:elasticsearch-x-content:generateProviderManifest
     |    \--- :libs:elasticsearch-x-content:impl:jar
     |         \--- :libs:elasticsearch-x-content:impl:classes
     |              \--- :libs:elasticsearch-x-content:impl:compileJava
     |                   \--- :libs:elasticsearch-x-content:jar
     |                        +--- :libs:elasticsearch-x-content:classes (*)
     |                        \--- :libs:elasticsearch-x-content:generateProviderImpl (*)

The root problem keeps being that running classes triggers the generateProviderImpl as we define
sourceSets.main.output.dir(generateProviderImpl).
This requires impl to be build which causes the circular dependency as impl has a compile dependency on x-content itself.

[rjernst]

@breskeby The circular issue was on a different branch, just one that now includes the x-content separation from this PR. This PR builds find and has passed CI. I pushed 5672c22, let me know if that matches what you were thinking.

[breskeby]

thanks @rjernst. looks good to me

[breskeby]

lgtm