[BUG] Repository Plugins Broken after XContent Base Clase Refactor

[dgilling]

Describe the bug
Looks like Refactor XContent base classes from xcontent to core library broke the installation of the repository plugins.

To Reproduce
Steps to reproduce the behavior:

1. Start with snapshot 2.7.0 based on after commit cb402cb867d05dc6e33a7b19f8d95548cfb3d445
2. Try to install a repo plugin opensearch-plugin install file:repository-s3-2.7.0.zip or opensearch-plugin install file:repository-azure-2.7.0.zip
3. You'll get an exception on installation:

Exception in thread "main" java.lang.NoClassDefFoundError: org/opensearch/core/xcontent/ToXContentFragment
	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1012)
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
	at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:862)
	at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:760)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:681)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:639)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
	at org.opensearch.cli.CommandLoggingConfigurator.configureLoggingWithoutConfig(CommandLoggingConfigurator.java:54)
	at org.opensearch.cli.Command.main(Command.java:98)
	at org.opensearch.common.settings.KeyStoreCli.main(KeyStoreCli.java:56)
Caused by: java.lang.ClassNotFoundException: org.opensearch.core.xcontent.ToXContentFragment
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)

Expected behavior
A clear and concise description of what you expected to happen.

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[kotwanikunal]

@dgilling Is the plugin built locally as well?

Tagging @nknize as well.

[dgilling]

Yes, this was built locally using opensearchstaging/ci-runner:ci-runner-centos7-opensearch-build-v2. I could also be doing something as well wrong. However, the flow worked fine up until this commit. Reverting to previous one fixes the issue.

[tlfeng]

I attempted to build OpenSearch and the 2 plugins from 2.x branch directly with Gradle command, and the native plugins can be installed properly, although I didn't use the docker image you mentioned.

./gradlew :distribution:archives:linux-tar:assemble
./gradlew :plugins:repository-s3:assemble
./gradlew :plugins:repository-azure:assemble

bin/opensearch-plugin install file:///home/user/Downloads/opensearch/2.7.0/repository-s3-2.7.0-SNAPSHOT.zip
-> Installing file:///home/user/Downloads/opensearch/2.7.0/repository-s3-2.7.0-SNAPSHOT.zip
-> Downloading file:///home/user/Downloads/opensearch/2.7.0/repository-s3-2.7.0-SNAPSHOT.zip
[=================================================] 100%   
...

Continue with installation? [y/N]y
-> Installed repository-s3 with folder name repository-s3

[dgilling]

My fault, after looking at it, my Dockerfile missed including the updated opensearch-cli jar