Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[build] include in plugins only needed jars #11944

Closed

Conversation

dadoonet
Copy link
Member

We don't shade anymore elasticsearch dependencies, so plugins might include jars in the distribution ZIP file which might not be needed anymore.

For example, elasticsearch-cloud-aws comes with:

Archive:  cloud-aws/target/releases/elasticsearch-cloud-aws-2.0.0-SNAPSHOT.zip
  Length     Date   Time    Name
 --------    ----   ----    ----
  1920788  05-18-15 09:42   aws-java-sdk-ec2-1.9.34.jar
   503963  05-18-15 09:42   aws-java-sdk-core-1.9.34.jar
   232771  01-19-15 09:24   commons-codec-1.6.jar
   915096  01-19-15 09:24   jackson-databind-2.3.2.jar
   252288  05-18-15 09:42   aws-java-sdk-kms-1.9.34.jar
    62050  01-19-15 09:24   commons-logging-1.1.3.jar
   282269  10-31-14 13:19   httpcore-4.3.2.jar
    35058  01-19-15 09:24   jackson-annotations-2.3.0.jar
   229998  05-29-15 12:28   jackson-core-2.5.3.jar
   589289  01-19-15 09:24   joda-time-2.7.jar
   562858  05-18-15 09:42   aws-java-sdk-s3-1.9.34.jar
   590533  10-31-14 13:19   httpclient-4.3.5.jar
    44854  06-12-15 19:22   elasticsearch-cloud-aws-2.0.0-SNAPSHOT.jar
 --------                   -------
  6221815                   13 files

A lot of those files are already distributed with elasticsearch itself so classes are available within the classloader.

Closes #11647.

If accepted, I'll most likely squash all commits in a single one.

dadoonet added 6 commits June 30, 2015 10:42
Based on discussion https://github.com/elastic/elasticsearch-analysis-kuromoji/issues/61, we don't need to include all transitive dependencies as they are already provided.
Follow up for https://github.com/elastic/elasticsearch-analysis-kuromoji/issues/61

We don't shade anymore elasticsearch dependencies, so plugins might include jars in the distribution ZIP file which might not be needed anymore.

For example, `elasticsearch-cloud-aws` comes with:

```
Archive:  cloud-aws/target/releases/elasticsearch-cloud-aws-2.0.0-SNAPSHOT.zip
  Length     Date   Time    Name
 --------    ----   ----    ----
  1920788  05-18-15 09:42   aws-java-sdk-ec2-1.9.34.jar
   503963  05-18-15 09:42   aws-java-sdk-core-1.9.34.jar
   232771  01-19-15 09:24   commons-codec-1.6.jar
   915096  01-19-15 09:24   jackson-databind-2.3.2.jar
   252288  05-18-15 09:42   aws-java-sdk-kms-1.9.34.jar
    62050  01-19-15 09:24   commons-logging-1.1.3.jar
   282269  10-31-14 13:19   httpcore-4.3.2.jar
    35058  01-19-15 09:24   jackson-annotations-2.3.0.jar
   229998  05-29-15 12:28   jackson-core-2.5.3.jar
   589289  01-19-15 09:24   joda-time-2.7.jar
   562858  05-18-15 09:42   aws-java-sdk-s3-1.9.34.jar
   590533  10-31-14 13:19   httpclient-4.3.5.jar
    44854  06-12-15 19:22   elasticsearch-cloud-aws-2.0.0-SNAPSHOT.jar
 --------                   -------
  6221815                   13 files
```

A lot of those files are already distributed with elasticsearch itself so classes are available within the classloader.

Closes elastic#11647.

First step. Needs some other work...
@rmuir
Copy link
Contributor

rmuir commented Jun 30, 2015

looks good to me, thanks for tackling this!

Hopefully we get a chance to tackle #11935 and we will have a running "test" in jenkins for these issues.

@dadoonet
Copy link
Member Author

dadoonet commented Jul 1, 2015

Closed by e429b8d

@dadoonet dadoonet closed this Jul 1, 2015
@dadoonet dadoonet deleted the plugins/reduce-analysis-size branch July 1, 2015 19:39
@apatrida
Copy link
Contributor

apatrida commented Jul 4, 2015

You sure you didn't delete one too many? Creating an S3 repo with master (as-of today) of elastic search tree, with the cloud-was plugin from the same tree results in missing classes:

org.elasticsearch.common.inject.CreationException: Guice creation errors:

1) Error injecting constructor, java.lang.NoClassDefFoundError: org/apache/http/protocol/HttpContext
  at org.elasticsearch.repositories.s3.S3Repository.<init>(Unknown Source)
  while locating org.elasticsearch.repositories.s3.S3Repository
  while locating org.elasticsearch.repositories.Repository

1 error
    at org.elasticsearch.common.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:344)
    at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:178)
    at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
    at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:140)
    at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:69)
    at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:404)
    at org.elasticsearch.repositories.RepositoriesService.registerRepository(RepositoriesService.java:368)
    at org.elasticsearch.repositories.RepositoriesService.access$100(RepositoriesService.java:55)
    at org.elasticsearch.repositories.RepositoriesService$1.execute(RepositoriesService.java:110)
    at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:378)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:209)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:179)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoClassDefFoundError: org/apache/http/protocol/HttpContext
    at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:129)
    at com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:432)
    at com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:414)
    at org.elasticsearch.cloud.aws.InternalAwsS3Service.getClient(InternalAwsS3Service.java:153)
    at org.elasticsearch.cloud.aws.InternalAwsS3Service.client(InternalAwsS3Service.java:82)
    at org.elasticsearch.repositories.s3.S3Repository.<init>(S3Repository.java:125)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
    at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:56)
    at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
    at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:104)
    at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:54)
    at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
    at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:865)
    at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
    at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
    at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:46)
    at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:201)
    at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
    at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:858)
    at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
    at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
    ... 13 more
Caused by: java.lang.ClassNotFoundException: org.apache.http.protocol.HttpContext
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 37 more

@mark-vieira mark-vieira added the Team:Delivery Meta label for Delivery team label Nov 11, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Delivery/Build Build or test infrastructure Team:Delivery Meta label for Delivery team v2.0.0-beta1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[build] include in plugins only needed jars
4 participants