Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] DockerTests test021InstallPlugin failing #110343

Closed
elasticsearchmachine opened this issue Jul 1, 2024 · 9 comments
Closed

[CI] DockerTests test021InstallPlugin failing #110343

elasticsearchmachine opened this issue Jul 1, 2024 · 9 comments
Labels
:Delivery/Packaging RPM and deb packaging, tar and zip archives, shell and batch scripts medium-risk An open issue or test failure that is a medium risk to future releases Team:Delivery Meta label for Delivery team >test-failure Triaged test failures from CI

Comments

@elasticsearchmachine
Copy link
Collaborator

elasticsearchmachine commented Jul 1, 2024 •
edited
Loading

Build Scans:

Reproduction Line:

null

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

org.elasticsearch.packaging.util.Shell$ShellException: Command was not successful: [bash -c docker exec --tty bc157673d677dca6c36ebf08e4daac0e0a48bc3623c0334777c1268ceefee40b /usr/share/elasticsearch/bin/elasticsearch-plugin list]
   result: exitCode = [137] stdout = [analysis-icu] stderr = []

Issue Reasons:

  • [main] 10 failures in test test021InstallPlugin (1.2% fail rate in 842 executions)
  • [main] 4 failures in step ubuntu-2004_destructivedistrotest.docker_packaging-tests-unix-sample (3.7% fail rate in 109 executions)
  • [main] 3 failures in step rhel-8_destructivedistrotest.docker_packaging-tests-unix-sample (2.8% fail rate in 108 executions)
  • [main] 3 failures in pipeline elasticsearch-periodic-packaging (33.3% fail rate in 9 executions)
  • [main] 7 failures in pipeline elasticsearch-pull-request (6.4% fail rate in 109 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.
@elasticsearchmachine elasticsearchmachine added :Delivery/Packaging RPM and deb packaging, tar and zip archives, shell and batch scripts >test-failure Triaged test failures from CI labels Jul 1, 2024
@elasticsearchmachine
Copy link
Collaborator Author

Pinging @elastic/es-delivery (Team:Delivery)

@elasticsearchmachine elasticsearchmachine added Team:Delivery Meta label for Delivery team needs:risk Requires assignment of a risk label (low, medium, blocker) labels Jul 1, 2024
@mark-vieira
Copy link
Contributor

The underlying error is the following:

java.lang.IllegalStateException: failed to load plugin analysis-icu while checking for jar hell
	at [email protected]/org.elasticsearch.plugins.PluginsUtils.checkBundleJarHell(PluginsUtils.java:353)
	at [email protected]/org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:294)
	at [email protected]/org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:165)
	at [email protected]/org.elasticsearch.node.NodeServiceProvider.newPluginService(NodeServiceProvider.java:55)
	at [email protected]/org.elasticsearch.node.NodeConstruction.createEnvironment(NodeConstruction.java:454)
	at [email protected]/org.elasticsearch.node.NodeConstruction.prepareConstruction(NodeConstruction.java:255)
	at [email protected]/org.elasticsearch.node.Node.<init>(Node.java:192)
	at [email protected]/org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:240)
	at [email protected]/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:240)
	at [email protected]/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:75)
Caused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/plugins/.installing-10801102274982928062/analysis-icu-8.15.0-SNAPSHOT.jar
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:56)
	at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:171)
	at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
	at java.base/java.nio.file.Files.readAttributes(Files.java:1858)
	at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1491)
	at java.base/java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:724)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:251)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:180)
	at java.base/java.util.jar.JarFile.<init>(JarFile.java:345)
	at java.base/java.util.jar.JarFile.<init>(JarFile.java:316)
	at java.base/java.util.jar.JarFile.<init>(JarFile.java:255)
	at [email protected]/org.elasticsearch.jdk.JarHell.checkJarHell(JarHell.java:221)
	at [email protected]/org.elasticsearch.plugins.PluginsUtils.checkBundleJarHell(PluginsUtils.java:349)
	... 9 more

@elastic/es-core-infra I'm not familiar with this .installing file and why it would be missing. As far as the logs show, the plugin was successfully installed. Has anything changes regarding plugin installation recently?

@mark-vieira mark-vieira added the :Core/Infra/Plugins Plugin API and infrastructure label Jul 2, 2024
@elasticsearchmachine
Copy link
Collaborator Author

Pinging @elastic/es-core-infra (Team:Core/Infra)

@elasticsearchmachine elasticsearchmachine added the Team:Core/Infra Meta label for core/infra team label Jul 2, 2024
@thecoop thecoop added low-risk An open issue or test failure that is a low risk to future releases and removed needs:risk Requires assignment of a risk label (low, medium, blocker) labels Jul 5, 2024
@rjernst
Copy link
Member

rjernst commented Jul 6, 2024

When a plugin is installed the zip is extracted into a temporary .installing directory. We check jarhell on the plugin there before moving it to the final plugin directory.

This test failed because when we tried to access a jar from the plugin the file system said it doesn't exist. Yet the way we know about the jar in the first place is we listed the extracted plugin dir. So something is messed up with the filesystem such that the file was there, and then it wasn't. From the plugin installer point of view, there is nothing we can do about that.

@rjernst rjernst removed the :Core/Infra/Plugins Plugin API and infrastructure label Jul 6, 2024
@elasticsearchmachine elasticsearchmachine removed the Team:Core/Infra Meta label for core/infra team label Jul 6, 2024
@mark-vieira
Copy link
Contributor

I have a suspicion this has something to do with some disk space issues in CI we're seeing. Let's get those sorted out and see if these failures continue.

@DaveCTurner
Copy link
Contributor

Failed again for me at https://gradle-enterprise.elastic.co/s/hywviexz5z5xw

@mark-vieira mark-vieira mentioned this issue Jul 18, 2024
Closed
@elasticsearchmachine
Copy link
Collaborator Author

This has been muted on branch 8.15

Mute Reasons:

  • [8.15] 7 failures in test test021InstallPlugin (0.7% fail rate in 996 executions)
  • [8.15] 6 failures in pipeline elasticsearch-periodic-packaging (42.9% fail rate in 14 executions)

Build Scans:

elasticsearchmachine added a commit that referenced this issue Jul 19, 2024
@elasticsearchmachine
Mute org.elasticsearch.packaging.test.DockerTests test021InstallPlugin …
0937b4e 
…#110343
@elasticsearchmachine
Copy link
Collaborator Author

This has been muted on branch main

Mute Reasons:

  • [main] 10 failures in test test021InstallPlugin (1.2% fail rate in 842 executions)
  • [main] 4 failures in step ubuntu-2004_destructivedistrotest.docker_packaging-tests-unix-sample (3.7% fail rate in 109 executions)
  • [main] 3 failures in step rhel-8_destructivedistrotest.docker_packaging-tests-unix-sample (2.8% fail rate in 108 executions)
  • [main] 3 failures in pipeline elasticsearch-periodic-packaging (33.3% fail rate in 9 executions)
  • [main] 7 failures in pipeline elasticsearch-pull-request (6.4% fail rate in 109 executions)

Build Scans:

elasticsearchmachine added a commit that referenced this issue Jul 21, 2024
@elasticsearchmachine
Mute org.elasticsearch.packaging.test.DockerTests test021InstallPlugin …
7869f41 
…#110343
ioanatia pushed a commit to ioanatia/elasticsearch that referenced this issue Jul 22, 2024
@elasticsearchmachine @ioanatia
Mute org.elasticsearch.packaging.test.DockerTests test021InstallPlugin …
827b450 
…elastic#110343
salvatore-campagna pushed a commit to salvatore-campagna/elasticsearch that referenced this issue Jul 23, 2024
@elasticsearchmachine @salvatore-campagna
Mute org.elasticsearch.packaging.test.DockerTests test021InstallPlugin …
b42817b 
…elastic#110343
@mark-vieira mark-vieira added medium-risk An open issue or test failure that is a medium risk to future releases and removed low-risk An open issue or test failure that is a low risk to future releases labels Jul 26, 2024
@lkts lkts mentioned this issue Aug 13, 2024
Closed
@elasticsearchmachine elasticsearchmachine closed this as not planned Won't fix, can't repro, duplicate, stale Nov 1, 2024
@elasticsearchmachine
Copy link
Collaborator Author

This issue has been closed because it has been open for too long with no activity.

Any muted tests that were associated with this issue have been unmuted.

If the tests begin failing again, a new issue will be opened, and they may be muted again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
:Delivery/Packaging RPM and deb packaging, tar and zip archives, shell and batch scripts medium-risk An open issue or test failure that is a medium risk to future releases Team:Delivery Meta label for Delivery team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@rjernst @thecoop @mark-vieira @DaveCTurner @elasticsearchmachine