Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] GeoIpCliTests classMethod failing on Windows #71145

Closed
mark-vieira opened this issue Mar 31, 2021 · 6 comments · Fixed by #71156 or #71598
Closed

[CI] GeoIpCliTests classMethod failing on Windows #71145

mark-vieira opened this issue Mar 31, 2021 · 6 comments · Fixed by #71156 or #71598
Assignees
Labels
:Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP Team:Data Management Meta label for data/management team >test-failure Triaged test failures from CI

Comments

@mark-vieira
Copy link
Contributor

mark-vieira commented Mar 31, 2021

Seems this is a newly added test. The LuceneTestCase cleanup code doesn't play well here on Windows.

Build scan:
https://gradle-enterprise.elastic.co/s/se7yv5fe6d4am/tests/:distribution:tools:geoip-cli:test/org.elasticsearch.geoip.GeoIpCliTests/classMethod

Reproduction line:
./gradlew.bat :distribution:tools:geoip-cli:test --tests "org.elasticsearch.geoip.GeoIpCliTests"

Applicable branches:
master

Reproduces locally?:
Didn't try

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.geoip.GeoIpCliTests&tests.test=classMethod

Failure excerpt:

java.io.IOException: Could not remove the following files (in the order of attempts):
   C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\os\windows-2012-r2\distribution\tools\geoip-cli\build\testrun\test\temp\org.elasticsearch.geoip.GeoIpCliTests_D0C1D7D454917487-001\tempDir-003: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\os\windows-2012-r2\distribution\tools\geoip-cli\build\testrun\test\temp\org.elasticsearch.geoip.GeoIpCliTests_D0C1D7D454917487-001\tempDir-003
   C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\os\windows-2012-r2\distribution\tools\geoip-cli\build\testrun\test\temp\org.elasticsearch.geoip.GeoIpCliTests_D0C1D7D454917487-001\tempDir-002: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\os\windows-2012-r2\distribution\tools\geoip-cli\build\testrun\test\temp\org.elasticsearch.geoip.GeoIpCliTests_D0C1D7D454917487-001\tempDir-002
   C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\os\windows-2012-r2\distribution\tools\geoip-cli\build\testrun\test\temp\org.elasticsearch.geoip.GeoIpCliTests_D0C1D7D454917487-001\tempDir-003\target: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\os\windows-2012-r2\distribution\tools\geoip-cli\build\testrun\test\temp\org.elasticsearch.geoip.GeoIpCliTests_D0C1D7D454917487-001\tempDir-003\target
   C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\os\windows-2012-r2\distribution\tools\geoip-cli\build\testrun\test\temp\org.elasticsearch.geoip.GeoIpCliTests_D0C1D7D454917487-001\tempDir-002\target: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\os\windows-2012-r2\distribution\tools\geoip-cli\build\testrun\test\temp\org.elasticsearch.geoip.GeoIpCliTests_D0C1D7D454917487-001\tempDir-002\target
   C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\os\windows-2012-r2\distribution\tools\geoip-cli\build\testrun\test\temp\org.elasticsearch.geoip.GeoIpCliTests_D0C1D7D454917487-001\tempDir-002\source: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\os\windows-2012-r2\distribution\tools\geoip-cli\build\testrun\test\temp\org.elasticsearch.geoip.GeoIpCliTests_D0C1D7D454917487-001\tempDir-002\source
   C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\os\windows-2012-r2\distribution\tools\geoip-cli\build\testrun\test\temp\org.elasticsearch.geoip.GeoIpCliTests_D0C1D7D454917487-001: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\os\windows-2012-r2\distribution\tools\geoip-cli\build\testrun\test\temp\org.elasticsearch.geoip.GeoIpCliTests_D0C1D7D454917487-001

  at __randomizedtesting.SeedInfo.seed([D0C1D7D454917487]:0)
  at org.apache.lucene.util.IOUtils.rm(IOUtils.java:319)
  at org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:216)
  at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831)
  at java.lang.Thread.run(Thread.java:834)

@mark-vieira mark-vieira added :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP >test-failure Triaged test failures from CI labels Mar 31, 2021
@elasticmachine elasticmachine added the Team:Data Management Meta label for data/management team label Mar 31, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features (Team:Core/Features)

@mark-vieira
Copy link
Contributor Author

@probakowski do you mind taking a look since you added this test today?

FYI, I've muted this on master, not sure if this test has been backported to other branches yet.

@probakowski probakowski self-assigned this Mar 31, 2021
@probakowski
Copy link
Contributor

Thanks @mark-vieira, I'll take a look

@mark-vieira
Copy link
Contributor Author

Muted this in 7.x as well.

probakowski added a commit that referenced this issue Apr 1, 2021
Instead of creating directories manually under temp dir we should simply use createTempDir() twice. This should make Windows be able to delete directories correctly.

This PR also removes unused method.

Fixes #71145
probakowski added a commit to probakowski/elasticsearch that referenced this issue Apr 1, 2021
Instead of creating directories manually under temp dir we should simply use createTempDir() twice. This should make Windows be able to delete directories correctly.

This PR also removes unused method.

Fixes elastic#71145
probakowski added a commit that referenced this issue Apr 1, 2021
Instead of creating directories manually under temp dir we should simply use createTempDir() twice. This should make Windows be able to delete directories correctly.

This PR also removes unused method.

Fixes #71145
@csoulios
Copy link
Contributor

csoulios commented Apr 1, 2021

Seems that test is failing again in Windows enviroments

Build scan:
https://gradle-enterprise.elastic.co/s/l5rpw7wgyof3y/tests/:distribution:tools:geoip-cli:test/org.elasticsearch.geoip.GeoIpCliTests/classMethod

Reproduction line:
./gradlew.bat :distribution:tools:geoip-cli:test --tests "org.elasticsearch.geoip.GeoIpCliTests"

Applicable branches:
7.x (commit 6a96e9e)

Reproduces locally?:
Didn't try

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.geoip.GeoIpCliTests&tests.test=classMethod

org.elasticsearch.geoip.GeoIpCliTests > classMethod FAILED
    java.io.IOException: Could not remove the following files (in the order of attempts):
       C:\Users\jenkins\workspace\elastic+elasticsearch+7.x+multijob-windows-compatibility\os\windows-2012-r2\distribution\tools\geoip-cli\build\testrun\test\temp\org.elasticsearch.geoip.GeoIpCliTests_707E1E6AD1D282EE-001\tempDir-002: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\elastic+elasticsearch+7.x+multijob-windows-compatibility\os\windows-2012-r2\distribution\tools\geoip-cli\build\testrun\test\temp\org.elasticsearch.geoip.GeoIpCliTests_707E1E6AD1D282EE-001\tempDir-002
       C:\Users\jenkins\workspace\elastic+elasticsearch+7.x+multijob-windows-compatibility\os\windows-2012-r2\distribution\tools\geoip-cli\build\testrun\test\temp\org.elasticsearch.geoip.GeoIpCliTests_707E1E6AD1D282EE-001\tempDir-001: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\elastic+elasticsearch+7.x+multijob-windows-compatibility\os\windows-2012-r2\distribution\tools\geoip-cli\build\testrun\test\temp\org.elasticsearch.geoip.GeoIpCliTests_707E1E6AD1D282EE-001\tempDir-001
       C:\Users\jenkins\workspace\elastic+elasticsearch+7.x+multijob-windows-compatibility\os\windows-2012-r2\distribution\tools\geoip-cli\build\testrun\test\temp\org.elasticsearch.geoip.GeoIpCliTests_707E1E6AD1D282EE-001\tempDir-004: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\elastic+elasticsearch+7.x+multijob-windows-compatibility\os\windows-2012-r2\distribution\tools\geoip-cli\build\testrun\test\temp\org.elasticsearch.geoip.GeoIpCliTests_707E1E6AD1D282EE-001\tempDir-004
       C:\Users\jenkins\workspace\elastic+elasticsearch+7.x+multijob-windows-compatibility\os\windows-2012-r2\distribution\tools\geoip-cli\build\testrun\test\temp\org.elasticsearch.geoip.GeoIpCliTests_707E1E6AD1D282EE-001: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+7.x+multijob-windows-compatibility\os\windows-2012-r2\distribution\tools\geoip-cli\build\testrun\test\temp\org.elasticsearch.geoip.GeoIpCliTests_707E1E6AD1D282EE-001
        at __randomizedtesting.SeedInfo.seed([707E1E6AD1D282EE]:0)
        at org.apache.lucene.util.IOUtils.rm(IOUtils.java:319)
        at org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:216)
        at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


Suite: Test class org.elasticsearch.geoip.GeoIpCliTests
  2> java.io.IOException: Could not remove the following files (in the order of attempts):
       C:\Users\jenkins\workspace\elastic+elasticsearch+7.x+multijob-windows-compatibility\os\windows-2012-r2\distribution\tools\geoip-cli\build\testrun\test\temp\org.elasticsearch.geoip.GeoIpCliTests_707E1E6AD1D282EE-001\tempDir-002: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\elastic+elasticsearch+7.x+multijob-windows-compatibility\os\windows-2012-r2\distribution\tools\geoip-cli\build\testrun\test\temp\org.elasticsearch.geoip.GeoIpCliTests_707E1E6AD1D282EE-001\tempDir-002
       C:\Users\jenkins\workspace\elastic+elasticsearch+7.x+multijob-windows-compatibility\os\windows-2012-r2\distribution\tools\geoip-cli\build\testrun\test\temp\org.elasticsearch.geoip.GeoIpCliTests_707E1E6AD1D282EE-001\tempDir-001: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\elastic+elasticsearch+7.x+multijob-windows-compatibility\os\windows-2012-r2\distribution\tools\geoip-cli\build\testrun\test\temp\org.elasticsearch.geoip.GeoIpCliTests_707E1E6AD1D282EE-001\tempDir-001
       C:\Users\jenkins\workspace\elastic+elasticsearch+7.x+multijob-windows-compatibility\os\windows-2012-r2\distribution\tools\geoip-cli\build\testrun\test\temp\org.elasticsearch.geoip.GeoIpCliTests_707E1E6AD1D282EE-001\tempDir-004: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\elastic+elasticsearch+7.x+multijob-windows-compatibility\os\windows-2012-r2\distribution\tools\geoip-cli\build\testrun\test\temp\org.elasticsearch.geoip.GeoIpCliTests_707E1E6AD1D282EE-001\tempDir-004
       C:\Users\jenkins\workspace\elastic+elasticsearch+7.x+multijob-windows-compatibility\os\windows-2012-r2\distribution\tools\geoip-cli\build\testrun\test\temp\org.elasticsearch.geoip.GeoIpCliTests_707E1E6AD1D282EE-001: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+7.x+multijob-windows-compatibility\os\windows-2012-r2\distribution\tools\geoip-cli\build\testrun\test\temp\org.elasticsearch.geoip.GeoIpCliTests_707E1E6AD1D282EE-001
        at __randomizedtesting.SeedInfo.seed([707E1E6AD1D282EE]:0)
        at org.apache.lucene.util.IOUtils.rm(IOUtils.java:319)
        at org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:216)
        at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)

@csoulios csoulios reopened this Apr 1, 2021
@mark-vieira
Copy link
Contributor Author

I've remuted this on master and 7.x.

@mark-vieira mark-vieira changed the title [CI] GeoIpCliTests classMethod failing [CI] GeoIpCliTests classMethod failing on Windows Apr 1, 2021
probakowski added a commit that referenced this issue Apr 13, 2021
This change fixes number of problems in GeoIPv2 code:

- closes streams from Files.list in GeoIpCli, which should fix tests on Windows
- makes sure that total download time in GeoIP stats is non-negative (we serialize it as vInt which can cause problems with negative numbers and it can happen when clock was changed during operation)
- fixes handling of failed/simultaneous downloads, #69951 was meant as a way to prevent 2 persistent tasks to index chunks but it would prevent any update if single download failed mid indexing, this change uses timestamp (lastUpdate) as sort of UUID. This should still prevent 2 tasks to step on each other toes (overwriting chunks) but in the end still only single task should be able to update task state (this is handled by persistent tasks framework)
Closes #71145
probakowski added a commit to probakowski/elasticsearch that referenced this issue Apr 13, 2021
This change fixes number of problems in GeoIPv2 code:

- closes streams from Files.list in GeoIpCli, which should fix tests on Windows
- makes sure that total download time in GeoIP stats is non-negative (we serialize it as vInt which can cause problems with negative numbers and it can happen when clock was changed during operation)
- fixes handling of failed/simultaneous downloads, elastic#69951 was meant as a way to prevent 2 persistent tasks to index chunks but it would prevent any update if single download failed mid indexing, this change uses timestamp (lastUpdate) as sort of UUID. This should still prevent 2 tasks to step on each other toes (overwriting chunks) but in the end still only single task should be able to update task state (this is handled by persistent tasks framework)
Closes elastic#71145
# Conflicts:
#	modules/ingest-geoip/src/main/java/org/elasticsearch/ingest/geoip/GeoIpDownloader.java
probakowski added a commit that referenced this issue Apr 13, 2021
* Fix problems in GeoIPv2 code (#71598)

This change fixes number of problems in GeoIPv2 code:

- closes streams from Files.list in GeoIpCli, which should fix tests on Windows
- makes sure that total download time in GeoIP stats is non-negative (we serialize it as vInt which can cause problems with negative numbers and it can happen when clock was changed during operation)
- fixes handling of failed/simultaneous downloads, #69951 was meant as a way to prevent 2 persistent tasks to index chunks but it would prevent any update if single download failed mid indexing, this change uses timestamp (lastUpdate) as sort of UUID. This should still prevent 2 tasks to step on each other toes (overwriting chunks) but in the end still only single task should be able to update task state (this is handled by persistent tasks framework)
Closes #71145
# Conflicts:
#	modules/ingest-geoip/src/main/java/org/elasticsearch/ingest/geoip/GeoIpDownloader.java

* fix compilation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP Team:Data Management Meta label for data/management team >test-failure Triaged test failures from CI
Projects
None yet
4 participants