Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI failures due to file handle leaks #39585

Closed
gwbrown opened this issue Mar 1, 2019 · 10 comments · Fixed by #59182
Closed

CI failures due to file handle leaks #39585

gwbrown opened this issue Mar 1, 2019 · 10 comments · Fixed by #59182
Assignees
@dnhatn
Labels
:Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. >test-failure Triaged test failures from CI

Comments

@gwbrown
Copy link
Contributor

gwbrown commented Mar 1, 2019 •
edited
Loading

CI link: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/2317/console

This has failed a few times with similar errors recently, the earliest being Jan. 7 2019.

Does not reproduce locally, reproduce line:

./gradlew :server:integTest \
  -Dtests.seed=F5F3A811C4C7323A \
  -Dtests.class=org.elasticsearch.action.admin.indices.create.SplitIndexIT \
  -Dtests.security.manager=true \
  -Dtests.locale=en-US \
  -Dtests.timezone=UTC \
  -Dcompiler.java=11 \
  -Druntime.java=8

Stack Trace:

java.lang.RuntimeException: file handle leaks: [FileChannel(/var/lib/jenkins/workspace/elastic+elasticsearch+master+intake/server/build/testrun/integTest/J1/temp/org.elasticsearch.action.admin.indices.create.SplitIndexIT_F5F3A811C4C7323A-001/tempDir-002/custom/FmRRWOOIXe/2/wyqeqwxnS2a6pcZROod50w/1/index/_1.fdt), FileChannel(/var/lib/jenkins/workspace/elastic+elasticsearch+master+intake/server/build/testrun/integTest/J1/temp/org.elasticsearch.action.admin.indices.create.SplitIndexIT_F5F3A811C4C7323A-001/tempDir-002/custom/FmRRWOOIXe/2/wyqeqwxnS2a6pcZROod50w/1/index/_1_Asserting_0.pos), FileChannel(/var/lib/jenkins/workspace/elastic+elasticsearch+master+intake/server/build/testrun/integTest/J1/temp/org.elasticsearch.action.admin.indices.create.SplitIndexIT_F5F3A811C4C7323A-001/tempDir-002/custom/FmRRWOOIXe/2/wyqeqwxnS2a6pcZROod50w/1/index/_1_Asserting_0.doc), FileChannel(/var/lib/jenkins/workspace/elastic+elasticsearch+master+intake/server/build/testrun/integTest/J1/temp/org.elasticsearch.action.admin.indices.create.SplitIndexIT_F5F3A811C4C7323A-001/tempDir-002/custom/FmRRWOOIXe/2/wyqeqwxnS2a6pcZROod50w/1/index/_1_Asserting_0.tip), FileChannel(/var/lib/jenkins/workspace/elastic+elasticsearch+master+intake/server/build/testrun/integTest/J1/temp/org.elasticsearch.action.admin.indices.create.SplitIndexIT_F5F3A811C4C7323A-001/tempDir-002/custom/FmRRWOOIXe/2/wyqeqwxnS2a6pcZROod50w/1/index/_1.dim)]
	at __randomizedtesting.SeedInfo.seed([F5F3A811C4C7323A]:0)
	at org.apache.lucene.mockfile.LeakFS.onClose(LeakFS.java:63)
	at org.apache.lucene.mockfile.FilterFileSystem.close(FilterFileSystem.java:77)
	at org.apache.lucene.mockfile.FilterFileSystem.close(FilterFileSystem.java:78)
	at org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:228)
	at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.Exception
	at org.apache.lucene.mockfile.LeakFS.onOpen(LeakFS.java:46)
	at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81)
	at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:197)
	at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:166)
	at java.nio.channels.FileChannel.open(FileChannel.java:287)
	at java.nio.channels.FileChannel.open(FileChannel.java:335)
	at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
	at org.elasticsearch.index.store.FsDirectoryService$HybridDirectory.openInput(FsDirectoryService.java:157)
	at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
	at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.<init>(CompressingStoredFieldsReader.java:150)
	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsReader(CompressingStoredFieldsFormat.java:121)
	at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsReader(Lucene50StoredFieldsFormat.java:173)
	at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsReader(AssertingStoredFieldsFormat.java:43)
	at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:126)
	at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:83)
	at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:172)
	at org.apache.lucene.index.BufferedUpdatesStream$SegmentState.<init>(BufferedUpdatesStream.java:271)
	at org.apache.lucene.index.FrozenBufferedUpdates.openSegmentStates(FrozenBufferedUpdates.java:329)
	at org.apache.lucene.index.FrozenBufferedUpdates.forceApply(FrozenBufferedUpdates.java:222)
	at org.apache.lucene.index.FrozenBufferedUpdates.tryApply(FrozenBufferedUpdates.java:160)
	at org.apache.lucene.index.IndexWriter.lambda$publishFrozenUpdates$3(IndexWriter.java:2589)
	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5112)
	at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:507)
	at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:294)
	at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:269)
	at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:259)
	at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112)
	at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140)
	at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156)
	at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58)
	at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
	at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
	at org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:319)
	at org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:294)
	at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
	at org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:225)
	at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1559)
	at org.elasticsearch.index.engine.InternalEngine.maybeRefresh(InternalEngine.java:1536)
	at org.elasticsearch.index.shard.IndexShard.scheduledRefresh(IndexShard.java:2936)
	at org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:759)
	at org.elasticsearch.index.IndexService.access$200(IndexService.java:99)
	at org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:883)
	at org.elasticsearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:141)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more

Logs:
consoleText.txt
@gwbrown gwbrown added >test-failure Triaged test failures from CI :Data Management/Indices APIs APIs to create and manage indices and templates labels Mar 1, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features

@davidkyle davidkyle changed the title SplitIndexIT failure due to file handle leaks CI failures due to file handle leaks Apr 29, 2019
@davidkyle
Copy link
Member

The same error occurred in ShrinkIndexIT

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.7+intake/638/

java.lang.RuntimeException: file handle leaks: [FileChannel(/var/lib/jenkins/workspace/elastic+elasticsearch+6.7+intake/server/build/testrun/integTest/J0/temp/org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_7ED14DFEA5A453A0-001/tempDir-002/d0/nodes/4/indices/E0f7ZrQ-T3Oym8Q_Bn8nEg/11/index/_0.cfs)]
	at __randomizedtesting.SeedInfo.seed([7ED14DFEA5A453A0]:0)
	at org.apache.lucene.mockfile.LeakFS.onClose(LeakFS.java:63)
	at org.apache.lucene.mockfile.FilterFileSystem.close(FilterFileSystem.java:77)
	at org.apache.lucene.mockfile.FilterFileSystem.close(FilterFileSystem.java:78)
	at org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:228)
	at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.Exception
	at org.apache.lucene.mockfile.LeakFS.onOpen(LeakFS.java:46)
	at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81)
	at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:197)
	at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:166)
	at java.nio.channels.FileChannel.open(FileChannel.java:287)
	at java.nio.channels.FileChannel.open(FileChannel.java:335)
	at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
	at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:756)
	at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
	at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
	at org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.<init>(Lucene50CompoundReader.java:78)
	at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.getCompoundReader(Lucene50CompoundFormat.java:70)
	at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:100)
	at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:83)
	at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:172)
	at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:214)
	at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:106)
	at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:525)
	at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:294)
	at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:269)
	at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:259)
	at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112)
	at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140)
	at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156)
	at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58)
	at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
	at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
	at org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:326)
	at org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:301)
	at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
	at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
	at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1639)
	at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1620)
	at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslogInternal(InternalEngine.java:461)
	at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:419)
	at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:110)
	at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1447)
	at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:440)
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95)
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:310)
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93)
	at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1737)
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2390)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more

./gradlew :server:integTest -Dtests.seed=7ED14DFEA5A453A0 -Dtests.class=org.elasticsearch.action.admin.indices.create.ShrinkIndexIT -Dtests.security.manager=true -Dtests.locale=en-US -Dtests.timezone=Etc/UTC -Dcompiler.java=12 -Druntime.java=8

@benwtrent
Copy link
Member

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+artifactory/649/consoleFull
Same issue again but this time with RestartIndexFollowingIT:

./gradlew :x-pack:plugin:ccr:internalClusterTest --tests "org.elasticsearch.xpack.ccr.RestartIndexFollowingIT" -Dtests.seed=7D8ACAA0717DC0C -Dtests.security.manager=true -Dtests.locale=en-US -Dtests.timezone=Etc/UTC -Dcompiler.java=12 -Druntime.java=8

Trace:

09:32:57 org.elasticsearch.xpack.ccr.RestartIndexFollowingIT > classMethod FAILED
09:32:57     java.lang.RuntimeException: file handle leaks: [FileChannel(/var/lib/jenkins/workspace/elastic+elasticsearch+7.x+artifactory/x-pack/plugin/ccr/build/testrun/internalClusterTest/temp/org.elasticsearch.xpack.ccr.RestartIndexFollowingIT_7D8ACAA0717DC0C-001/tempDir-003/d0/nodes/0/node.lock), FileChannel(/var/lib/jenkins/workspace/elastic+elasticsearch+7.x+artifactory/x-pack/plugin/ccr/build/testrun/internalClusterTest/temp/org.elasticsearch.xpack.ccr.RestartIndexFollowingIT_7D8ACAA0717DC0C-001/tempDir-003/d1/nodes/0/node.lock), FileChannel(/var/lib/jenkins/workspace/elastic+elasticsearch+7.x+artifactory/x-pack/plugin/ccr/build/testrun/internalClusterTest/temp/org.elasticsearch.xpack.ccr.RestartIndexFollowingIT_7D8ACAA0717DC0C-001/tempDir-003/d2/nodes/0/node.lock), FileChannel(/var/lib/jenkins/workspace/elastic+elasticsearch+7.x+artifactory/x-pack/plugin/ccr/build/testrun/internalClusterTest/temp/org.elasticsearch.xpack.ccr.RestartIndexFollowingIT_7D8ACAA0717DC0C-001/tempDir-003/d2/nodes/1/node.lock), FileChannel(/var/lib/jenkins/workspace/elastic+elasticsearch+7.x+artifactory/x-pack/plugin/ccr/build/testrun/internalClusterTest/temp/org.elasticsearch.xpack.ccr.RestartIndexFollowingIT_7D8ACAA0717DC0C-001/tempDir-003/d1/nodes/1/node.lock), FileChannel(/var/lib/jenkins/workspace/elastic+elasticsearch+7.x+artifactory/x-pack/plugin/ccr/build/testrun/internalClusterTest/temp/org.elasticsearch.xpack.ccr.RestartIndexFollowingIT_7D8ACAA0717DC0C-001/tempDir-003/d0/nodes/1/node.lock)]
09:32:57 
09:32:57         Caused by:
09:32:57         java.lang.Exception
09:32:57             at org.apache.lucene.mockfile.LeakFS.onOpen(LeakFS.java:46)
09:32:57             at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81)
09:32:57             at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:197)
09:32:57             at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:166)
09:32:57             at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202)
09:32:57             at java.nio.channels.FileChannel.open(FileChannel.java:287)
09:32:57             at java.nio.channels.FileChannel.open(FileChannel.java:335)
09:32:57             at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:125)
09:32:57             at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41)
09:32:57             at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45)
09:32:57             at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:212)
09:32:57             at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:267)
09:32:57             at org.elasticsearch.node.Node.<init>(Node.java:270)
09:32:57             at org.elasticsearch.node.MockNode.<init>(MockNode.java:97)
09:32:57             at org.elasticsearch.node.MockNode.<init>(MockNode.java:87)
09:32:57             at org.elasticsearch.node.MockNode.<init>(MockNode.java:79)
09:32:57             at org.elasticsearch.node.MockNode.<init>(MockNode.java:72)
09:32:57             at org.elasticsearch.test.InternalTestCluster$NodeAndClient.recreateNode(InternalTestCluster.java:1032)
09:32:57             at org.elasticsearch.test.InternalTestCluster$NodeAndClient.access$900(InternalTestCluster.java:860)
09:32:57             at org.elasticsearch.test.InternalTestCluster.fullRestart(InternalTestCluster.java:1926)
09:32:57             at org.elasticsearch.test.InternalTestCluster.fullRestart(InternalTestCluster.java:1771)
09:32:57             at org.elasticsearch.xpack.ccr.RestartIndexFollowingIT.testFollowIndex(RestartIndexFollowingIT.java:61)

@dnhatn
Copy link
Member

dnhatn commented May 23, 2019 •
edited
Loading

This failure and #30290 have the same root cause. I am closing this in favour of #30290.

@dnhatn dnhatn closed this as completed May 23, 2019
@dnhatn dnhatn added :Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. and removed :Data Management/Indices APIs APIs to create and manage indices and templates labels May 23, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@DaveCTurner
Copy link
Contributor

Another failure: https://scans.gradle.com/s/okxtr6zgle3mk/console-log?task=:server:integTest#L3306

@pgomulka
Copy link
Contributor

pgomulka commented Jul 3, 2020

I had another failure which looks very similar on 7.x https://gradle-enterprise.elastic.co/s/fz4pznt6bautm

./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.cluster.allocation.AwarenessAllocationIT" \
  -Dtests.seed=69FEE0612865CE52 \
  -Dtests.security.manager=true \
  -Dtests.locale=en-US \
  -Dtests.timezone=Etc/UTC \
  -Druntime.java=8

stacktrace

  Caused by:
13:22:19         java.lang.Exception
13:22:19             at org.apache.lucene.mockfile.LeakFS.onOpen(LeakFS.java:46)
13:22:19             at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81)
13:22:19             at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:197)
13:22:19             at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:166)
13:22:19             at java.nio.channels.FileChannel.open(FileChannel.java:287)
13:22:19             at java.nio.channels.FileChannel.open(FileChannel.java:335)
13:22:19             at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
13:22:19             at org.elasticsearch.index.store.FsDirectoryFactory$HybridDirectory.openInput(FsDirectoryFactory.java:143)
13:22:19             at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
13:22:19             at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
13:22:19             at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:157)
13:22:19             at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289)
13:22:19             at org.elasticsearch.index.engine.CombinedDeletionPolicy.getDocCountOfCommit(CombinedDeletionPolicy.java:131)
13:22:19             at org.elasticsearch.index.engine.CombinedDeletionPolicy.onCommit(CombinedDeletionPolicy.java:103)
13:22:19             at org.apache.lucene.index.IndexFileDeleter.revisitPolicy(IndexFileDeleter.java:482)
13:22:19             at org.apache.lucene.index.IndexWriter.deleteUnusedFiles(IndexWriter.java:4991)
13:22:19             at org.elasticsearch.index.engine.InternalEngine.releaseIndexCommit(InternalEngine.java:2043)
13:22:19             at org.elasticsearch.index.engine.InternalEngine.lambda$acquireSafeIndexCommit$5(InternalEngine.java:2034)
13:22:19             at org.elasticsearch.index.engine.Engine$IndexCommitRef.close(Engine.java:1780)
13:22:19             at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:106)
13:22:19             at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:64)
13:22:19             at org.elasticsearch.indices.recovery.RecoverySourceHandler.lambda$recoverToTarget$8(RecoverySourceHandler.java:258)
13:22:19             at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:71)
13:22:19             at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:88)
13:22:19             at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39)
13:22:19             at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226)
13:22:19             at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
13:22:19             at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
13:22:19             at java.util.ArrayList.forEach(ArrayList.java:1257)
13:22:19             at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
13:22:19             at org.elasticsearch.common.util.concurrent.BaseFuture.setException(BaseFuture.java:162)
13:22:19             at org.elasticsearch.common.util.concurrent.ListenableFuture.onFailure(ListenableFuture.java:135)
13:22:19             at org.elasticsearch.action.StepListener.innerOnFailure(StepListener.java:67)
13:22:19             at org.elasticsearch.action.NotifyOnceListener.onFailure(NotifyOnceListener.java:47)
13:22:19             at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:71)
13:22:19             at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:88)
13:22:19             at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39)
13:22:19             at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226)
13:22:19             at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
13:22:19             at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
13:22:19             at java.util.ArrayList.forEach(ArrayList.java:1257)
13:22:19             at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
13:22:19             at org.elasticsearch.common.util.concurrent.BaseFuture.setException(BaseFuture.java:162)
13:22:19             at org.elasticsearch.common.util.concurrent.ListenableFuture.onFailure(ListenableFuture.java:135)
13:22:19             at org.elasticsearch.action.StepListener.innerOnFailure(StepListener.java:67)
13:22:19             at org.elasticsearch.action.NotifyOnceListener.onFailure(NotifyOnceListener.java:47)
13:22:19             at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:71)
13:22:19             at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:88)
13:22:19             at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39)
13:22:19             at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226)
13:22:19             at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
13:22:19             at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:68)
13:22:19             at org.elasticsearch.action.StepListener.whenComplete(StepListener.java:78)
13:22:19             at org.elasticsearch.indices.recovery.RecoverySourceHandler.lambda$createRetentionLease$30(RecoverySourceHandler.java:587)
13:22:19             at org.elasticsearch.indices.recovery.RecoverySourceHandler.lambda$runUnderPrimaryPermit$21(RecoverySourceHandler.java:387)
13:22:19             at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:108)
13:22:19             at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:89)
13:22:19             at org.elasticsearch.indices.recovery.RecoverySourceHandler.runUnderPrimaryPermit(RecoverySourceHandler.java:365)
13:22:19             at org.elasticsearch.indices.recovery.RecoverySourceHandler.createRetentionLease(RecoverySourceHandler.java:571)
13:22:19             at org.elasticsearch.indices.recovery.RecoverySourceHandler.lambda$phase1$24(RecoverySourceHandler.java:528)
13:22:19             at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
13:22:19             at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
13:22:19             at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
13:22:19             at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226)
13:22:19             at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
13:22:19             at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
13:22:19             at java.util.ArrayList.forEach(ArrayList.java:1257)
13:22:19             at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
13:22:19             at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
13:22:19             at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
13:22:19             at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
13:22:19             at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
13:22:19             at org.elasticsearch.indices.recovery.MultiFileTransfer.onCompleted(MultiFileTransfer.java:149)
13:22:19             at org.elasticsearch.indices.recovery.MultiFileTransfer.handleItems(MultiFileTransfer.java:119)
13:22:19             at org.elasticsearch.indices.recovery.MultiFileTransfer.access$000(MultiFileTransfer.java:60)
13:22:19             at org.elasticsearch.indices.recovery.MultiFileTransfer$1.write(MultiFileTransfer.java:79)
13:22:19             at org.elasticsearch.common.util.concurrent.AsyncIOProcessor.processList(AsyncIOProcessor.java:108)
13:22:19             at org.elasticsearch.common.util.concurrent.AsyncIOProcessor.drainAndProcessAndRelease(AsyncIOProcessor.java:96)
13:22:19             at org.elasticsearch.common.util.concurrent.AsyncIOProcessor.put(AsyncIOProcessor.java:84)
13:22:19             at org.elasticsearch.indices.recovery.MultiFileTransfer.addItem(MultiFileTransfer.java:90)
13:22:19             at org.elasticsearch.indices.recovery.MultiFileTransfer.lambda$handleItems$3(MultiFileTransfer.java:125)
13:22:19             at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
13:22:19             at org.elasticsearch.action.ActionListener$6.onResponse(ActionListener.java:282)
13:22:19             at org.elasticsearch.action.ActionListener$4.onResponse(ActionListener.java:163)
13:22:19             at org.elasticsearch.action.ActionListener$6.onResponse(ActionListener.java:282)
13:22:19             at org.elasticsearch.action.support.RetryableAction$RetryingListener.onResponse(RetryableAction.java:136)
13:22:19             at org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:54)
13:22:19             at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1162)
13:22:19             at org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:213)
13:22:19             at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:710)
13:22:19             at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
13:22:19             at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
13:22:19             at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
13:22:19             ... 1 more

@pgomulka pgomulka reopened this Jul 3, 2020
@cbuescher
Copy link
Member

cbuescher commented Jul 7, 2020 •
edited
Loading

Got something very similar looking today in SimpleAllocationIT, doesn't reproduce locally though.

./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.cluster.allocation.SimpleAllocationIT" \
  -Dtests.seed=350E3A83EFD295EF \
  -Dtests.security.manager=true \
  -Dtests.locale=en-US \
  -Dtests.timezone=Etc/UTC \
  -Druntime.java=11

https://gradle-enterprise.elastic.co/s/h4b5xjeymyqeo

java.lang.RuntimeException: file handle leaks: [FileChannel(/var/lib/jenkins/workspace/elastic+elasticsearch+master+multijob-unix-compatibility/os/fedora-29&&immutable/server/build/testrun/internalClusterTest/temp/org.elasticsearch.cluster.allocation.SimpleAllocationIT_350E3A83EFD295EF-001/tempDir-002/node_s3/indices/63blRb5gRXWGYX-ytqyjRA/1/index/segments_3)]
	at __randomizedtesting.SeedInfo.seed([350E3A83EFD295EF]:0)
	at org.apache.lucene.mockfile.LeakFS.onClose(LeakFS.java:63)
	at org.apache.lucene.mockfile.FilterFileSystem.close(FilterFileSystem.java:77)
	at org.apache.lucene.mockfile.FilterFileSystem.close(FilterFileSystem.java:78)
	at org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:228)
	at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.Exception
	at org.apache.lucene.mockfile.LeakFS.onOpen(LeakFS.java:46)
	at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81)
	at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:197)
	at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:166)
	at java.base/java.nio.channels.FileChannel.open(FileChannel.java:292)
	at java.base/java.nio.channels.FileChannel.open(FileChannel.java:345)
	at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
	at org.elasticsearch.index.store.FsDirectoryFactory$HybridDirectory.openInput(FsDirectoryFactory.java:143)
	at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
	at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
	at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:157)
	at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289)
	at org.elasticsearch.index.engine.CombinedDeletionPolicy.getDocCountOfCommit(CombinedDeletionPolicy.java:131)
	at org.elasticsearch.index.engine.CombinedDeletionPolicy.onCommit(CombinedDeletionPolicy.java:103)
	at org.apache.lucene.index.IndexFileDeleter.revisitPolicy(IndexFileDeleter.java:482)
	at org.apache.lucene.index.IndexWriter.deleteUnusedFiles(IndexWriter.java:4991)
	at org.elasticsearch.index.engine.InternalEngine.releaseIndexCommit(InternalEngine.java:1915)
	at org.elasticsearch.index.engine.InternalEngine.lambda$acquireSafeIndexCommit$5(InternalEngine.java:1906)
	at org.elasticsearch.index.engine.Engine$IndexCommitRef.close(Engine.java:1666)
	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:106)
	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:64)
	at org.elasticsearch.indices.recovery.RecoverySourceHandler.lambda$recoverToTarget$8(RecoverySourceHandler.java:248)
	at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:71)
	at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:88)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39)
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:178)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
	at org.elasticsearch.common.util.concurrent.BaseFuture.setException(BaseFuture.java:162)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onFailure(ListenableFuture.java:135)
	at org.elasticsearch.action.StepListener.innerOnFailure(StepListener.java:67)
	at org.elasticsearch.action.NotifyOnceListener.onFailure(NotifyOnceListener.java:47)
	at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:71)
	at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:88)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39)
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:178)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
	at org.elasticsearch.common.util.concurrent.BaseFuture.setException(BaseFuture.java:162)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onFailure(ListenableFuture.java:135)
	at org.elasticsearch.action.StepListener.innerOnFailure(StepListener.java:67)
	at org.elasticsearch.action.NotifyOnceListener.onFailure(NotifyOnceListener.java:47)
	at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:71)
	at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:88)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39)
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:178)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:68)
	at org.elasticsearch.action.StepListener.whenComplete(StepListener.java:78)
	at org.elasticsearch.indices.recovery.RecoverySourceHandler.lambda$createRetentionLease$30(RecoverySourceHandler.java:580)
	at org.elasticsearch.indices.recovery.RecoverySourceHandler.lambda$runUnderPrimaryPermit$21(RecoverySourceHandler.java:380)
	at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:108)
	at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:89)
	at org.elasticsearch.indices.recovery.RecoverySourceHandler.runUnderPrimaryPermit(RecoverySourceHandler.java:358)
	at org.elasticsearch.indices.recovery.RecoverySourceHandler.createRetentionLease(RecoverySourceHandler.java:564)
	at org.elasticsearch.indices.recovery.RecoverySourceHandler.lambda$phase1$24(RecoverySourceHandler.java:521)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:178)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
	at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
	at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
	at org.elasticsearch.indices.recovery.MultiFileTransfer.onCompleted(MultiFileTransfer.java:149)
	at org.elasticsearch.indices.recovery.MultiFileTransfer.handleItems(MultiFileTransfer.java:119)
	at org.elasticsearch.indices.recovery.MultiFileTransfer$1.write(MultiFileTransfer.java:79)
	at org.elasticsearch.common.util.concurrent.AsyncIOProcessor.processList(AsyncIOProcessor.java:108)
	at org.elasticsearch.common.util.concurrent.AsyncIOProcessor.drainAndProcessAndRelease(AsyncIOProcessor.java:96)
	at org.elasticsearch.common.util.concurrent.AsyncIOProcessor.put(AsyncIOProcessor.java:84)
	at org.elasticsearch.indices.recovery.MultiFileTransfer.addItem(MultiFileTransfer.java:90)
	at org.elasticsearch.indices.recovery.MultiFileTransfer.lambda$handleItems$3(MultiFileTransfer.java:125)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.action.ActionListener$6.onResponse(ActionListener.java:282)
	at org.elasticsearch.action.ActionListener$4.onResponse(ActionListener.java:163)
	at org.elasticsearch.action.ActionListener$6.onResponse(ActionListener.java:282)
	at org.elasticsearch.action.support.RetryableAction$RetryingListener.onResponse(RetryableAction.java:136)
	at org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:54)
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1102)
	at org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:213)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:706)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	... 1 more

@dnhatn dnhatn self-assigned this Jul 7, 2020
@dnhatn
Copy link
Member

dnhatn commented Jul 7, 2020

I think this is a bug in Lucene where we do not properly close a FileChannel if we failed to create an NIOFSIndexInput. I will work on a fix for this on Lucene.

@dnhatn dnhatn mentioned this issue Jul 7, 2020
Merged
@dnhatn
Copy link
Member

dnhatn commented Jul 7, 2020

I've opened apache/lucene-solr#1658 to propose a fix in Lucene.

dnhatn added a commit that referenced this issue Jul 8, 2020
@dnhatn
Do not release safe commit with CancellableThreads (#59182)
5688e0e 
We are leaking a FileChannel in #39585 if we release a safe commit with 
CancellableThreads. Although it is a bug in Lucene where we do not close
a FileChannel if we failed to create a NIOFSIndexInput, I think it's
safer if we release a safe commit using the generic thread pool instead.

Closes #39585
Relates #45409
dnhatn added a commit that referenced this issue Jul 8, 2020
@dnhatn
Do not release safe commit with CancellableThreads (#59182)
6a0f741 
We are leaking a FileChannel in #39585 if we release a safe commit with 
CancellableThreads. Although it is a bug in Lucene where we do not close
a FileChannel if we failed to create a NIOFSIndexInput, I think it's
safer if we release a safe commit using the generic thread pool instead.

Closes #39585
Relates #45409
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dnhatn dnhatn

Labels
:Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Do not release safe commit with CancellableThreads dnhatn/elasticsearch
8 participants
@gwbrown @davidkyle @benwtrent @DaveCTurner @cbuescher @pgomulka @dnhatn @elasticmachine