Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

InternalEngineTests#testRecoverFromHardDeletesIndex leaked a file #67884

Closed
DaveCTurner opened this issue Jan 25, 2021 · 3 comments
Closed

InternalEngineTests#testRecoverFromHardDeletesIndex leaked a file #67884

DaveCTurner opened this issue Jan 25, 2021 · 3 comments
Assignees
@dnhatn
Labels
:Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. >test-failure Triaged test failures from CI

Comments

@DaveCTurner
Copy link
Contributor

Build scan: https://gradle-enterprise.elastic.co/s/vmzea7zf6jcvs/console-log?task=:server:test

Repro line: REPRODUCE WITH: ./gradlew ':server:test' --tests "org.elasticsearch.index.engine.InternalEngineTests.testRecoverFromHardDeletesIndex" -Dtests.seed=14620F98836DC8 -Dtests.security.manager=true -Dtests.locale=ar-MA -Dtests.timezone=America/St_Thomas -Druntime.java=11

Reproduces locally?: No

Applicable branches: Only seen on master

Failure history: Once in the last 90 days.

Failure excerpt:

org.elasticsearch.index.engine.InternalEngineTests > testRecoverFromHardDeletesIndex FAILED
    java.lang.RuntimeException: MockDirectoryWrapper: cannot close: there are still 1 open files: {_0.cfs=1}
        at __randomizedtesting.SeedInfo.seed([14620F98836DC8:D546594BB8CD2D2A]:0)
        at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:843)
        at org.apache.lucene.store.FilterDirectory.close(FilterDirectory.java:110)
        at org.apache.lucene.store.FilterDirectory.close(FilterDirectory.java:110)
        at org.elasticsearch.index.store.Store$StoreDirectory.innerClose(Store.java:722)
        at org.elasticsearch.index.store.Store.closeInternal(Store.java:426)
        at org.elasticsearch.index.store.Store$1.closeInternal(Store.java:154)
        at org.elasticsearch.common.util.concurrent.AbstractRefCounted.decRef(AbstractRefCounted.java:63)
        at org.elasticsearch.index.store.Store.decRef(Store.java:401)
        at org.elasticsearch.index.store.Store.close(Store.java:408)
        at org.elasticsearch.index.engine.InternalEngineTests.testRecoverFromHardDeletesIndex(InternalEngineTests.java:5513)

        Caused by:
        java.lang.RuntimeException: unclosed IndexInput: _0.cfs
            at org.apache.lucene.store.MockDirectoryWrapper.addFileHandle(MockDirectoryWrapper.java:735)
            at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:778)
            at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
            at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
            at org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.<init>(Lucene50CompoundReader.java:77)
            at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.getCompoundReader(Lucene50CompoundFormat.java:71)
            at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:101)
            at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:83)
            at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:171)
            at org.apache.lucene.index.ReadersAndUpdates.getLatestReader(ReadersAndUpdates.java:235)
            at org.apache.lucene.index.SoftDeletesRetentionMergePolicy.keepFullyDeletedSegment(SoftDeletesRetentionMergePolicy.java:75)
            at org.apache.lucene.index.FilterMergePolicy.keepFullyDeletedSegment(FilterMergePolicy.java:103)
            at org.apache.lucene.index.FilterMergePolicy.keepFullyDeletedSegment(FilterMergePolicy.java:103)
            at org.apache.lucene.index.ReadersAndUpdates.keepFullyDeletedSegment(ReadersAndUpdates.java:752)
            at org.apache.lucene.index.IndexWriter.isFullyDeleted(IndexWriter.java:5511)
            at org.apache.lucene.index.IndexWriter.publishFlushedSegment(IndexWriter.java:2819)
            at org.apache.lucene.index.IndexWriter.lambda$publishFlushedSegments$22(IndexWriter.java:5386)
            at org.apache.lucene.index.DocumentsWriterFlushQueue.innerPurge(DocumentsWriterFlushQueue.java:119)
            at org.apache.lucene.index.DocumentsWriterFlushQueue.tryPurge(DocumentsWriterFlushQueue.java:150)
            at org.apache.lucene.index.DocumentsWriter.purgeFlushTickets(DocumentsWriter.java:191)
            at org.apache.lucene.index.IndexWriter.publishFlushedSegments(IndexWriter.java:5366)
            at org.apache.lucene.index.IndexWriter.access$300(IndexWriter.java:219)
            at org.apache.lucene.index.IndexWriter$1.afterSegmentsFlushed(IndexWriter.java:446)
            at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:521)
            at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:656)
            at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:605)
            at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:290)
            at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:265)
            at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:255)
            at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112)
            at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140)
            at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:66)
            at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:40)
            at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
            at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
            at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:353)
            at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:335)
            at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
            at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
            at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1632)
            at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1609)
            at org.elasticsearch.index.engine.EngineTestCase.getDocIds(EngineTestCase.java:1030)
            at org.elasticsearch.index.engine.InternalEngineTests.testRecoverFromHardDeletesIndex(InternalEngineTests.java:5487)
WARNING: Illegal reflective access by org.mockito.cglib.core.ReflectUtils$2 (file:/var/lib/jenkins/.gradle/caches/modules-2/files-2.1/org.elasticsearch/securemock/1.2/98201d4ad5ac93f6b415ae9172d52b5e7cda490e/securemock-1.2.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Illegal reflective access by org.mockito.internal.creation.jmock.ClassImposterizer$3 (file:/var/lib/jenkins/.gradle/caches/modules-2/files-2.1/org.elasticsearch/securemock/1.2/98201d4ad5ac93f6b415ae9172d52b5e7cda490e/securemock-1.2.jar) to constructor java.nio.file.FileStore()
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
REPRODUCE WITH: ./gradlew ':server:test' --tests "org.elasticsearch.index.engine.InternalEngineTests.testRecoverFromHardDeletesIndex" -Dtests.seed=14620F98836DC8 -Dtests.security.manager=true -Dtests.locale=ar-MA -Dtests.timezone=America/St_Thomas -Druntime.java=11
@DaveCTurner DaveCTurner added >test-failure Triaged test failures from CI :Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. labels Jan 25, 2021
@elasticmachine elasticmachine added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label Jan 25, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@DaveCTurner DaveCTurner mentioned this issue Jan 25, 2021
Merged
@dnhatn dnhatn self-assigned this Jan 25, 2021
@dnhatn dnhatn mentioned this issue Feb 2, 2021
Merged
@dnhatn
Copy link
Member

dnhatn commented Feb 2, 2021

I think this is a Lucene bug. I've opened apache/lucene-solr#2288.

dnhatn added a commit to apache/lucene-solr that referenced this issue Feb 2, 2021
@dnhatn
LUCENE-9722: Close merged readers on abort (#2288)
47e3d06 
We fail to close the merged readers of an aborted merge if its 
output segment contains no document.

This bug was discovered by a test in Elasticsearch 
(elastic/elasticsearch#67884).
@dnhatn
Copy link
Member

dnhatn commented Feb 2, 2021

Fixed in apache/lucene-solr#2288.

@dnhatn dnhatn closed this as completed Feb 2, 2021
asfgit pushed a commit to apache/lucene-solr that referenced this issue Feb 2, 2021
@dnhatn
LUCENE-9722: Close merged readers on abort (#2288)
4ade962 
We fail to close the merged readers of an aborted merge if its 
output segment contains no document.

This bug was discovered by a test in Elasticsearch 
(elastic/elasticsearch#67884).
asfgit pushed a commit to apache/lucene-solr that referenced this issue Feb 2, 2021
@dnhatn
LUCENE-9722: Close merged readers on abort (#2288)
2e7cfbd 
We fail to close the merged readers of an aborted merge if its 
output segment contains no document.

This bug was discovered by a test in Elasticsearch 
(elastic/elasticsearch#67884).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dnhatn dnhatn

Labels
:Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@DaveCTurner @dnhatn @elasticmachine