Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace exact numDocs by soft-del count in SegmentCommitInfo #31086

Merged
merged 12 commits into from
Jun 5, 2018

Conversation

dnhatn
Copy link
Member

@dnhatn dnhatn commented Jun 5, 2018

This PR adapts/utilizes recent enhancements in Lucene-7.4:

  1. Replaces exactNumDocs by the soft-deletes count in SegmentCommitInfo. This enhancement allows us to back out changes introduced in Use exact numDocs in synced-flush and metadata snapshot #30228.

  2. Always configure the soft-deletes field in IWC

@dnhatn dnhatn added >feature :Distributed Indexing/Store Issues around managing unopened Lucene indices. If it touches Store.java, this is a likely label. :Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. labels Jun 5, 2018
@dnhatn dnhatn requested review from s1monw and bleskes June 5, 2018 01:11
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@dnhatn dnhatn changed the title Replace exactNumDocs by soft-del count in SegmentCommitInfo Replace exact numDocs by soft-del count in SegmentCommitInfo Jun 5, 2018
@dnhatn
Copy link
Member Author

dnhatn commented Jun 5, 2018

run sample packaging tests

@s1monw
Copy link
Contributor

s1monw commented Jun 5, 2018

@elasticmachine run sample packaging tests

Copy link
Contributor

@s1monw s1monw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM I am glad we dont' have to jump through hoops anymore for the doc stats.

@@ -39,6 +40,7 @@
import static org.hamcrest.Matchers.containsString;
import static org.hamcrest.Matchers.notNullValue;

@LuceneTestCase.AwaitsFix(bugUrl = "should-be-fixed-by-upstream")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you elaborate what caused this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These tests are failing on the master branch.

Copy link
Contributor

@bleskes bleskes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Nice stats :)

dnhatn added 4 commits June 5, 2018 08:58
This snapshot includes:
- LUCENE-8341: Record soft deletes in SegmentCommitInfo which will resolve elastic#30851
- LUCENE-8335: Enforce soft-deletes field up-front
@dnhatn
Copy link
Member Author

dnhatn commented Jun 5, 2018

Thanks @s1monw and @bleskes

@dnhatn dnhatn merged commit 755a25a into elastic:ccr Jun 5, 2018
@dnhatn dnhatn deleted the soft-deletes-stats branch June 5, 2018 18:11
dnhatn added a commit that referenced this pull request Jun 5, 2018
This PR adapts/utilizes recent enhancements in Lucene-7.4:

- Replaces exactNumDocs by the soft-deletes count in SegmentCommitInfo.
This enhancement allows us to back out changes introduced in #30228.

- Always configure the soft-deletes field in IWC
jasontedor added a commit to jasontedor/elasticsearch that referenced this pull request Jun 5, 2018
* ccr:
  [DOCS] Creates rest-api folder in docs
  [Rollup] Disallow index patterns that match the rollup index (elastic#30491)
  Replace exact numDocs by soft-del count in SegmentInfo (elastic#31086)
  Upgrade to Lucene-7.4.0-snapshot-0a7c3f462f (elastic#31073)
  Add cors support to NioHttpServerTransport (elastic#30827)
  [DOCS] Fixes security example (elastic#31082)
  Allow terms query in _rollup_search (elastic#30973)
  Removing erroneous repeat
  Adapt bwc versions after backporting elastic#30983 to 6.4
  [Tests] Muting RatedRequestsTests#testXContentParsingIsNotLenient
  TEST:  Retry synced-flush if ongoing ops on primary (elastic#30978)
  Fix docs build.
  Only auto-update license signature if all nodes ready (elastic#30859)
  Add BlobContainer.writeBlobAtomic() (elastic#30902)
  Add a doc value format to binary fields. (elastic#30860)
jasontedor added a commit to jasontedor/elasticsearch that referenced this pull request Jun 5, 2018
* ccr:
  [DOCS] Creates rest-api folder in docs
  [Rollup] Disallow index patterns that match the rollup index (elastic#30491)
  Replace exact numDocs by soft-del count in SegmentInfo (elastic#31086)
  Upgrade to Lucene-7.4.0-snapshot-0a7c3f462f (elastic#31073)
  Add cors support to NioHttpServerTransport (elastic#30827)
  [DOCS] Fixes security example (elastic#31082)
  Allow terms query in _rollup_search (elastic#30973)
  Removing erroneous repeat
  Adapt bwc versions after backporting elastic#30983 to 6.4
  [Tests] Muting RatedRequestsTests#testXContentParsingIsNotLenient
  TEST:  Retry synced-flush if ongoing ops on primary (elastic#30978)
  Fix docs build.
  Only auto-update license signature if all nodes ready (elastic#30859)
  Add BlobContainer.writeBlobAtomic() (elastic#30902)
  Add a doc value format to binary fields. (elastic#30860)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. :Distributed Indexing/Store Issues around managing unopened Lucene indices. If it touches Store.java, this is a likely label. >feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants