Implement/smarter index selection #5642
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
December 10, 2015 16:55
… implement/smarterIndexSelection
When using segmented fetch we currently assume that documents are stored in index patterns based on time, but this is no long necessary with the use of the field stats API. Since documents can be in any order we need to fetch documents from every index and then sort them client side. Since fetching the documents can take some time, we are using the bounds of the time field in an index to try and identify indices which can't produce hits for a result set, and then not fetch their documents. It works like this: - get the list of indices and the min/max of their timefield - fetch the entire sample size from each index until we have enough documents to satisfy the sample. - for each remaining index pattern - if the min/max of it's timefield overlaps the min/max of the fetched documents also fetch the documents for that index pattern - otherwise fetch with size=0
|
This isn't for 4.2.2 since the field stats stuff was added in 4.3.0. |
|
Please correct me if I'm wrong! |
| const hitWindow = this._hitWindow; | ||
|
|
||
| // the order of documents isn't important, just get us more | ||
| if (!this._sortFn) return Math.max(this._desiredSize - hitWindow.size, 0); |
There was a problem hiding this comment.
hitWindow is used before it is verified to exist
|
Get rid of the mix of es5 and es6. Stick with es5 in this pull because it changes functionality. You're welcome to convert to es6 in a separate pull that only contains es6 changes. |
| .sort(this._sortFn) | ||
| .slice(0, this._desiredSize); | ||
| }); | ||
| } |
There was a problem hiding this comment.
merged.hits.hits needs to be sliced even if this._sortFn is not defined
added 6 commits
December 11, 2015 11:47
…rterIndexSelection
…rterIndexSelection
added 4 commits
December 11, 2015 12:06
|
|
||
| notify.event('flatten hit and count fields', function () { | ||
| var counts = $scope.fieldCounts; | ||
| var counts = $scope.fieldCounts = (sortFn ? {} : $scope.fieldCounts) || {}; |
There was a problem hiding this comment.
Doesn't need to block this PR, but this line is pretty complicated - two assignments, 2 conditionals, 3 branches. In the future we should try to err on the side of multiple lines for something like this.
There was a problem hiding this comment.
I hoped no one would notice
|
I'm all for changing the default value of desiredSize to |
spalger
force-pushed
the
implement/smarterIndexSelection
branch
from
24fd838
to
e07f8ac
Compare
spalger
pushed a commit
that referenced
this pull request
Dec 11, 2015
spalger
pushed a commit
that referenced
this pull request
Dec 11, 2015
spalger
pushed a commit
that referenced
this pull request
Dec 11, 2015
spalger
pushed a commit
that referenced
this pull request
Dec 11, 2015
spalger
pushed a commit
that referenced
this pull request
Dec 11, 2015
When using segmented fetch we currently assume that documents are stored in index patterns based on time, but this is no long necessary with the use of the field stats API. Since documents can be in any order we need to fetch documents from every index and then sort them client side. Since fetching the documents can take some time, we are using the bounds of the time field in an index to try and identify indices which can't produce hits for a result set, and then not fetch their documents. It works like this: - get the list of indices and the min/max of their timefield - fetch the entire sample size from each index until we have enough documents to satisfy the sample. - for each remaining index pattern - if the min/max of it's timefield overlaps the min/max of the fetched documents also fetch the documents for that index pattern - otherwise fetch with size=0 Fixes #5642
spalger
pushed a commit
that referenced
this pull request
Dec 11, 2015
spalger
pushed a commit
that referenced
this pull request
Dec 11, 2015
spalger
pushed a commit
that referenced
this pull request
Dec 11, 2015
spalger
pushed a commit
that referenced
this pull request
Dec 11, 2015
spalger
pushed a commit
that referenced
this pull request
Dec 11, 2015
spalger
pushed a commit
that referenced
this pull request
Dec 11, 2015
spalger
pushed a commit
that referenced
this pull request
Dec 11, 2015
spalger
pushed a commit
that referenced
this pull request
Dec 11, 2015
When using segmented fetch we currently assume that documents are stored in index patterns based on time, but this is no long necessary with the use of the field stats API. Since documents can be in any order we need to fetch documents from every index and then sort them client side. Since fetching the documents can take some time, we are using the bounds of the time field in an index to try and identify indices which can't produce hits for a result set, and then not fetch their documents. It works like this: - get the list of indices and the min/max of their timefield - fetch the entire sample size from each index until we have enough documents to satisfy the sample. - for each remaining index pattern - if the min/max of it's timefield overlaps the min/max of the fetched documents also fetch the documents for that index pattern - otherwise fetch with size=0 Fixes #5642
spalger
pushed a commit
that referenced
this pull request
Dec 11, 2015
spalger
pushed a commit
that referenced
this pull request
Dec 11, 2015
spalger
pushed a commit
that referenced
this pull request
Dec 11, 2015
spalger
pushed a commit
that referenced
this pull request
Dec 11, 2015
spalger
pushed a commit
that referenced
this pull request
Dec 11, 2015
spalger
pushed a commit
that referenced
this pull request
Dec 11, 2015
spalger
pushed a commit
that referenced
this pull request
Dec 11, 2015
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When using segmented fetch we currently assume that documents are stored in index patterns based on time, but this is no longer a safe assumption. Since documents can be in any order we need to fetch documents from every index and then sort them client side. Since fetching the documents can take some time, we are using the bounds of the time field in an index to try and identify indices which can't produce hits for a result set, and then not fetch their documents.
It works like this:
Issues:
For #5605