Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Handle _shard_doc field for sort optimization #69321

Merged
merged 2 commits into from
Feb 22, 2021
Merged

Handle _shard_doc field for sort optimization #69321

merged 2 commits into from
Feb 22, 2021

Conversation

jimczi
Copy link
Contributor

@jimczi jimczi commented Feb 22, 2021

This commit ensures that the automatic tiebreaker _shard_doc does
not disable sort optimization.

Relates #56828

@jimczi
Handle _shard_doc field for sort optimization
8c7e179 
This commit ensures that the automatic tiebreaker `_shard_doc` does
not disable sort optimization.

Relates elastic#56828
@jimczi jimczi added >non-issue :Search/Search Search-related issues that do not fall into other categories v8.0.0 v7.12.0 v7.13.0 labels Feb 22, 2021
@elasticmachine elasticmachine added the Team:Search Meta label for search team label Feb 22, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search (Team:Search)

@matriv
Copy link
Contributor

matriv commented Feb 22, 2021 •
edited
Loading

benchmark comparison against 1 commit before the regression: dde3df2

|                                                Min Throughput |                                 sequence_2stage_nofilter_fetch1000_size_1000 |     1.66032 |     1.72739 |  0.06707 |  ops/s |
|                                               Mean Throughput |                                 sequence_2stage_nofilter_fetch1000_size_1000 |     1.87836 |     1.91174 |  0.03338 |  ops/s |
|                                             Median Throughput |                                 sequence_2stage_nofilter_fetch1000_size_1000 |     1.90079 |      1.9281 |  0.02731 |  ops/s |
|                                                Max Throughput |                                 sequence_2stage_nofilter_fetch1000_size_1000 |     1.98016 |     2.00448 |  0.02433 |  ops/s |
|                                       50th percentile latency |                                 sequence_2stage_nofilter_fetch1000_size_1000 |     2257.35 |     2273.56 |   16.212 |     ms |
|                                       90th percentile latency |                                 sequence_2stage_nofilter_fetch1000_size_1000 |     2350.18 |     2363.45 |  13.2685 |     ms |
|                                       99th percentile latency |                                 sequence_2stage_nofilter_fetch1000_size_1000 |     2464.76 |     2447.16 |  -17.604 |     ms |
|                                      100th percentile latency |                                 sequence_2stage_nofilter_fetch1000_size_1000 |     2495.89 |     2464.96 |  -30.928 |     ms |
|                                  50th percentile service time |                                 sequence_2stage_nofilter_fetch1000_size_1000 |     2257.35 |     2273.56 |   16.212 |     ms |
|                                  90th percentile service time |                                 sequence_2stage_nofilter_fetch1000_size_1000 |     2350.18 |     2363.45 |  13.2685 |     ms |
|                                  99th percentile service time |                                 sequence_2stage_nofilter_fetch1000_size_1000 |     2464.76 |     2447.16 |  -17.604 |     ms |
|                                 100th percentile service time |                                 sequence_2stage_nofilter_fetch1000_size_1000 |     2495.89 |     2464.96 |  -30.928 |     ms |
|                                                    error rate |                                 sequence_2stage_nofilter_fetch1000_size_1000 |           0 |           0 |        0 |      % |
|                                                Min Throughput |                 sequence_2stage_nofilter_fetch1000_size1000_tail1000_head200 |     10.4273 |     10.2966 |  -0.1307 |  ops/s |
|                                               Mean Throughput |                 sequence_2stage_nofilter_fetch1000_size1000_tail1000_head200 |     10.6541 |     10.4738 | -0.18034 |  ops/s |
|                                             Median Throughput |                 sequence_2stage_nofilter_fetch1000_size1000_tail1000_head200 |      10.688 |     10.4905 | -0.19755 |  ops/s |
|                                                Max Throughput |                 sequence_2stage_nofilter_fetch1000_size1000_tail1000_head200 |     10.7571 |     10.5654 | -0.19178 |  ops/s |
|                                       50th percentile latency |                 sequence_2stage_nofilter_fetch1000_size1000_tail1000_head200 |     448.928 |     456.561 |   7.6332 |     ms |
|                                       90th percentile latency |                 sequence_2stage_nofilter_fetch1000_size1000_tail1000_head200 |      480.12 |     490.836 |  10.7158 |     ms |
|                                       99th percentile latency |                 sequence_2stage_nofilter_fetch1000_size1000_tail1000_head200 |     516.885 |     524.768 |  7.88284 |     ms |
|                                      100th percentile latency |                 sequence_2stage_nofilter_fetch1000_size1000_tail1000_head200 |     535.535 |     578.278 |   42.743 |     ms |
|                                  50th percentile service time |                 sequence_2stage_nofilter_fetch1000_size1000_tail1000_head200 |     448.928 |     456.561 |   7.6332 |     ms |
|                                  90th percentile service time |                 sequence_2stage_nofilter_fetch1000_size1000_tail1000_head200 |      480.12 |     490.836 |  10.7158 |     ms |
|                                  99th percentile service time |                 sequence_2stage_nofilter_fetch1000_size1000_tail1000_head200 |     516.885 |     524.768 |  7.88284 |     ms |
|                                 100th percentile service time |                 sequence_2stage_nofilter_fetch1000_size1000_tail1000_head200 |     535.535 |     578.278 |   42.743 |     ms |
|                                                    error rate |                 sequence_2stage_nofilter_fetch1000_size1000_tail1000_head200 |           0 |           0 |        0 |      % |
|                                                Min Throughput |          sequence_4stage_nofilter_maxspan5m_fetch1000_size100_head100_tail50 |    0.320413 |    0.327511 |   0.0071 |  ops/s |
|                                               Mean Throughput |          sequence_4stage_nofilter_maxspan5m_fetch1000_size100_head100_tail50 |    0.332867 |    0.335927 |  0.00306 |  ops/s |
|                                             Median Throughput |          sequence_4stage_nofilter_maxspan5m_fetch1000_size100_head100_tail50 |    0.333985 |      0.3363 |  0.00232 |  ops/s |
|                                                Max Throughput |          sequence_4stage_nofilter_maxspan5m_fetch1000_size100_head100_tail50 |    0.338837 |    0.339455 |  0.00062 |  ops/s |
|                                       50th percentile latency |          sequence_4stage_nofilter_maxspan5m_fetch1000_size100_head100_tail50 |     14573.7 |     14679.7 |  106.036 |     ms |
|                                       90th percentile latency |          sequence_4stage_nofilter_maxspan5m_fetch1000_size100_head100_tail50 |     14831.6 |     14905.9 |  74.3569 |     ms |
|                                       99th percentile latency |          sequence_4stage_nofilter_maxspan5m_fetch1000_size100_head100_tail50 |     15409.1 |     15092.2 | -316.886 |     ms |
|                                      100th percentile latency |          sequence_4stage_nofilter_maxspan5m_fetch1000_size100_head100_tail50 |     15478.5 |     15172.2 | -306.264 |     ms |
|                                  50th percentile service time |          sequence_4stage_nofilter_maxspan5m_fetch1000_size100_head100_tail50 |     14573.7 |     14679.7 |  106.036 |     ms |
|                                  90th percentile service time |          sequence_4stage_nofilter_maxspan5m_fetch1000_size100_head100_tail50 |     14831.6 |     14905.9 |  74.3569 |     ms |
|                                  99th percentile service time |          sequence_4stage_nofilter_maxspan5m_fetch1000_size100_head100_tail50 |     15409.1 |     15092.2 | -316.886 |     ms |
|                                 100th percentile service time |          sequence_4stage_nofilter_maxspan5m_fetch1000_size100_head100_tail50 |     15478.5 |     15172.2 | -306.264 |     ms |
|                                                    error rate |          sequence_4stage_nofilter_maxspan5m_fetch1000_size100_head100_tail50 |           0 |           0 |        0 |      % |
|                                                Min Throughput | sequence_3stage_startsWithfilter_maxspan30m_fetch1000_size100_tail100_head50 |     19.5643 |     22.5926 |  3.02826 |  ops/s |
|                                               Mean Throughput | sequence_3stage_startsWithfilter_maxspan30m_fetch1000_size100_tail100_head50 |     20.7939 |     25.0813 |   4.2874 |  ops/s |
|                                             Median Throughput | sequence_3stage_startsWithfilter_maxspan30m_fetch1000_size100_tail100_head50 |     21.0291 |      25.341 |  4.31194 |  ops/s |
|                                                Max Throughput | sequence_3stage_startsWithfilter_maxspan30m_fetch1000_size100_tail100_head50 |     21.4223 |     26.1454 |  4.72301 |  ops/s |
|                                       50th percentile latency | sequence_3stage_startsWithfilter_maxspan30m_fetch1000_size100_tail100_head50 |      221.67 |      177.52 | -44.1494 |     ms |
|                                       90th percentile latency | sequence_3stage_startsWithfilter_maxspan30m_fetch1000_size100_tail100_head50 |     240.552 |     190.919 | -49.6331 |     ms |
|                                       99th percentile latency | sequence_3stage_startsWithfilter_maxspan30m_fetch1000_size100_tail100_head50 |     258.081 |     207.992 | -50.0887 |     ms |
|                                      100th percentile latency | sequence_3stage_startsWithfilter_maxspan30m_fetch1000_size100_tail100_head50 |     291.379 |     216.152 | -75.2275 |     ms |
|                                  50th percentile service time | sequence_3stage_startsWithfilter_maxspan30m_fetch1000_size100_tail100_head50 |      221.67 |      177.52 | -44.1494 |     ms |
|                                  90th percentile service time | sequence_3stage_startsWithfilter_maxspan30m_fetch1000_size100_tail100_head50 |     240.552 |     190.919 | -49.6331 |     ms |
|                                  99th percentile service time | sequence_3stage_startsWithfilter_maxspan30m_fetch1000_size100_tail100_head50 |     258.081 |     207.992 | -50.0887 |     ms |
|                                 100th percentile service time | sequence_3stage_startsWithfilter_maxspan30m_fetch1000_size100_tail100_head50 |     291.379 |     216.152 | -75.2275 |     ms |
|                                                    error rate | sequence_3stage_startsWithfilter_maxspan30m_fetch1000_size100_tail100_head50 |           0 |           0 |        0 |      % |

benchmark comparison against ab7dd46 that introduced the regression:

|                                                Min Throughput |                                 sequence_2stage_nofilter_fetch1000_size_1000 |     1.87392 |     1.72739 | -0.14653 |  ops/s |
|                                               Mean Throughput |                                 sequence_2stage_nofilter_fetch1000_size_1000 |     1.94773 |     1.91174 | -0.03599 |  ops/s |
|                                             Median Throughput |                                 sequence_2stage_nofilter_fetch1000_size_1000 |     1.93745 |      1.9281 | -0.00934 |  ops/s |
|                                                Max Throughput |                                 sequence_2stage_nofilter_fetch1000_size_1000 |     2.10322 |     2.00448 | -0.09874 |  ops/s |
|                                       50th percentile latency |                                 sequence_2stage_nofilter_fetch1000_size_1000 |     2702.05 |     2273.56 | -428.491 |     ms |
|                                       90th percentile latency |                                 sequence_2stage_nofilter_fetch1000_size_1000 |     2877.07 |     2363.45 | -513.628 |     ms |
|                                       99th percentile latency |                                 sequence_2stage_nofilter_fetch1000_size_1000 |     2997.72 |     2447.16 | -550.561 |     ms |
|                                      100th percentile latency |                                 sequence_2stage_nofilter_fetch1000_size_1000 |     3104.37 |     2464.96 | -639.407 |     ms |
|                                  50th percentile service time |                                 sequence_2stage_nofilter_fetch1000_size_1000 |     2702.05 |     2273.56 | -428.491 |     ms |
|                                  90th percentile service time |                                 sequence_2stage_nofilter_fetch1000_size_1000 |     2877.07 |     2363.45 | -513.628 |     ms |
|                                  99th percentile service time |                                 sequence_2stage_nofilter_fetch1000_size_1000 |     2997.72 |     2447.16 | -550.561 |     ms |
|                                 100th percentile service time |                                 sequence_2stage_nofilter_fetch1000_size_1000 |     3104.37 |     2464.96 | -639.407 |     ms |
|                                                    error rate |                                 sequence_2stage_nofilter_fetch1000_size_1000 |           0 |           0 |        0 |      % |
|                                                Min Throughput |                 sequence_2stage_nofilter_fetch1000_size1000_tail1000_head200 |     7.10992 |     10.2966 |  3.18671 |  ops/s |
|                                               Mean Throughput |                 sequence_2stage_nofilter_fetch1000_size1000_tail1000_head200 |     7.42643 |     10.4738 |  3.04734 |  ops/s |
|                                             Median Throughput |                 sequence_2stage_nofilter_fetch1000_size1000_tail1000_head200 |      7.4651 |     10.4905 |  3.02538 |  ops/s |
|                                                Max Throughput |                 sequence_2stage_nofilter_fetch1000_size1000_tail1000_head200 |     7.56932 |     10.5654 |  2.99604 |  ops/s |
|                                       50th percentile latency |                 sequence_2stage_nofilter_fetch1000_size1000_tail1000_head200 |     637.277 |     456.561 | -180.716 |     ms |
|                                       90th percentile latency |                 sequence_2stage_nofilter_fetch1000_size1000_tail1000_head200 |     681.997 |     490.836 | -191.161 |     ms |
|                                       99th percentile latency |                 sequence_2stage_nofilter_fetch1000_size1000_tail1000_head200 |     722.972 |     524.768 | -198.204 |     ms |
|                                      100th percentile latency |                 sequence_2stage_nofilter_fetch1000_size1000_tail1000_head200 |     743.856 |     578.278 | -165.578 |     ms |
|                                  50th percentile service time |                 sequence_2stage_nofilter_fetch1000_size1000_tail1000_head200 |     637.277 |     456.561 | -180.716 |     ms |
|                                  90th percentile service time |                 sequence_2stage_nofilter_fetch1000_size1000_tail1000_head200 |     681.997 |     490.836 | -191.161 |     ms |
|                                  99th percentile service time |                 sequence_2stage_nofilter_fetch1000_size1000_tail1000_head200 |     722.972 |     524.768 | -198.204 |     ms |
|                                 100th percentile service time |                 sequence_2stage_nofilter_fetch1000_size1000_tail1000_head200 |     743.856 |     578.278 | -165.578 |     ms |
|                                                    error rate |                 sequence_2stage_nofilter_fetch1000_size1000_tail1000_head200 |           0 |           0 |        0 |      % |
|                                                Min Throughput |          sequence_4stage_nofilter_maxspan5m_fetch1000_size100_head100_tail50 |     0.32833 |    0.327511 | -0.00082 |  ops/s |
|                                               Mean Throughput |          sequence_4stage_nofilter_maxspan5m_fetch1000_size100_head100_tail50 |    0.336481 |    0.335927 | -0.00055 |  ops/s |
|                                             Median Throughput |          sequence_4stage_nofilter_maxspan5m_fetch1000_size100_head100_tail50 |    0.336648 |      0.3363 | -0.00035 |  ops/s |
|                                                Max Throughput |          sequence_4stage_nofilter_maxspan5m_fetch1000_size100_head100_tail50 |    0.339835 |    0.339455 | -0.00038 |  ops/s |
|                                       50th percentile latency |          sequence_4stage_nofilter_maxspan5m_fetch1000_size100_head100_tail50 |     14694.3 |     14679.7 | -14.5586 |     ms |
|                                       90th percentile latency |          sequence_4stage_nofilter_maxspan5m_fetch1000_size100_head100_tail50 |     14869.8 |     14905.9 |  36.1597 |     ms |
|                                       99th percentile latency |          sequence_4stage_nofilter_maxspan5m_fetch1000_size100_head100_tail50 |     15058.7 |     15092.2 |    33.46 |     ms |
|                                      100th percentile latency |          sequence_4stage_nofilter_maxspan5m_fetch1000_size100_head100_tail50 |     15216.9 |     15172.2 | -44.6797 |     ms |
|                                  50th percentile service time |          sequence_4stage_nofilter_maxspan5m_fetch1000_size100_head100_tail50 |     14694.3 |     14679.7 | -14.5586 |     ms |
|                                  90th percentile service time |          sequence_4stage_nofilter_maxspan5m_fetch1000_size100_head100_tail50 |     14869.8 |     14905.9 |  36.1597 |     ms |
|                                  99th percentile service time |          sequence_4stage_nofilter_maxspan5m_fetch1000_size100_head100_tail50 |     15058.7 |     15092.2 |    33.46 |     ms |
|                                 100th percentile service time |          sequence_4stage_nofilter_maxspan5m_fetch1000_size100_head100_tail50 |     15216.9 |     15172.2 | -44.6797 |     ms |
|                                                    error rate |          sequence_4stage_nofilter_maxspan5m_fetch1000_size100_head100_tail50 |           0 |           0 |        0 |      % |
|                                                Min Throughput | sequence_3stage_startsWithfilter_maxspan30m_fetch1000_size100_tail100_head50 |     18.3705 |     22.5926 |  4.22208 |  ops/s |
|                                               Mean Throughput | sequence_3stage_startsWithfilter_maxspan30m_fetch1000_size100_tail100_head50 |       20.17 |     25.0813 |   4.9113 |  ops/s |
|                                             Median Throughput | sequence_3stage_startsWithfilter_maxspan30m_fetch1000_size100_tail100_head50 |     20.4233 |      25.341 |  4.91774 |  ops/s |
|                                                Max Throughput | sequence_3stage_startsWithfilter_maxspan30m_fetch1000_size100_tail100_head50 |     20.9296 |     26.1454 |  5.21577 |  ops/s |
|                                       50th percentile latency | sequence_3stage_startsWithfilter_maxspan30m_fetch1000_size100_tail100_head50 |     224.035 |      177.52 | -46.5144 |     ms |
|                                       90th percentile latency | sequence_3stage_startsWithfilter_maxspan30m_fetch1000_size100_tail100_head50 |     246.908 |     190.919 | -55.9884 |     ms |
|                                       99th percentile latency | sequence_3stage_startsWithfilter_maxspan30m_fetch1000_size100_tail100_head50 |     269.799 |     207.992 | -61.8068 |     ms |
|                                      100th percentile latency | sequence_3stage_startsWithfilter_maxspan30m_fetch1000_size100_tail100_head50 |     281.261 |     216.152 | -65.1094 |     ms |
|                                  50th percentile service time | sequence_3stage_startsWithfilter_maxspan30m_fetch1000_size100_tail100_head50 |     224.035 |      177.52 | -46.5144 |     ms |
|                                  90th percentile service time | sequence_3stage_startsWithfilter_maxspan30m_fetch1000_size100_tail100_head50 |     246.908 |     190.919 | -55.9884 |     ms |
|                                  99th percentile service time | sequence_3stage_startsWithfilter_maxspan30m_fetch1000_size100_tail100_head50 |     269.799 |     207.992 | -61.8068 |     ms |
|                                 100th percentile service time | sequence_3stage_startsWithfilter_maxspan30m_fetch1000_size100_tail100_head50 |     281.261 |     216.152 | -65.1094 |     ms |
|                                                    error rate | sequence_3stage_startsWithfilter_maxspan30m_fetch1000_size100_tail100_head50 |           0 |           0 |        0 |      % |

matriv
matriv approved these changes Feb 22, 2021
View reviewed changes
Copy link
Contributor

@matriv matriv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jimczi jimczi merged commit f27da75 into elastic:master Feb 22, 2021
@jimczi jimczi deleted the shard_doc_sort_optim branch February 22, 2021 16:52
jimczi added a commit that referenced this pull request Feb 22, 2021
@jimczi
Handle _shard_doc field for sort optimization (#69321)
bd4a585 
This commit ensures that the automatic tiebreaker `_shard_doc` does
not disable sort optimization.

Relates #56828
jimczi added a commit that referenced this pull request Feb 22, 2021
@jimczi
Handle _shard_doc field for sort optimization (#69321)
395bc05 
This commit ensures that the automatic tiebreaker `_shard_doc` does
not disable sort optimization.

Relates #56828
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@matriv matriv matriv approved these changes

Assignees
No one assigned
Labels
>non-issue :Search/Search Search-related issues that do not fall into other categories Team:Search Meta label for search team v7.12.0 v7.13.0 v8.0.0-alpha1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@jimczi @elasticmachine @matriv @jakelandis