Enable BloomFilter for _id of non-datastream indices #88409

dnhatn · 2022-07-10T22:33:02Z

This PR adds BloomFilter to Elasticsearch and enables it for the _id field of non-data stream indices. BloomFilter should speed up the performance of mget and update requests at a small expense of refresh, merge, and storage.

Fork Lucene's BloomFilter to support an on-disk format and integrate with CircuitBreaker (i.e., ensure no OOM when building BloomFilter). We will also have a special treatment to skip reading BloomFilter for the archive tier. See: cbb0800.
Enable BloomFilter for _id of non-data-stream indices: bc8c0d3#diff-5779b7efbee3f9220189f8bdf03991e6f23cdac6506c5924e31231d8d0b0314aR52

If we are okay with this approach, I will split this PR into two smaller PRs.

Benchmark results:

http_logs/update: 30% faster

|                                                        Metric |   Task |        Baseline |       Contender |       Diff |   Unit |   Diff % |
|--------------------------------------------------------------:|-------:|----------------:|----------------:|-----------:|-------:|---------:|
|                    Cumulative indexing time of primary shards |        |   204.488       |   134.085       |  -70.4033  |    min |  -34.43% |
|             Min cumulative indexing time across primary shard |        |     0           |     0           |    0       |    min |    0.00% |
|          Median cumulative indexing time across primary shard |        |     5.12353     |     4.30781     |   -0.81573 |    min |  -15.92% |
|             Max cumulative indexing time across primary shard |        |   173.89        |   108.429       |  -65.4613  |    min |  -37.65% |
|           Cumulative indexing throttle time of primary shards |        |     0           |     0           |    0       |    min |    0.00% |
|    Min cumulative indexing throttle time across primary shard |        |     0           |     0           |    0       |    min |    0.00% |
| Median cumulative indexing throttle time across primary shard |        |     0           |     0           |    0       |    min |    0.00% |
|    Max cumulative indexing throttle time across primary shard |        |     0           |     0           |    0       |    min |    0.00% |
|                       Cumulative merge time of primary shards |        |    51.1106      |    53.4685      |    2.35792 |    min |   +4.61% |
|                      Cumulative merge count of primary shards |        |    39           |    38           |   -1       |        |   -2.56% |
|                Min cumulative merge time across primary shard |        |     0           |     0           |    0       |    min |    0.00% |
|             Median cumulative merge time across primary shard |        |     0           |     0           |    0       |    min |    0.00% |
|                Max cumulative merge time across primary shard |        |    51.0884      |    53.3766      |    2.28818 |    min |   +4.48% |
|              Cumulative merge throttle time of primary shards |        |    33.221       |    32.3083      |   -0.91268 |    min |   -2.75% |
|       Min cumulative merge throttle time across primary shard |        |     0           |     0           |    0       |    min |    0.00% |
|    Median cumulative merge throttle time across primary shard |        |     0           |     0           |    0       |    min |    0.00% |
|       Max cumulative merge throttle time across primary shard |        |    33.221       |    32.3083      |   -0.91268 |    min |   -2.75% |
|                     Cumulative refresh time of primary shards |        |     3.9797      |     4.25122     |    0.27152 |    min |   +6.82% |
|                    Cumulative refresh count of primary shards |        |   240           |   236           |   -4       |        |   -1.67% |
|              Min cumulative refresh time across primary shard |        |     0           |     0           |    0       |    min |    0.00% |
|           Median cumulative refresh time across primary shard |        |     0.18635     |     0.197367    |    0.01102 |    min |   +5.91% |
|              Max cumulative refresh time across primary shard |        |     2.83907     |     3.11542     |    0.27635 |    min |   +9.73% |
|                       Cumulative flush time of primary shards |        |    19.2593      |    20.9628      |    1.70352 |    min |   +8.85% |
|                      Cumulative flush count of primary shards |        |   100           |   100           |    0       |        |    0.00% |
|                Min cumulative flush time across primary shard |        |     0           |     0           |    0       |    min |    0.00% |
|             Median cumulative flush time across primary shard |        |     0.553225    |     0.669417    |    0.11619 |    min |  +21.00% |
|                Max cumulative flush time across primary shard |        |    15.857       |    17.0961      |    1.23905 |    min |   +7.81% |
|                                       Total Young Gen GC time |        |   214.232       |   221.676       |    7.444   |      s |   +3.47% |
|                                      Total Young Gen GC count |        |  1510           |  1510           |    0       |        |    0.00% |
|                                         Total Old Gen GC time |        |     0           |     0           |    0       |      s |    0.00% |
|                                        Total Old Gen GC count |        |     0           |     0           |    0       |        |    0.00% |
|                                                    Store size |        |    15.5356      |    17.207       |    1.6714  |     GB |  +10.76% |
|                                                 Translog size |        |     4.09782e-07 |     4.09782e-07 |    0       |     GB |    0.00% |
|                                        Heap used for segments |        |     0           |     0           |    0       |     MB |    0.00% |
|                                      Heap used for doc values |        |     0           |     0           |    0       |     MB |    0.00% |
|                                           Heap used for terms |        |     0           |     0           |    0       |     MB |    0.00% |
|                                           Heap used for norms |        |     0           |     0           |    0       |     MB |    0.00% |
|                                          Heap used for points |        |     0           |     0           |    0       |     MB |    0.00% |
|                                   Heap used for stored fields |        |     0           |     0           |    0       |     MB |    0.00% |
|                                                 Segment count |        |   136           |   129           |   -7       |        |   -5.15% |
|                                   Total Ingest Pipeline count |        |     0           |     0           |    0       |        |    0.00% |
|                                    Total Ingest Pipeline time |        |     0           |     0           |    0       |     ms |    0.00% |
|                                  Total Ingest Pipeline failed |        |     0           |     0           |    0       |        |    0.00% |
|                                                Min Throughput | update | 17222.7         | 19471           | 2248.38    | docs/s |  +13.05% |
|                                               Mean Throughput | update | 25597.6         | 28594.9         | 2997.25    | docs/s |  +11.71% |
|                                             Median Throughput | update | 23691.3         | 26620.9         | 2929.63    | docs/s |  +12.37% |
|                                                Max Throughput | update | 38246.6         | 41607.9         | 3361.28    | docs/s |   +8.79% |
|                                       50th percentile latency | update |   102.637       |    80.426       |  -22.2111  |     ms |  -21.64% |
|                                       90th percentile latency | update |   186.097       |   158.1         |  -27.9972  |     ms |  -15.04% |
|                                       99th percentile latency | update |   356.333       |   329.398       |  -26.9353  |     ms |   -7.56% |
|                                     99.9th percentile latency | update |  1038.69        |  1061.98        |   23.2905  |     ms |   +2.24% |
|                                    99.99th percentile latency | update |  8934.49        |  9564.69        |  630.199   |     ms |   +7.05% |
|                                      100th percentile latency | update | 15070.3         | 17995.3         | 2925.01    |     ms |  +19.41% |
|                                  50th percentile service time | update |   102.637       |    80.426       |  -22.2111  |     ms |  -21.64% |
|                                  90th percentile service time | update |   186.097       |   158.1         |  -27.9972  |     ms |  -15.04% |
|                                  99th percentile service time | update |   356.333       |   329.398       |  -26.9353  |     ms |   -7.56% |
|                                99.9th percentile service time | update |  1038.69        |  1061.98        |   23.2905  |     ms |   +2.24% |
|                               99.99th percentile service time | update |  8934.49        |  9564.69        |  630.199   |     ms |   +7.05% |
|                                 100th percentile service time | update | 15070.3         | 17995.3         | 2925.01    |     ms |  +19.41% |
|                                                    error rate | update |     0           |     0           |    0       |      % |    0.00% |

geonames/append-fast-with-conflicts: 11% faster

|                                                        Metric |         Task |        Baseline |       Contender |        Diff |   Unit |   Diff % |
|--------------------------------------------------------------:|-------------:|----------------:|----------------:|------------:|-------:|---------:|
|                    Cumulative indexing time of primary shards |              |    11.5674      |    10.2602      |    -1.30725 |    min |  -11.30% |
|             Min cumulative indexing time across primary shard |              |     1.79325     |     1.58932     |    -0.20393 |    min |  -11.37% |
|          Median cumulative indexing time across primary shard |              |     1.92863     |     1.69722     |    -0.23141 |    min |  -12.00% |
|             Max cumulative indexing time across primary shard |              |     2.11288     |     1.84133     |    -0.27155 |    min |  -12.85% |
|           Cumulative indexing throttle time of primary shards |              |     0           |     0           |     0       |    min |    0.00% |
|    Min cumulative indexing throttle time across primary shard |              |     0           |     0           |     0       |    min |    0.00% |
| Median cumulative indexing throttle time across primary shard |              |     0           |     0           |     0       |    min |    0.00% |
|    Max cumulative indexing throttle time across primary shard |              |     0           |     0           |     0       |    min |    0.00% |
|                       Cumulative merge time of primary shards |              |     0           |     0           |     0       |    min |    0.00% |
|                      Cumulative merge count of primary shards |              |     0           |     0           |     0       |        |    0.00% |
|                Min cumulative merge time across primary shard |              |     0           |     0           |     0       |    min |    0.00% |
|             Median cumulative merge time across primary shard |              |     0           |     0           |     0       |    min |    0.00% |
|                Max cumulative merge time across primary shard |              |     0           |     0           |     0       |    min |    0.00% |
|              Cumulative merge throttle time of primary shards |              |     0           |     0           |     0       |    min |    0.00% |
|       Min cumulative merge throttle time across primary shard |              |     0           |     0           |     0       |    min |    0.00% |
|    Median cumulative merge throttle time across primary shard |              |     0           |     0           |     0       |    min |    0.00% |
|       Max cumulative merge throttle time across primary shard |              |     0           |     0           |     0       |    min |    0.00% |
|                     Cumulative refresh time of primary shards |              |     2.97885     |     3.04907     |     0.07022 |    min |   +2.36% |
|                    Cumulative refresh count of primary shards |              |    51           |    53           |     2       |        |   +3.92% |
|              Min cumulative refresh time across primary shard |              |     0.459       |     0.45675     |    -0.00225 |    min |   -0.49% |
|           Median cumulative refresh time across primary shard |              |     0.49        |     0.508175    |     0.01818 |    min |   +3.71% |
|              Max cumulative refresh time across primary shard |              |     0.554917    |     0.54805     |    -0.00687 |    min |   -1.24% |
|                       Cumulative flush time of primary shards |              |     0           |     0           |     0       |    min |    0.00% |
|                      Cumulative flush count of primary shards |              |     0           |     0           |     0       |        |    0.00% |
|                Min cumulative flush time across primary shard |              |     0           |     0           |     0       |    min |    0.00% |
|             Median cumulative flush time across primary shard |              |     0           |     0           |     0       |    min |    0.00% |
|                Max cumulative flush time across primary shard |              |     0           |     0           |     0       |    min |    0.00% |
|                                       Total Young Gen GC time |              |     2.623       |     2.864       |     0.241   |      s |   +9.19% |
|                                      Total Young Gen GC count |              |    60           |    65           |     5       |        |   +8.33% |
|                                         Total Old Gen GC time |              |     0           |     0           |     0       |      s |    0.00% |
|                                        Total Old Gen GC count |              |     0           |     0           |     0       |        |    0.00% |
|                                                    Store size |              |     2.62687     |     2.65214     |     0.02527 |     GB |   +0.96% |
|                                                 Translog size |              |     3.07336e-07 |     3.07336e-07 |     0       |     GB |    0.00% |
|                                        Heap used for segments |              |     0           |     0           |     0       |     MB |    0.00% |
|                                      Heap used for doc values |              |     0           |     0           |     0       |     MB |    0.00% |
|                                           Heap used for terms |              |     0           |     0           |     0       |     MB |    0.00% |
|                                           Heap used for norms |              |     0           |     0           |     0       |     MB |    0.00% |
|                                          Heap used for points |              |     0           |     0           |     0       |     MB |    0.00% |
|                                   Heap used for stored fields |              |     0           |     0           |     0       |     MB |    0.00% |
|                                                 Segment count |              |    62           |    65           |     3       |        |   +4.84% |
|                                   Total Ingest Pipeline count |              |     0           |     0           |     0       |        |    0.00% |
|                                    Total Ingest Pipeline time |              |     0           |     0           |     0       |     ms |    0.00% |
|                                  Total Ingest Pipeline failed |              |     0           |     0           |     0       |        |    0.00% |
|                                                Min Throughput | index-update | 55712.1         | 58663.4         |  2951.27    | docs/s |   +5.30% |
|                                               Mean Throughput | index-update | 59827.6         | 63974.9         |  4147.34    | docs/s |   +6.93% |
|                                             Median Throughput | index-update | 60077.9         | 64345.7         |  4267.81    | docs/s |   +7.10% |
|                                                Max Throughput | index-update | 62332.9         | 65957.4         |  3624.41    | docs/s |   +5.81% |
|                                       50th percentile latency | index-update |    18.2543      |    17.7749      |    -0.47942 |     ms |   -2.63% |
|                                       90th percentile latency | index-update |    25.993       |    25.738       |    -0.25502 |     ms |   -0.98% |
|                                       99th percentile latency | index-update |    96.5377      |   105.799       |     9.26124 |     ms |   +9.59% |
|                                     99.9th percentile latency | index-update |  2374.35        |   655.37        | -1718.98    |     ms |  -72.40% |
|                                      100th percentile latency | index-update | 11559.5         | 14077.1         |  2517.68    |     ms |  +21.78% |
|                                  50th percentile service time | index-update |    18.2543      |    17.7749      |    -0.47942 |     ms |   -2.63% |
|                                  90th percentile service time | index-update |    25.993       |    25.738       |    -0.25502 |     ms |   -0.98% |
|                                  99th percentile service time | index-update |    96.5377      |   105.799       |     9.26124 |     ms |   +9.59% |
|                                99.9th percentile service time | index-update |  2374.35        |   655.37        | -1718.98    |     ms |  -72.40% |
|                                 100th percentile service time | index-update | 11559.5         | 14077.1         |  2517.68    |     ms |  +21.78% |
|                                                    error rate | index-update |     0           |     0           |     0       |      % |    0.00% |

http_logs/append-no-conflicts-index-only: 4% faster

|                                                        Metric |         Task |         Baseline |        Contender |       Diff |   Unit |   Diff % |
|--------------------------------------------------------------:|-------------:|-----------------:|-----------------:|-----------:|-------:|---------:|
|                    Cumulative indexing time of primary shards |              |     94.639       |     90.2686      |   -4.3704  |    min |   -4.62% |
|             Min cumulative indexing time across primary shard |              |      0           |      0           |    0       |    min |    0.00% |
|          Median cumulative indexing time across primary shard |              |      0.854917    |      0.799267    |   -0.05565 |    min |   -6.51% |
|             Max cumulative indexing time across primary shard |              |     14.4983      |     13.7474      |   -0.75092 |    min |   -5.18% |
|           Cumulative indexing throttle time of primary shards |              |      0           |      0           |    0       |    min |    0.00% |
|    Min cumulative indexing throttle time across primary shard |              |      0           |      0           |    0       |    min |    0.00% |
| Median cumulative indexing throttle time across primary shard |              |      0           |      0           |    0       |    min |    0.00% |
|    Max cumulative indexing throttle time across primary shard |              |      0           |      0           |    0       |    min |    0.00% |
|                       Cumulative merge time of primary shards |              |     10.655       |     11.2547      |    0.59968 |    min |   +5.63% |
|                      Cumulative merge count of primary shards |              |     29           |     28           |   -1       |        |   -3.45% |
|                Min cumulative merge time across primary shard |              |      0           |      0           |    0       |    min |    0.00% |
|             Median cumulative merge time across primary shard |              |      0           |      0           |    0       |    min |    0.00% |
|                Max cumulative merge time across primary shard |              |      2.39942     |      2.57625     |    0.17683 |    min |   +7.37% |
|              Cumulative merge throttle time of primary shards |              |      4.43008     |      4.80807     |    0.37798 |    min |   +8.53% |
|       Min cumulative merge throttle time across primary shard |              |      0           |      0           |    0       |    min |    0.00% |
|    Median cumulative merge throttle time across primary shard |              |      0           |      0           |    0       |    min |    0.00% |
|       Max cumulative merge throttle time across primary shard |              |      0.992817    |      1.1482      |    0.15538 |    min |  +15.65% |
|                     Cumulative refresh time of primary shards |              |      8.6277      |      9.14297     |    0.51527 |    min |   +5.97% |
|                    Cumulative refresh count of primary shards |              |    467           |    456           |  -11       |        |   -2.36% |
|              Min cumulative refresh time across primary shard |              |      0           |      0           |    0       |    min |    0.00% |
|           Median cumulative refresh time across primary shard |              |      0.135583    |      0.146125    |    0.01054 |    min |   +7.78% |
|              Max cumulative refresh time across primary shard |              |      1.55802     |      1.69002     |    0.132   |    min |   +8.47% |
|                       Cumulative flush time of primary shards |              |      8.83267     |      9.4904      |    0.65773 |    min |   +7.45% |
|                      Cumulative flush count of primary shards |              |    123           |    123           |    0       |        |    0.00% |
|                Min cumulative flush time across primary shard |              |      0           |      0           |    0       |    min |    0.00% |
|             Median cumulative flush time across primary shard |              |      0.0178583   |      0.017375    |   -0.00048 |    min |   -2.71% |
|                Max cumulative flush time across primary shard |              |      2.29292     |      2.23348     |   -0.05943 |    min |   -2.59% |
|                                       Total Young Gen GC time |              |      8.215       |      8.511       |    0.296   |      s |   +3.60% |
|                                      Total Young Gen GC count |              |    757           |    744           |  -13       |        |   -1.72% |
|                                         Total Old Gen GC time |              |      0           |      0           |    0       |      s |    0.00% |
|                                        Total Old Gen GC count |              |      0           |      0           |    0       |        |    0.00% |
|                                                    Store size |              |     16.469       |     16.8836      |    0.41457 |     GB |   +2.52% |
|                                                 Translog size |              |      2.04891e-06 |      2.04891e-06 |    0       |     GB |    0.00% |
|                                        Heap used for segments |              |      0           |      0           |    0       |     MB |    0.00% |
|                                      Heap used for doc values |              |      0           |      0           |    0       |     MB |    0.00% |
|                                           Heap used for terms |              |      0           |      0           |    0       |     MB |    0.00% |
|                                           Heap used for norms |              |      0           |      0           |    0       |     MB |    0.00% |
|                                          Heap used for points |              |      0           |      0           |    0       |     MB |    0.00% |
|                                   Heap used for stored fields |              |      0           |      0           |    0       |     MB |    0.00% |
|                                                 Segment count |              |    278           |    296           |   18       |        |   +6.47% |
|                                   Total Ingest Pipeline count |              |      0           |      0           |    0       |        |    0.00% |
|                                    Total Ingest Pipeline time |              |      0           |      0           |    0       |     ms |    0.00% |
|                                  Total Ingest Pipeline failed |              |      0           |      0           |    0       |        |    0.00% |
|                                                Min Throughput | index-append | 128322           | 128995           |  673.199   | docs/s |   +0.52% |
|                                               Mean Throughput | index-append | 131048           | 134264           | 3216.09    | docs/s |   +2.45% |
|                                             Median Throughput | index-append | 131401           | 134944           | 3542.48    | docs/s |   +2.70% |
|                                                Max Throughput | index-append | 132607           | 138927           | 6319.52    | docs/s |   +4.77% |
|                                       50th percentile latency | index-append |     11.7579      |     11.7469      |   -0.011   |     ms |   -0.09% |
|                                       90th percentile latency | index-append |     14.3791      |     14.6843      |    0.30518 |     ms |   +2.12% |
|                                       99th percentile latency | index-append |     23.3543      |     23.7754      |    0.4211  |     ms |   +1.80% |
|                                     99.9th percentile latency | index-append |    230.45        |    248.695       |   18.245   |     ms |   +7.92% |
|                                    99.99th percentile latency | index-append |   6502.86        |   7400.32        |  897.454   |     ms |  +13.80% |
|                                      100th percentile latency | index-append |  10307           |  10792.7         |  485.707   |     ms |   +4.71% |
|                                  50th percentile service time | index-append |     11.7579      |     11.7469      |   -0.011   |     ms |   -0.09% |
|                                  90th percentile service time | index-append |     14.3791      |     14.6843      |    0.30518 |     ms |   +2.12% |
|                                  99th percentile service time | index-append |     23.3543      |     23.7754      |    0.4211  |     ms |   +1.80% |
|                                99.9th percentile service time | index-append |    230.45        |    248.695       |   18.245   |     ms |   +7.92% |
|                               99.99th percentile service time | index-append |   6502.86        |   7400.32        |  897.454   |     ms |  +13.80% |
|                                 100th percentile service time | index-append |  10307           |  10792.7         |  485.707   |     ms |   +4.71% |
|                                                    error rate | index-append |      0.000463411 |      0.00327129  |    0.00281 |      % | +605.91% |

geonames/append-no-conflicts-index-only: 4% slower

|                                                        Metric |         Task |        Baseline |       Contender |        Diff |   Unit |   Diff % |
|--------------------------------------------------------------:|-------------:|----------------:|----------------:|------------:|-------:|---------:|
|                    Cumulative indexing time of primary shards |              |     9.71393     |    10.1481      |     0.43418 |    min |   +4.47% |
|             Min cumulative indexing time across primary shard |              |     1.79678     |     1.87212     |     0.07533 |    min |   +4.19% |
|          Median cumulative indexing time across primary shard |              |     2.02563     |     2.11442     |     0.08878 |    min |   +4.38% |
|             Max cumulative indexing time across primary shard |              |     2.05453     |     2.13473     |     0.0802  |    min |   +3.90% |
|           Cumulative indexing throttle time of primary shards |              |     0           |     0           |     0       |    min |    0.00% |
|    Min cumulative indexing throttle time across primary shard |              |     0           |     0           |     0       |    min |    0.00% |
| Median cumulative indexing throttle time across primary shard |              |     0           |     0           |     0       |    min |    0.00% |
|    Max cumulative indexing throttle time across primary shard |              |     0           |     0           |     0       |    min |    0.00% |
|                       Cumulative merge time of primary shards |              |     0           |     0           |     0       |    min |    0.00% |
|                      Cumulative merge count of primary shards |              |     0           |     0           |     0       |        |    0.00% |
|                Min cumulative merge time across primary shard |              |     0           |     0           |     0       |    min |    0.00% |
|             Median cumulative merge time across primary shard |              |     0           |     0           |     0       |    min |    0.00% |
|                Max cumulative merge time across primary shard |              |     0           |     0           |     0       |    min |    0.00% |
|              Cumulative merge throttle time of primary shards |              |     0           |     0           |     0       |    min |    0.00% |
|       Min cumulative merge throttle time across primary shard |              |     0           |     0           |     0       |    min |    0.00% |
|    Median cumulative merge throttle time across primary shard |              |     0           |     0           |     0       |    min |    0.00% |
|       Max cumulative merge throttle time across primary shard |              |     0           |     0           |     0       |    min |    0.00% |
|                     Cumulative refresh time of primary shards |              |     1.9438      |     2.14998     |     0.20618 |    min |  +10.61% |
|                    Cumulative refresh count of primary shards |              |    54           |    52           |    -2       |        |   -3.70% |
|              Min cumulative refresh time across primary shard |              |     0.222633    |     0.266867    |     0.04423 |    min |  +19.87% |
|           Median cumulative refresh time across primary shard |              |     0.429133    |     0.476483    |     0.04735 |    min |  +11.03% |
|              Max cumulative refresh time across primary shard |              |     0.518733    |     0.602817    |     0.08408 |    min |  +16.21% |
|                       Cumulative flush time of primary shards |              |     0.79775     |     1.05328     |     0.25553 |    min |  +32.03% |
|                      Cumulative flush count of primary shards |              |     5           |     5           |     0       |        |    0.00% |
|                Min cumulative flush time across primary shard |              |     0.10165     |     0.144867    |     0.04322 |    min |  +42.52% |
|             Median cumulative flush time across primary shard |              |     0.142433    |     0.2163      |     0.07387 |    min |  +51.86% |
|                Max cumulative flush time across primary shard |              |     0.2252      |     0.297917    |     0.07272 |    min |  +32.29% |
|                                       Total Young Gen GC time |              |     1.032       |     1.135       |     0.103   |      s |   +9.98% |
|                                      Total Young Gen GC count |              |    48           |    51           |     3       |        |   +6.25% |
|                                         Total Old Gen GC time |              |     0           |     0           |     0       |      s |    0.00% |
|                                        Total Old Gen GC count |              |     0           |     0           |     0       |        |    0.00% |
|                                                    Store size |              |     2.63287     |     2.64842     |     0.01555 |     GB |   +0.59% |
|                                                 Translog size |              |     2.56114e-07 |     2.56114e-07 |     0       |     GB |    0.00% |
|                                        Heap used for segments |              |     0           |     0           |     0       |     MB |    0.00% |
|                                      Heap used for doc values |              |     0           |     0           |     0       |     MB |    0.00% |
|                                           Heap used for terms |              |     0           |     0           |     0       |     MB |    0.00% |
|                                           Heap used for norms |              |     0           |     0           |     0       |     MB |    0.00% |
|                                          Heap used for points |              |     0           |     0           |     0       |     MB |    0.00% |
|                                   Heap used for stored fields |              |     0           |     0           |     0       |     MB |    0.00% |
|                                                 Segment count |              |    84           |    83           |    -1       |        |   -1.19% |
|                                   Total Ingest Pipeline count |              |     0           |     0           |     0       |        |    0.00% |
|                                    Total Ingest Pipeline time |              |     0           |     0           |     0       |     ms |    0.00% |
|                                  Total Ingest Pipeline failed |              |     0           |     0           |     0       |        |    0.00% |
|                                                Min Throughput | index-append | 65646.2         | 63341.9         | -2304.33    | docs/s |   -3.51% |
|                                               Mean Throughput | index-append | 70545           | 66032.9         | -4512.07    | docs/s |   -6.40% |
|                                             Median Throughput | index-append | 70773.1         | 65718.9         | -5054.22    | docs/s |   -7.14% |
|                                                Max Throughput | index-append | 72382.5         | 68391.2         | -3991.26    | docs/s |   -5.51% |
|                                       50th percentile latency | index-append |    18.2187      |    18.9626      |     0.74386 |     ms |   +4.08% |
|                                       90th percentile latency | index-append |    22.3928      |    24.868       |     2.47521 |     ms |  +11.05% |
|                                       99th percentile latency | index-append |    72.5306      |    68.4397      |    -4.09095 |     ms |   -5.64% |
|                                     99.9th percentile latency | index-append |   260.41        |   463.791       |   203.381   |     ms |  +78.10% |
|                                      100th percentile latency | index-append |  9553.07        |  9428.72        |  -124.35    |     ms |   -1.30% |
|                                  50th percentile service time | index-append |    18.2187      |    18.9626      |     0.74386 |     ms |   +4.08% |
|                                  90th percentile service time | index-append |    22.3928      |    24.868       |     2.47521 |     ms |  +11.05% |
|                                  99th percentile service time | index-append |    72.5306      |    68.4397      |    -4.09095 |     ms |   -5.64% |
|                                99.9th percentile service time | index-append |   260.41        |   463.791       |   203.381   |     ms |  +78.10% |
|                                 100th percentile service time | index-append |  9553.07        |  9428.72        |  -124.35    |     ms |   -1.30% |
|                                                    error rate | index-append |     0           |     0           |     0       |      % |    0.00% |

elasticsearchmachine · 2022-07-12T14:05:54Z

Hi @dnhatn, I've created a changelog YAML for you.

server/src/main/java/org/elasticsearch/index/codec/bloomfilter/BloomFilterPostingsFormat.java

…/BloomFilterPostingsFormat.java Co-authored-by: Adrien Grand <[email protected]>

dnhatn · 2022-07-14T02:19:28Z

@jpountz Thank you for the review. I have addressed your comments and added an index setting in 22c6a0e.

elasticmachine · 2022-07-14T02:19:34Z

Pinging @elastic/es-search (Team:Search)

jpountz

I left some suggestions but the change looks good to me.

server/src/main/java/org/elasticsearch/index/IndexSettings.java

jpountz · 2022-07-29T08:53:53Z

server/src/main/java/org/elasticsearch/index/codec/PerFieldMapperCodec.java

+        if (IdFieldMapper.NAME.equals(field)
+            && mapperService.mappingLookup().isDataStreamTimestampFieldEnabled() == false
+            && IndexSettings.BLOOM_FILTER_ID_FIELD_ENABLED_SETTING.get(mapperService.getIndexSettings().getSettings())) {
+            return new ES84BloomFilterPostingsFormat(super.getPostingsFormatForField(field), bigArrays);


Nit: Every instance that is not the same ends up in its own file. It doesn't matter in practice here since we're enabling bloom filters on a single field, but it's a better practice to have the bloom filter format as a member on this class that is initialized upon initialization and to return this shared instance here for the id field?

...r/src/main/java/org/elasticsearch/index/codec/bloomfilter/ES84BloomFilterPostingsFormat.java

.../test/java/org/elasticsearch/index/codec/bloomfilter/ES84BloomFilterPostingsFormatTests.java

…/ES84BloomFilterPostingsFormat.java Co-authored-by: Adrien Grand <[email protected]>

dnhatn · 2022-08-01T14:40:01Z

@jpountz I pushed 8518341 to share a single BloomFilterPostingsFormat instance between multiple fields. Can you take another look? Thank you!

dnhatn · 2022-08-08T15:13:49Z

@jpountz Thanks for reviews.

elasticsearchmachine added the v8.4.0 label Jul 10, 2022

dnhatn added 2 commits July 11, 2022 23:19

Add BloomFilterPostingsFormat

cbb0800

Enable BloomFilter for _id of non-datastream indices

bc8c0d3

dnhatn force-pushed the bloom_filter branch from 3c3cc22 to bc8c0d3 Compare July 12, 2022 03:20

dnhatn added :Search/Search Search-related issues that do not fall into other categories >enhancement team-discuss labels Jul 12, 2022

Update docs/changelog/88409.yaml

cd82bb2

jpountz reviewed Jul 12, 2022

View reviewed changes

dnhatn and others added 13 commits July 12, 2022 13:30

Update server/src/main/java/org/elasticsearch/index/codec/bloomfilter…

082605c

…/BloomFilterPostingsFormat.java Co-authored-by: Adrien Grand <[email protected]>

Update server/src/main/java/org/elasticsearch/index/codec/bloomfilter…

e7d18bf

…/BloomFilterPostingsFormat.java Co-authored-by: Adrien Grand <[email protected]>

Update server/src/main/java/org/elasticsearch/index/codec/bloomfilter…

554c5b0

…/BloomFilterPostingsFormat.java Co-authored-by: Adrien Grand <[email protected]>

Fix initialize toCloses size

7ca0708

Remove TODO

f24426c

Use murmurhash3_x86_32

861dde4

typo: field -> file

002beed

BloomFilterPostingsFormat -> ES84BloomFilterPostingsFormat

0fabf37

retrieveChecksum

62aedf9

fix license

291043a

Add index settings

22c6a0e

Merge remote-tracking branch 'elastic/main' into bloom_filter

9baa588

fix test

87bb8ff

dnhatn marked this pull request as ready for review July 14, 2022 02:19

elasticmachine added the Team:Search Meta label for search team label Jul 14, 2022

dnhatn requested a review from jpountz July 14, 2022 02:19

dnhatn removed the team-discuss label Jul 18, 2022

elasticsearchmachine changed the base branch from master to main July 22, 2022 23:05

mark-vieira added v8.5.0 and removed v8.4.0 labels Jul 27, 2022

Merge branch 'main' into bloom_filter

e84f037

jpountz approved these changes Jul 29, 2022

View reviewed changes

dnhatn and others added 7 commits July 29, 2022 10:03

Update server/src/main/java/org/elasticsearch/index/codec/bloomfilter…

a402f86

…/ES84BloomFilterPostingsFormat.java Co-authored-by: Adrien Grand <[email protected]>

rename to ES85

5356d51

Mark setting as deprecated_warning

a1bc3cf

Add hash test

eb868eb

fully check index file

ab24abc

Merge remote-tracking branch 'elastic/main' into bloom_filter

cfc05d6

Support multiple fields

8518341

dnhatn requested a review from jpountz August 1, 2022 14:40

dnhatn merged commit cfad420 into elastic:main Aug 8, 2022

dnhatn deleted the bloom_filter branch August 8, 2022 15:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable BloomFilter for _id of non-datastream indices #88409

Enable BloomFilter for _id of non-datastream indices #88409

dnhatn commented Jul 10, 2022 •

edited

Loading

elasticsearchmachine commented Jul 12, 2022

dnhatn commented Jul 14, 2022

elasticmachine commented Jul 14, 2022

jpountz left a comment

jpountz Jul 29, 2022

dnhatn commented Aug 1, 2022 •

edited

Loading

dnhatn commented Aug 8, 2022

Enable BloomFilter for _id of non-datastream indices #88409

Enable BloomFilter for _id of non-datastream indices #88409

Conversation

dnhatn commented Jul 10, 2022 • edited Loading

Benchmark results:

elasticsearchmachine commented Jul 12, 2022

dnhatn commented Jul 14, 2022

elasticmachine commented Jul 14, 2022

jpountz left a comment

Choose a reason for hiding this comment

jpountz Jul 29, 2022

Choose a reason for hiding this comment

dnhatn commented Aug 1, 2022 • edited Loading

dnhatn commented Aug 8, 2022

dnhatn commented Jul 10, 2022 •

edited

Loading

dnhatn commented Aug 1, 2022 •

edited

Loading