Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ESQL: Block access benchmark #102872

Merged
merged 26 commits into from
Dec 20, 2023

Conversation

alex-spies
Copy link
Contributor

@alex-spies alex-spies commented Dec 1, 2023

Benchmark how long it takes to access elements of IntBlocks, BooleanBlocks, etc.

This helps guard against performance regressions when refactoring Block and Vector classes.

@alex-spies alex-spies force-pushed the esql-block-access-benchmark branch from f40ba93 to 14726e6 Compare December 1, 2023 15:22
Copy link
Member

@nik9000 nik9000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like the right direction, yeah!

@alex-spies alex-spies added the :Analytics/ES|QL AKA ESQL label Dec 19, 2023
@alex-spies alex-spies marked this pull request as ready for review December 19, 2023 18:16
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-ql (Team:QL)

@elasticsearchmachine elasticsearchmachine added the Team:QL (Deprecated) Meta label for query languages team label Dec 19, 2023
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/elasticsearch-esql (:Query Languages/ES|QL)

@alex-spies
Copy link
Contributor Author

Ran the bench on my machine to get a first idea:

Result "org.elasticsearch.benchmark.compute.operator.BlockBenchmark.run":
  2.161 ±(99.9%) 0.501 ns/op [Average]
  (min, avg, max) = (1.981, 2.161, 2.403), stdev = 0.222
  CI (99.9%): [1.660, 2.662] (assumes normal distribution)


# Run complete. Total time: 01:52:55

REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.

Benchmark           (accessType)             (dataTypeAndBlockKind)  Mode  Cnt   Score   Error  Units
BlockBenchmark.run    sequential                      boolean/array  avgt    7   1.432 ± 0.022  ns/op
BlockBenchmark.run    sequential      boolean/array-multivalue-null  avgt    7   0.841 ± 0.004  ns/op
BlockBenchmark.run    sequential  boolean/big-array-multivalue-null  avgt    7   1.711 ± 0.107  ns/op
BlockBenchmark.run    sequential                     boolean/vector  avgt    7   2.292 ± 0.408  ns/op
BlockBenchmark.run    sequential           boolean/vector-big-array  avgt    7   3.261 ± 0.014  ns/op
BlockBenchmark.run    sequential               boolean/vector-const  avgt    7   1.983 ± 0.547  ns/op
BlockBenchmark.run    sequential                     BytesRef/array  avgt    7   8.471 ± 0.090  ns/op
BlockBenchmark.run    sequential     BytesRef/array-multivalue-null  avgt    7   6.654 ± 0.047  ns/op
BlockBenchmark.run    sequential                    BytesRef/vector  avgt    7   8.442 ± 0.074  ns/op
BlockBenchmark.run    sequential              BytesRef/vector-const  avgt    7   2.720 ± 0.010  ns/op
BlockBenchmark.run    sequential                       double/array  avgt    7   1.748 ± 0.015  ns/op
BlockBenchmark.run    sequential       double/array-multivalue-null  avgt    7   0.797 ± 0.005  ns/op
BlockBenchmark.run    sequential   double/big-array-multivalue-null  avgt    7   1.313 ± 0.005  ns/op
BlockBenchmark.run    sequential                      double/vector  avgt    7   2.298 ± 0.399  ns/op
BlockBenchmark.run    sequential            double/vector-big-array  avgt    7   3.100 ± 0.032  ns/op
BlockBenchmark.run    sequential                double/vector-const  avgt    7   2.141 ± 0.502  ns/op
BlockBenchmark.run    sequential                          int/array  avgt    7   1.319 ± 0.006  ns/op
BlockBenchmark.run    sequential          int/array-multivalue-null  avgt    7   0.753 ± 0.005  ns/op
BlockBenchmark.run    sequential      int/big-array-multivalue-null  avgt    7   1.315 ± 0.076  ns/op
BlockBenchmark.run    sequential                         int/vector  avgt    7   2.320 ± 0.522  ns/op
BlockBenchmark.run    sequential               int/vector-big-array  avgt    7   3.061 ± 0.012  ns/op
BlockBenchmark.run    sequential                   int/vector-const  avgt    7   2.237 ± 0.244  ns/op
BlockBenchmark.run    sequential                         long/array  avgt    7   1.343 ± 0.015  ns/op
BlockBenchmark.run    sequential         long/array-multivalue-null  avgt    7   0.831 ± 0.006  ns/op
BlockBenchmark.run    sequential     long/big-array-multivalue-null  avgt    7   1.394 ± 0.009  ns/op
BlockBenchmark.run    sequential                        long/vector  avgt    7   2.351 ± 0.459  ns/op
BlockBenchmark.run    sequential              long/vector-big-array  avgt    7   3.072 ± 0.015  ns/op
BlockBenchmark.run    sequential                  long/vector-const  avgt    7   2.156 ± 0.508  ns/op
BlockBenchmark.run        random                      boolean/array  avgt    7   1.490 ± 0.009  ns/op
BlockBenchmark.run        random      boolean/array-multivalue-null  avgt    7   0.779 ± 0.003  ns/op
BlockBenchmark.run        random  boolean/big-array-multivalue-null  avgt    7   1.300 ± 0.005  ns/op
BlockBenchmark.run        random                     boolean/vector  avgt    7   2.396 ± 0.300  ns/op
BlockBenchmark.run        random           boolean/vector-big-array  avgt    7   3.277 ± 0.013  ns/op
BlockBenchmark.run        random               boolean/vector-const  avgt    7   2.002 ± 0.533  ns/op
BlockBenchmark.run        random                     BytesRef/array  avgt    7  10.405 ± 0.087  ns/op
BlockBenchmark.run        random     BytesRef/array-multivalue-null  avgt    7   7.277 ± 0.070  ns/op
BlockBenchmark.run        random                    BytesRef/vector  avgt    7   9.598 ± 0.157  ns/op
BlockBenchmark.run        random              BytesRef/vector-const  avgt    7   2.403 ± 0.017  ns/op
BlockBenchmark.run        random                       double/array  avgt    7   2.013 ± 0.011  ns/op
BlockBenchmark.run        random       double/array-multivalue-null  avgt    7   1.085 ± 0.005  ns/op
BlockBenchmark.run        random   double/big-array-multivalue-null  avgt    7   1.703 ± 0.042  ns/op
BlockBenchmark.run        random                      double/vector  avgt    7   3.099 ± 0.017  ns/op
BlockBenchmark.run        random            double/vector-big-array  avgt    7   4.055 ± 0.028  ns/op
BlockBenchmark.run        random                double/vector-const  avgt    7   2.155 ± 0.517  ns/op
BlockBenchmark.run        random                          int/array  avgt    7   1.630 ± 0.011  ns/op
BlockBenchmark.run        random          int/array-multivalue-null  avgt    7   0.869 ± 0.017  ns/op
BlockBenchmark.run        random      int/big-array-multivalue-null  avgt    7   1.438 ± 0.009  ns/op
BlockBenchmark.run        random                         int/vector  avgt    7   2.572 ± 0.147  ns/op
BlockBenchmark.run        random               int/vector-big-array  avgt    7   3.434 ± 0.032  ns/op
BlockBenchmark.run        random                   int/vector-const  avgt    7   2.249 ± 0.244  ns/op
BlockBenchmark.run        random                         long/array  avgt    7   2.035 ± 0.015  ns/op
BlockBenchmark.run        random         long/array-multivalue-null  avgt    7   1.062 ± 0.008  ns/op
BlockBenchmark.run        random     long/big-array-multivalue-null  avgt    7   1.654 ± 0.059  ns/op
BlockBenchmark.run        random                        long/vector  avgt    7   3.046 ± 0.011  ns/op
BlockBenchmark.run        random              long/vector-big-array  avgt    7   4.083 ± 0.024  ns/op
BlockBenchmark.run        random                  long/vector-const  avgt    7   2.161 ± 0.501  ns/op

(Note: I didn't take particular precautions to guard against noise, didn't disable optimizations, didn't bind to single CPU, so we could also get more precise results.)

Copy link
Contributor

@ChrisHegarty ChrisHegarty left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nik9000
Copy link
Member

nik9000 commented Dec 20, 2023

Looks useful! Let's land it. I imagine we'll play with this a lot over the next few years.

@alex-spies alex-spies merged commit 812686a into elastic:main Dec 20, 2023
15 checks passed
@alex-spies alex-spies deleted the esql-block-access-benchmark branch December 20, 2023 13:57
navarone-feekery pushed a commit to navarone-feekery/elasticsearch that referenced this pull request Dec 22, 2023
Add a micro benchmark for access times to individual block elements.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Analytics/ES|QL AKA ESQL >non-issue Team:QL (Deprecated) Meta label for query languages team v8.13.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants