Fork TDigest library #96086

kkrik-es · 2023-05-15T06:45:04Z

The TDigest library is used for percentile aggregations in ES. The library contains two underlying implementations: (a) one using an AVL tree to maintain a sorted set of centroids and (b) one using pre-allocated arrays of centroids that get sorted and merged repeatedly as more samples are added. ES uses (a) by default, but
early results show that switching to the merging implementation can offers speedups exceeding 10x in percentile calculations.

The currently used library (v.3.2) has bugs in the merging implementation that were later addressed, and the latest available version (v.3.3) provides inaccurate percentile calculations even for simple median calls. We are therefore forking the latest version and addressing these shortcomings for both implementations, before switching to the merging implementation. This PR includes fixes for MergingDigest, but keeps using AVLTreeDigest in TDigestState. The switch to MergingDigest will happen in a subsequent PR, to better track the impact of each change in isolation.

Major changes compared to the initial implementation:

Quantile calculation uses interpolation between centroids instead of rounding, for singleton centroids
Quantile calculation properly tracks the centroid range (off-by-one error fixed)
Smaller sizes for weight and mean buffer arrays in MergingDigest, to reduce memory pressure
Centroid retrieval in MergingDigest triggers merging (if needed), as it is used when writing to StreamOutput
Call compress() on TDigestState objects when passed to aggregation objects, so that the former are immutable throughout the lifetime of the latter - needed for thread safety
No tracking of added data, other than through centroids
Removal of misfiring asserts and dead code, esp. around Serialization
Style changes to match ES codebase requirements

The top 2 changes lead to slightly different results, compared to the previous implementation from v.3.2. Still, they lead to more accurate quantile calculations. The change is thus deemed acceptable and will be documented in the release notes.

More work needed for TDigestPercentile*Tests and the TDigestTest (and the rest of the tests) in the tdigest lib to pass.

github-actions · 2023-05-15T06:45:22Z

Documentation preview:

✨ Changed pages

elasticsearchmachine · 2023-05-15T06:45:50Z

Hi @kkrik-es, I've created a changelog YAML for you.

Remove wrong asserts from tests and MergingDigest.

Remove redundant serializing interfaces from the library.

These tests don't address compatibility issues in mixed cluster tests as the latter contain a mix of older and newer nodes, so the output depends on which node is picked as a data node since the forked TDigest library is not backwards compatible (produces slightly different results).

martijnvg · 2023-05-23T08:29:21Z

...her/src/test/java/org/elasticsearch/xpack/watcher/notification/email/HtmlSanitizerTests.java

@@ -82,7 +82,7 @@ public void testAllowStyles() {

    public void testDefaultFormattingAllowed() {
        String html = "<b></b><i></i><s></s><u></u><o></o><sup></sup><sub></sub><ins></ins><del></del><strong></strong>"
-            + "<strike></strike><tt></tt><code></code><big></big><small></small><span></span><br /><em></em><hr />";
+            + "<strike></strike><code></code><code></code><big></big><small></small><span></span><br /><em></em><hr />";


Why is this change required?

Unrelated, reverted.

martijnvg · 2023-05-23T08:33:46Z

server/src/main/java/org/elasticsearch/search/aggregations/metrics/TDigestState.java

 */
-public class TDigestState extends AVLTreeDigest {
+public class TDigestState extends MergingDigest {


So the wire format between avl and merging can stay the same?

Right, serialization is handled by this class so there are no compatibility issues - other than the changes in quantile calculations.

martijnvg · 2023-05-23T08:36:05Z

qa/mixed-cluster/build.gradle

@@ -63,5 +63,19 @@ BuildParams.bwcVersions.withWireCompatible { bwcVersion, baseName ->
    tasks.register(bwcTaskName(bwcVersion)) {
      dependsOn "${baseName}#mixedClusterTest"
    }
+
+    tasks.named("${baseName}#mixedClusterTest").configure {


I think we should use skip versions instead? I think there are other qa tests that reuse the rest api spec tests too.
This should work, since we test new behaviour and not old behavior.

We should also ensure that at least a base percentile and percentile_ranks tests are ran in mixed cluster tests.

Looks like we're doing both. I saw a number of ml tests that are using skip in the YAML. We should align on one or the other.

Right, this has been reverted.

martijnvg · 2023-05-23T08:37:07Z

benchmarks/src/main/java/org/elasticsearch/benchmark/tdigest/ApproxLogBench.java

+ * computing the asin function involved in the merge is expensive. This argues for collecting more samples
+ * before sorting and merging them into the digest.
+ */
+@BenchmarkMode(Mode.AverageTime)


Should we fork all these micro benchmarks?

I kept just one for the tdigest and one for sorting, in case they come handy.

jpountz

I only looked at AVLTreeDigest, which is the bit I'm familiar with. One thing looks a bit fishy, but other than that it looks good to me.

libs/tdigest/src/main/java/org/elasticsearch/tdigest/AVLTreeDigest.java

libs/tdigest/src/test/java/org/elasticsearch/tdigest/BigCount.java

martijnvg

I think is ready to be merged - LGTM

libs/tdigest/src/main/java/org/elasticsearch/tdigest/AVLGroupTree.java

benchmarks/src/main/java/org/elasticsearch/benchmark/tdigest/TDigestBench.java

While this helps with the case where the digest contains only singletons (perfect accuracy), it has a major issue problem (non-monotonic quantile function) when the first singleton is followed by a non-singleton centroid. It's preferable to revert to the old version from 3.2; inaccuracies in a singleton-only digest should be mitigated by using a sorted array for small sample counts.

This is due to restoring quantile functions to match head.

Update Dist.cdf to use interpolation, use the same cdf version in AVLTreeDigest and MergingDigest.

This reverts commit 7718dbb.

kkrik-es · 2023-06-12T15:44:35Z

@elasticsearchmachine run elasticsearch-ci/*

mark-vieira

LGTM

mark-vieira · 2023-06-12T18:06:53Z

libs/tdigest/build.gradle

+ * under the License.
+ */
+apply plugin: 'elasticsearch.build'
+apply plugin: 'elasticsearch.publish'


Here's an example PR: https://github.com/elastic/infra/pull/38270/files

martijnvg · 2023-06-13T08:45:16Z

nice! 🚀

## Summary Closes #159615 This started faling most possibly because ES upgraded the TDigest library used for percentiles elastic/elasticsearch#96086 and now we can have more accurate results. Runner https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2397

kkrik-es added 4 commits May 9, 2023 10:23

Initial import for TDigest forking.

3c6c424

Fix MedianTest.

d719767

More work needed for TDigestPercentile*Tests and the TDigestTest (and the rest of the tests) in the tdigest lib to pass.

Fix Dist.

e816b57

Fix AVLTreeDigest.quantile to match Dist for uniform centroids.

0079943

kkrik-es added >feature :Analytics/Aggregations Aggregations Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) labels May 15, 2023

kkrik-es self-assigned this May 15, 2023

elasticsearchmachine added the v8.9.0 label May 15, 2023

kkrik-es and others added 17 commits May 15, 2023 09:45

Update docs/changelog/96086.yaml

0a18e70

Fix MergingDigest.quantile to match Dist on uniform distribution.

96ea462

Add merging to TDigestState.hashCode and .equals.

56350ee

Remove wrong asserts from tests and MergingDigest.

Fix style violations for tdigest library.

d731aee

Fix typo.

18c1cd5

Fix more style violations.

f96b4d2

Fix more style violations.

6d0201b

Fix remaining style violations in tdigest library.

9d48856

Update results in docs based on the forked tdigest.

68e489f

Fix YAML tests in aggs module.

2b3b463

Fix YAML tests in x-pack/plugin.

0d6ea1e

Skip failing V7 compat tests in modules/aggregations.

98c1146

Fix TDigest library unittests.

d49ed67

Remove redundant serializing interfaces from the library.

Fix test failures in docs and mixed cluster.

5d9ae8e

Reduce buffer sizes in MergingDigest to avoid oom.

1ad14f8

Exclude more failing V7 compatibility tests.

af93c26

martijnvg reviewed May 23, 2023

View reviewed changes

Update results for JdbcCsvSpecIT tests.

d82ec3e

jpountz reviewed Jun 7, 2023

View reviewed changes

kkrik-es added 2 commits June 8, 2023 11:14

Small fixes.

063ef27

Merge branch 'main' into fix/95903

c028d9c

martijnvg approved these changes Jun 9, 2023

View reviewed changes

libs/tdigest/src/main/java/org/elasticsearch/tdigest/AVLGroupTree.java Outdated Show resolved Hide resolved

benchmarks/src/main/java/org/elasticsearch/benchmark/tdigest/TDigestBench.java Outdated Show resolved Hide resolved

kkrik-es added 2 commits June 9, 2023 16:51

Add javadoc and tests.

dc739af

kkrik-es removed the :Analytics/Aggregations Aggregations label Jun 10, 2023

elasticsearchmachine removed the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Jun 10, 2023

kkrik-es added 4 commits June 10, 2023 13:17

Revert changes to expected values in tests.

079a8c3

This is due to restoring quantile functions to match head.

Revert changes to expected values in tests.

47414c7

This is due to restoring quantile functions to match head.

Merge branch 'main' into fix/95903

b8f7c00

Tentatively restore percentile rank expected results.

7718dbb

jpountz approved these changes Jun 12, 2023

View reviewed changes

kkrik-es added 3 commits June 12, 2023 17:48

Use cdf version from 3.2

4df40da

Update Dist.cdf to use interpolation, use the same cdf version in AVLTreeDigest and MergingDigest.

Revert "Tentatively restore percentile rank expected results."

77c13eb

This reverts commit 7718dbb.

Revert remaining changes compared to main.

4724905

kkrik-es added 2 commits June 12, 2023 19:50

Revert excluded V7 compat tests.

64950c4

Exclude V7 compat tests still failing.

7251150

mark-vieira approved these changes Jun 12, 2023

View reviewed changes

Exclude V7 compat tests still failing.

4a78005

kkrik-es mentioned this pull request Jun 13, 2023

Make TDigestState configurable #96787

Closed

Restore bySize function in TDigest and subclasses.

97ad6a9

kkrik-es merged commit 67211be into elastic:main Jun 13, 2023

kkrik-es mentioned this pull request Jun 13, 2023

Make TDigestState configurable #96794

Merged

stratoula mentioned this pull request Jun 19, 2023

[Visualizations] Unskips precalculater histogram FTs elastic/kibana#159887

Merged

martijnvg mentioned this pull request Jun 19, 2023

Consider upgrading to T-Digest 3.3 #73928

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fork TDigest library #96086

Fork TDigest library #96086

kkrik-es commented May 15, 2023 •

edited

Loading

github-actions bot commented May 15, 2023

elasticsearchmachine commented May 15, 2023

martijnvg May 23, 2023

kkrik-es May 23, 2023

martijnvg May 23, 2023

kkrik-es May 23, 2023

martijnvg May 23, 2023

martijnvg May 23, 2023

mark-vieira May 24, 2023

kkrik-es May 30, 2023

martijnvg May 23, 2023

kkrik-es May 30, 2023

jpountz left a comment

martijnvg left a comment

kkrik-es commented Jun 12, 2023

mark-vieira left a comment

mark-vieira Jun 12, 2023

martijnvg commented Jun 13, 2023

Fork TDigest library #96086

Fork TDigest library #96086

Conversation

kkrik-es commented May 15, 2023 • edited Loading

github-actions bot commented May 15, 2023

elasticsearchmachine commented May 15, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jpountz left a comment

Choose a reason for hiding this comment

martijnvg left a comment

Choose a reason for hiding this comment

kkrik-es commented Jun 12, 2023

mark-vieira left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martijnvg commented Jun 13, 2023

kkrik-es commented May 15, 2023 •

edited

Loading