Skip to content

Commit

Permalink
Add note about random sampler consistency (elastic#107479) (elastic#1…
Browse files Browse the repository at this point in the history
  • Loading branch information
benwtrent authored Apr 16, 2024
1 parent 3b02039 commit 46b4a73
Showing 1 changed file with 12 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,17 @@ higher sampling rates, the relative error is still low.

NOTE: This represents the result of aggregations against a typical positively skewed APM data set which also has outliers in the upper tail. The linear dependence of the relative error on the sample size is found to hold widely, but the slope depends on the variation in the quantity being aggregated. As such, the variance in your own data may
cause relative error rates to increase or decrease at a different rate.
[[random-sampler-consistency]]
==== Random sampler consistency

For a given `probability` and `seed`, the random sampler aggregation is consistent when sampling unchanged data from the same shard.
However, this is background random sampling if a particular document is included in the sampled set or not is dependent on current number of segments.

Meaning, replica vs. primary shards could return different values as different particular documents are sampled.

If the shard changes in via doc addition, update, deletion, or segment merging, the particular documents sampled could change, and thus the resulting statistics could change.

The resulting statistics used from the random sampler aggregation are approximate and should be treated as such.

[[random-sampler-special-cases]]
==== Random sampling special cases
Expand All @@ -105,6 +116,6 @@ for a bucket is `10,000` with `probability: 0.1`, the actual number of documents

An exception to this is <<search-aggregations-metrics-cardinality-aggregation, cardinality aggregation>>. Unique item
counts are not suitable for automatic scaling. When interpreting the cardinality count, compare it
to the number of sampled docs provided in the top level `doc_count` within the random_sampler aggregation. It gives
to the number of sampled docs provided in the top level `doc_count` within the random_sampler aggregation. It gives
you an idea of unique values as a percentage of total values. It may not reflect, however, the exact number of unique values
for the given field.

0 comments on commit 46b4a73

Please sign in to comment.