Infer the count of maximum distinct values from min/max #3837
Conversation
isidentical
force-pushed
the
gh-3813-distinct-calc
branch
from
0ce00ff to
b265204
Compare
|
Benchmark runs are scheduled for baseline = e02376d and contender = fe0000e. fe0000e is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
Which issue does this PR close?
Part of #3813.
Rationale for this change
This was a point that came out during the initial join cardinality computation PR (link) where the logic only gave an estimate when the distinct count information was available directly in the statistics. This was effective for certain use cases where the distinct count was already calculated (e.g. propagated statistics from stuff like aggregates) but for statistics that originate from initial user input, having
distinct_countis very unlikely (e.g. there is no way to save distinct count when exporting a parquet file from pandas, none of the official backends [pyarrow/fastparquet] even support such a thing in their write APIs). So one main thing we can do is actually usemin/maxvalues (which are nearly universal at this point) to calculate the maximum possible distinct count (which is actually what we need for selectivity).What changes are included in this PR?
A fallback option for inferring the maximum distinct count when the actual distinct count information is not available. It only works with numeric values (more specifically, integers) at this point (we can technically determine the range for timestamps or floats, but neither of them feels close to accurate since that would be essentially brute forcing every possible value within the precision boundaries, something that feels very unlikely to happen in real world, but open for discussion).
Are there any user-facing changes?
No backwards incompatible changes.