fix(ingest/bigquery): use correct row count in null count profiling c… #9123
Conversation
github-actions
bot
added
the
ingestion
| # may be slightly different (more or less) than configured `sample_size`. | ||
| # However not doing so to start with, as that adds another query overhead | ||
| # plus approximate metrics should work for sampling based profiling. | ||
| profile.rowCount = self.config.sample_size |
There was a problem hiding this comment.
Imo the row count should still ultimately reflect the full number of rows, and the size of the sample should be stored in the partition spec
There was a problem hiding this comment.
The actual table's rowCount here is always overridden in bigquery source from information_schema, whenever partitionSpec is set. partitionSpec is set whenever limit or sampling is used in profiling or only latest partition is used for partitioned tables. The computed rowCount in ge_data_profiler.py is used primarily for column level profiling computation only. (this is also the case for snowflake and redshift)
Here - https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/source/sql/sql_generic_profiler.py#L112
There was a problem hiding this comment.
I'm open to handling this only for null counts computation and not updating local profile aspect. Attempted changing profile aspect to make sure that we don't run into this for any other calculation - maybe in future.
There was a problem hiding this comment.
Yup that might result in more understandable code
At minimum, we need a comment here explaining why it's ok to override, but ideally we just tweak the null count / unique percentage calculations
hsheth2
added
the
merge-pending-ci
maggiehays
added
the
hacktoberfest-accepted
…omputation
Checklist