Report zero values instead of unknown for empty ingest queries #15674
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- MSQ engine allows empty ingest by default. The statement response and task status API should show 0 rows instead of null rows. If worker counter snapshot is present but empty that denotes that 0 rows were processed. - Add tests for this scenario. - Fixup miscellaneous interpollation in QueryHostFinder (extra space, incorrect interpolations).
github-actions
bot
added
Area - Batch Ingestion
Area - MSQ
abhishekrb19
force-pushed
the
msq_fix_empty_counters
branch
from
ed13089
to
3adc996
Compare
abhishekrb19
force-pushed
the
msq_fix_empty_counters
branch
from
a356901
to
4a3b368
Compare
cryptoe
reviewed
Jan 16, 2024
...re/multi-stage-query/src/main/java/org/apache/druid/msq/util/SqlStatementResourceHelper.java
Outdated
Show resolved
Hide resolved
...re/multi-stage-query/src/main/java/org/apache/druid/msq/util/SqlStatementResourceHelper.java
Outdated
Show resolved
Hide resolved
|
@abhishekrb19 Could you also add a release note for the above PR. It helps out when release notes are compiled. |
LakshSingla
pushed a commit
to LakshSingla/druid
that referenced
this pull request
Jan 30, 2024
…e#15674) MSQ now allows empty ingest queries by default. For such queries that don't generate any output rows, the query counters in the async status result object/task report don't contain numTotalRows and totalSizeInBytes. These properties when not set/undefined can be confusing to API clients. For example, the web-console treats it as unknown values. This patch fixes the counters by explicitly reporting them as 0 instead of null for empty ingest queries.
abhishekagarwal87
pushed a commit
that referenced
this pull request
Jan 30, 2024
… (#15791) MSQ now allows empty ingest queries by default. For such queries that don't generate any output rows, the query counters in the async status result object/task report don't contain numTotalRows and totalSizeInBytes. These properties when not set/undefined can be confusing to API clients. For example, the web-console treats it as unknown values. This patch fixes the counters by explicitly reporting them as 0 instead of null for empty ingest queries. Co-authored-by: Abhishek Radhakrishnan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
MSQ now allows empty ingest queries by default. For such queries that don't generate any output rows, the query counters in the async status result object/task report don't contain
numTotalRowsandtotalSizeInBytes. These properties when not set/undefined can be confusing to API clients. For example, the web-console treats it as unknown values.This patch fixes the counters by explicitly reporting them as 0 instead of null for empty ingest queries.
SqlMSQStatementResourcePostTest.java. Validate the responses which now includeResultSetInformationwith 0 rows instead of null.DataSourceMSQDestinationandTaskReportMSQDestinationinSqlStatementResourceHelperTest.java. We don't report 0 values forDurableStorageDestinationtype as it only applies to select queries as it can be unknown, so the behavior is preserved as-is.QueryHostFinder.javathat I found while testing this patch.Before the fix, web-console interpreting as unknown number of rows:
After the fix:
Key changed/added classes in this PR
Release Note
Related to #15495, MSQE will report report zero values for
numTotalRowsandtotalSizeInBytes.This PR has: