Check if a map contains a specific key [skip ci] #8209
Conversation
firestarman
added
non-breaking
firestarman
changed the title
| @@ -127,6 +127,35 @@ get_gather_map_for_map_values(column_view const &input, string_scalar &lookup_ke | |||
| } // namespace | |||
|
|
|||
| namespace jni { | |||
|
|
|||
| std::unique_ptr<column> map_contains(column_view const &map_column, string_scalar lookup_key, | |||
There was a problem hiding this comment.
I think it's better to reuse this function in map_lookup instead of duplicating codes.
| * @param has_nulls Whether the input column might contain null list-rows, or null keys. | ||
| * @param stream The CUDA stream | ||
| * @param mr The device memory resource to be used for allocations | ||
| * @return A string_view column with the value from the first match in each list. |
There was a problem hiding this comment.
The comment for return is not correct.
Codecov Report
@@ Coverage Diff @@
## branch-0.20 #8209 +/- ##
===============================================
- Coverage 82.88% 82.88% -0.01%
===============================================
Files 103 104 +1
Lines 17668 17907 +239
===============================================
+ Hits 14645 14843 +198
- Misses 3023 3064 +41
Continue to review full report at Codecov.
|
wjxiz1992
changed the title
|
|
||
| auto gather_map = has_nulls ? |
There was a problem hiding this comment.
I think there is a better alternative here. We have a list<struct<key, value>> column and we want to find whether or not each row contains the incoming key. That to me sounds like cudf::contains() if we were passing a list(string) column where the strings were just the keys. It should be possible to construct a fake column_view here that gives us this structure. Roughly:
lists_column_view lcv(map_column);
structs_column_view scv(lcv.child());
std::vector<column_view> children;
children.push_back(lcv.offsets()); // offsets
children.push_back(scv.child(0)); // keys (a string column)
column_view list_of_keys(map_column.type(), map_column.size(),
nullptr, map_column.null_mask(), map_column.null_count(), 0, children);
@mythrocks is my thinking here about contains() sound?
There was a problem hiding this comment.
I think there is a better alternative here. We have a
list<struct<key, value>>column and we want to find whether or not each row contains the incoming key. That to me sounds likecudf::contains()if we were passing alist(string)column where the strings were just the keys. It should be possible to construct a fake column_view here that gives us this structure. Roughly:lists_column_view lcv(map_column); structs_column_view scv(lcv.child()); std::vector<column_view> children; children.push_back(lcv.offsets()); // offsets children.push_back(scv.child(0)); // keys (a string column) column_view list_of_keys(map_column.type(), map_column.size(), nullptr, map_column.null_mask(), map_column.null_count(), 0, children);@mythrocks is my thinking here about
contains()sound?
Thank you Dave, I've updated the code according to cudf::contains().
The only change here is I manully added a null_mask(0) for the contains_column. To make nulls count in all_aggregation.
Please help review, thanks!
There was a problem hiding this comment.
@nvdbaranec, that makes perfect sense. Thank you for the suggestion.
I'll need to examine the null mask issue more closely before I can comment.
| List<HostColumnVector.StructData> list4 = Arrays.asList(new HostColumnVector.StructData("a", "g")); | ||
| List<HostColumnVector.StructData> list5 = Arrays.asList(new HostColumnVector.StructData("f", "h")); | ||
| List<HostColumnVector.StructData> list6 = Arrays.asList(new HostColumnVector.StructData("a", null)); | ||
| List<HostColumnVector.StructData> list7 = Arrays.asList(new HostColumnVector.StructData(null, null)); |
There was a problem hiding this comment.
Since: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/types/MapType.scala#L27
this case seems not useful, but still keep it here since we manually added a null_mask(0) for null values.
|
@gpucibot merge |
To close #8120
As required in Spark 3.1.1, when ANSI mode is enabled, GetMapValue should throw an exception when the key is not found in the map in a row.
So plugin side needs to check if a map column contains the specific key in all rows.
The new added method
mapContainsin this PR should return a column of boolean, where false means key is not found.