add test for skip_values in DictionaryDecoder and fix it #2105
Conversation
| @@ -346,6 +346,7 @@ where | |||
| // Keys will be validated on conversion to arrow | |||
| let keys_slice = keys.spare_capacity_mut(range.start + len); | |||
| let len = decoder.get_batch(&mut keys_slice[range.start..])?; | |||
| *max_remaining_values -= len; | |||
There was a problem hiding this comment.
miss this will fail at record skip count.
Codecov Report
@@ Coverage Diff @@
## master #2105 +/- ##
==========================================
+ Coverage 83.73% 83.74% +0.01%
==========================================
Files 225 225
Lines 59412 59458 +46
==========================================
+ Hits 49748 49796 +48
+ Misses 9664 9662 -2
Continue to review full report at Codecov.
|
|
|
||
| let valid = vec![true, true, true, true, true]; | ||
| let valid_buffer = Buffer::from_iter(valid.iter().cloned()); | ||
| output.pad_nulls(0, 5, 5, valid_buffer.as_slice()); |
There was a problem hiding this comment.
@tustvold i found without pad_nulls then call into_array will return empty.
Is there other way to change into array without padding🤔
There was a problem hiding this comment.
Yes, call BufferQueue::set_len, which is what RecordReader will do
|
Benchmark runs are scheduled for baseline = 8d7e2ae and contender = 3096591. 3096591 is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
Which issue does this PR close?
Related #2079
Rationale for this change
What changes are included in this PR?
Are there any user-facing changes?