-
Notifications
You must be signed in to change notification settings - Fork 821
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
(DEMO) tests pass when validate_full
is forced
#987
Conversation
} | ||
}; | ||
|
||
// Bug discovery mechanism: call validate_full here |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is the demo -- it is not for merging, but for demonstrating that all the tests pass even when we run validate_full()
on them
validate_full
is forcedvalidate_full
is forced
ca5a429
to
72906d5
Compare
@@ -670,6 +670,7 @@ mod tests { | |||
} | |||
|
|||
#[test] | |||
#[ignore] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This test fails as follows:
---- arrow::arrow_reader::tests::test_read_maps stdout ----
thread 'arrow::arrow_reader::tests::test_read_maps' panicked at 'called `Result::unwrap()` on an `Err` value: InvalidArgumentError("Child type mismatch for Struct([Field { name: \"key\", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }, Field { name: \"value\", data_type: Map(Field { name: \"key_value\", data_type: Struct([Field { name: \"key\", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }, Field { name: \"value\", data_type: Boolean, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }, false), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: None }]). Expected Map(Field { name: \"key_value\", data_type: Struct([Field { name: \"key\", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }, Field { name: \"value\", data_type: Boolean, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }, false) but child data had Map(Field { name: \"key_value\", data_type: Struct([Field { name: \"key\", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }, Field { name: \"value\", data_type: Boolean, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: None }, false)")', arrow/src/array/data.rs:308:34
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
(the issue is the nullability of the child field is declared differently on the Struct's datatype than on the ArrayData which I think is an actual bug in the array reader, and will file a follow on ticket)
Codecov Report
@@ Coverage Diff @@
## master #987 +/- ##
==========================================
- Coverage 82.31% 82.31% -0.01%
==========================================
Files 168 168
Lines 48763 49008 +245
==========================================
+ Hits 40139 40339 +200
- Misses 8624 8669 +45
Continue to review full report at Codecov.
|
This PR has served its purpose. RIP |
Built on #921
This PR demonstrates that (most) of the tests pass if I (temporarily) force
ArrayData::new_unchecked()
to check all inputs, which gives me additional confidence thatvalidate_full()
is validating inputs as it supposed to.The only one that fails is related to the arrow_reader not having its nullability set correctly which I think is a bug in the array reader and which I will file separately