Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Coral-Schema: Return nullable schema for inner fields created by operator #369

Merged
merged 1 commit into from
Mar 17, 2023

Conversation

ljfgem
Copy link
Collaborator

@ljfgem ljfgem commented Mar 17, 2023

Change Summary

At present, we are using RelDataType.isNullable() to determine if an inner field should be nullable or not while converting RelDataType to Avro schema, but Spark expects all the inner fields created by operator to be nullable. This PR is to make Coral-Schema return nullable schema for inner fields created by operator to be aligned with Spark's expectation.

Tests

  1. Modified UTs, these changed fields are created by operator, they are nullable now with this PR
  2. Regression test on all the prod views, no translation regression
  3. Tested on the affected prod views, can be queried on Spark shell with this PR

Copy link
Contributor

@yiqiangin yiqiangin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. It looks like this change is done for Spark. Pls make sure it does work for other query engines as well.

@ljfgem ljfgem merged commit 806b18e into linkedin:master Mar 17, 2023
@ljfgem
Copy link
Collaborator Author

ljfgem commented Mar 17, 2023

LGTM. It looks like this change is done for Spark. Pls make sure it does work for other query engines as well.

Thanks for review! Coral-Schema is only used for view queries on Spark at present.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants