feat: implement basic struct handling #91
Conversation
|
ACTION NEEDED Substrait follows the Conventional Commits The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. |
|
@mbrobbel Could you take a look at this PR please? |
| if not current_symbol: | ||
| raise InternalError( | ||
| f'Could not find plan id {self._current_plan_id} constructed earlier.') |
There was a problem hiding this comment.
Should we move this lookup code to a separate method? Because it's used in many places.
There was a problem hiding this comment.
I tried a few approaches here. All of this is added just to make pyright happy so I could just remove all of the checks and the code would run fine. I also considered moving the check into the code that does the lookup but there are a few cases where None is appropriate. I can remove these for now and can add back a better solution when a pyright github action is added.
There was a problem hiding this comment.
In that case I would be in favor of keeping them until we can improve.
Edit: looks like you've already removed them, maybe add a task to track adding these checks again?
There was a problem hiding this comment.
I did add an issue for adding pyright, that would require fixing all of the issues (about 100) mostly involving type|None mismatches.
|
Thanks for the review @mbrobbel ! Could you merge it please? Thanks! |
Adds the capability to create structs one level deep using the Spark struct() data frame API.
To assist with this functionality the field handling data structure has been upgraded from a
string reference into a full type (Field) allowing for tracking of additional names today and
precise type tracking in the future.
Future PRs will add arbitrary type adding and will make getField() work on structures.