Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wrongly converting columns to nullable may cause momory problem #431

Open
shuai-xu opened this issue Apr 13, 2023 · 0 comments · May be fixed by #433
Open

Wrongly converting columns to nullable may cause momory problem #431

shuai-xu opened this issue Apr 13, 2023 · 0 comments · May be fixed by #433

Comments

@shuai-xu
Copy link

Describe what's wrong
When I run a query with gouping set, It failed with exception java.lang.UnsupportedOperationException: Not enough spark off-heap execution memory. Acquired: 17592186700480, granted: 2143226266. The stack shows as following:
image
It took a long time for me to find that this is due to wrongly convert columns to nullable in ExpandTransform. This action may cause that data in Block is nullable, but the header in the next transform is not nullable, when seriaze the data and then deseriaze it, it will throw exception.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant