Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

large bitmap serialization failed because of overflow of size #3600

Closed
kangpinghuang opened this issue May 15, 2020 · 2 comments · Fixed by #3724
Closed

large bitmap serialization failed because of overflow of size #3600

kangpinghuang opened this issue May 15, 2020 · 2 comments · Fixed by #3724
Labels
api-review Categorizes an issue or PR as actively needing an API review. area/sql/execution Issues or PRs related to the execution engine kind/fix Categorizes issue or PR as related to a bug.

Comments

@kangpinghuang
Copy link
Contributor

Describe the bug
when use bitmap to do insertsect, if the bitmap is large than the MAX_INT, the allocation of memory from free pool will failed, which will lead to serialization failed because of core dump.

the following is the bitmap size I come to is 3G+.
@kangpinghuang
Copy link
Contributor Author

One of the reason is that 3G+ is large than MAX_INT, but FreePool->allocate accept int as argment, overflow will lead to allocation and serialation bug.

kangpinghuang added a commit to kangpinghuang/incubator-doris that referenced this issue May 15, 2020
@kangpinghuang
fix large string val allocation failure, apache#3600
c8dbd14 
Large bitmap will need use StringVal to allocate large memory, which is large than MAX_INT.
The overflow will cause serialization failure of bitmap.
@imay imay added area/sql/execution Issues or PRs related to the execution engine kind/fix Categorizes issue or PR as related to a bug. labels May 21, 2020
@imay
Copy link
Contributor

imay commented May 21, 2020

Hi, @kangpinghuang
Can you explain the scenario that cause this problem?
I think it will cause many problems if the body is so large. And if we are going to support it, we should review all the related places.

@imay imay added the api-review Categorizes an issue or PR as actively needing an API review. label May 21, 2020
EmmyMiao87 added a commit to EmmyMiao87/incubator-doris that referenced this issue May 29, 2020
@EmmyMiao87
Large bitmap will need use StringVal to allocate large memory, which …
efe3429 
…is large than MAX_INT.

The overflow will cause serialization failure of bitmap.

Fixed apache#3600

Change-Id: I720b9ea4646188a8e1402630601928b5f16fedb2
EmmyMiao87 added a commit to EmmyMiao87/incubator-doris that referenced this issue May 29, 2020
@EmmyMiao87
Large bitmap will need use StringVal to allocate large memory, which …
02b7b11 
…is large than MAX_INT.

The overflow will cause serialization failure of bitmap.

Fixed apache#3600

Change-Id: I720b9ea4646188a8e1402630601928b5f16fedb2
EmmyMiao87 added a commit to EmmyMiao87/incubator-doris that referenced this issue May 29, 2020
@EmmyMiao87
Large bitmap will need use StringVal to allocate large memory, which …
7bfaa73 
…is large than MAX_INT.

The overflow will cause serialization failure of bitmap.

Fixed apache#3600

Change-Id: I720b9ea4646188a8e1402630601928b5f16fedb2
EmmyMiao87 added a commit to EmmyMiao87/incubator-doris that referenced this issue May 29, 2020
@EmmyMiao87
Fix large string val allocation failure
c149175 
Large bitmap will need use StringVal to allocate large memory, which is large than MAX_INT.
The overflow will cause serialization failure of bitmap.

Fixed apache#3600

Change-Id: I720b9ea4646188a8e1402630601928b5f16fedb2
EmmyMiao87 added a commit to EmmyMiao87/incubator-doris that referenced this issue May 29, 2020
@EmmyMiao87
Fix large string val allocation failure
d78fe34 
Large bitmap will need use StringVal to allocate large memory, which is large than MAX_INT.
The overflow will cause serialization failure of bitmap.

Fixed apache#3600

Change-Id: I720b9ea4646188a8e1402630601928b5f16fedb2
@EmmyMiao87 EmmyMiao87 mentioned this issue May 29, 2020
Merged
EmmyMiao87 added a commit that referenced this issue Jun 3, 2020
@EmmyMiao87
Fix large string val allocation failure (#3724)
e16873a 
* Fix large string val allocation failure

Large bitmap will need use StringVal to allocate large memory, which is large than MAX_INT.
The overflow will cause serialization failure of bitmap.

Fixed #3600
morningman pushed a commit to morningman/doris that referenced this issue Jun 4, 2020
@EmmyMiao87 @morningman-cmy
Fix large string val allocation failure (apache#3724)
2022b8a 
* Fix large string val allocation failure

Large bitmap will need use StringVal to allocate large memory, which is large than MAX_INT.
The overflow will cause serialization failure of bitmap.

Fixed apache#3600
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
api-review Categorizes an issue or PR as actively needing an API review. area/sql/execution Issues or PRs related to the execution engine kind/fix Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix large string val allocation failure EmmyMiao87/incubator-doris
2 participants
@imay @kangpinghuang