TiDB gets OOM killed on pruning hash partitions

[zyguan]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

drop table if exists tbl_28;
create table `tbl_28` (`col_209` bigint(20) not null default '-4413003002508764546',`col_210` double not null default '570.6896586720441',`col_211` decimal(61,30) not null default '9939',primary key (`col_209`) /*t![clustered_index] clustered */,unique key `idx_68` (`col_211`,`col_210`,`col_209`),key `idx_69` (`col_210`)) engine=innodb default charset=utf8 collate=utf8_general_ci partition by hash( `col_209` ) partitions 6;
select /*+ stream_agg() */ bit_or( col_209 ) aggcol from (select   * from tbl_28 where not( tbl_28.col_211 <= 26.76 ) and not( tbl_28.col_209 in ( 6043174761261718958 ) ) and tbl_28.col_209 between -7622884926923238988 and 4467923861679270794 order by col_209  ) ordered_tbl group by col_210,col_211 order by aggcol for update;

2. What did you expect to see? (Required)

The last query succeed.

3. What did you see instead (Required)

Connection refused because tidb-server get oom-killed.

4. What is your TiDB version? (Required)

release-5.1 (8d62202)

[ChenPeng2013]

It seems like #25598

[XuHuaiyu]

@zyguan Log or the source data is needed.

[zyguan]

@zyguan Log or the source data is needed.

@XuHuaiyu The issue can be reproduced on release-5.0. Memory is allocated at here, it seems posLow and posHigh were evaled incorrectly. You can also find the tmp-storage here.

[tiancaiamao]

It seems like #25598

Not the same cause @ChenPeng2013

between -7622884926923238988 and 4467923861679270794

Current hash pruning is toooooooooooo inefficient for this case...
It calculate the range and then take every point data from the range to check if it fit in any partition

[tiancaiamao]

A simple fix here is to change the used from a []int to a map, that would reduce the allocation. (but not avoid the repeated calculation)

[qw4990]

It can be solved by #25599. I forgot to pick it into v5.1. I'll pick it soon.

[tiancaiamao]

It can be solved by #25599. I forgot to pick it into v5.1. I'll pick it soon.

Not the same problem... #25599 is overflow, but this one is the problem of the algorithm

[qw4990]

It can be solved by #25599. I forgot to pick it into v5.1. I'll pick it soon.

Not the same problem... #25599 is overflow, but this one is the problem of the algorithm

Emm, I cannot reproduce this bug after picking #26471.
I inspected the stack of this issue yesterday, and found it was also stuck in the same loop as #25599:

Could you check it again at your convenience~ @tiancaiamao

[tiancaiamao]

I can't reproduce this bug too, so it maybe cause by the overflow ...

We can leave the used = append(used, int(idx)) as an enhancement when [posLow, posHigh] range is too large.

Since it can't reproduce, I think we can close this issue. @zyguan @qw4990

[ti-srebot]

Please edit this comment or add a new comment to complete the following information

Not a bug

1. Remove the 'type/bug' label
2. Add notes to indicate why it is not a bug

Duplicate bug

1. Add the 'type/duplicate' label
2. Add the link to the original bug

Bug

Note: Make Sure that 'component', and 'severity' labels are added
Example for how to fill out the template: #20100

1. Root Cause Analysis (RCA) (optional)

2. Symptom (optional)

3. All Trigger Conditions (optional)

4. Workaround (optional)

5. Affected versions

6. Fixed versions