TiFlash crash randomly after cluster restart when TLS is enabled

[guo-shaoge]

Bug Report

Looks very like: grpc/grpc#23144

1. Minimal reproduce step (Required)

1. start a cluster of v5.4.0 and enable TLS, there should be multiple tiflash nodes(the more the better)
2. load tpch100/20
3. run query like select count(*) from (select count(*) from lineitem group by l_orderkey, l_suppkey)d;
4. restart cluster

repeat step 3 and step 4 multiple time, TiFlash will crash

2. What did you expect to see? (Required)

run normally

3. What did you see instead (Required)

TiFlash will crash

4. What is your TiFlash version? (Required)

v5.4.0

[LittleFall]

the root cause may be the compatible issue of grpc and system openssl. To fix it, we will cherry-pick the new build process to 5.x.

[LittleFall]

I will do the cherry-pick of 5.0/5.1/5.2/5.3

[zanmato1984]

Thanks for the great work of @s3nt3, the root cause has been confirmed to be the openssl error stack issue.

Grpc has fixed this issue in grpc/grpc#26834, v1.41.0.

[VelocityLight]

update

[zanmato1984]

update

[zanmato1984]

All PR merged to affected branches. Closing.