query exceeded per-node user memory after killed,resource not freed and service crashed

[ic4y]

If a query’s resource occupation exceeds the resource of the node (Query exceeded per-node user memory limit of XXGB) , the query will be killed.

At this time, if you continue to submit the query that takes up a lot of resources (but it can be successfully executed normally). There will be insufficient resources, leading to continuous Major GC until the node hangs.

I suspect this is caused by the fact that the resource was not released after the query was killed.

The screenshot is a single machine test environment, and we also encountered such a problem in our production cluster.

[ic4y]

I need your help. Why is that

[sopel39]

Hi @ic4y

Which Trino version is this? There have been some improvements for releasing memory in operators recently.

[ic4y]

Hi @sopel39 thanks for the reply.

I found this problem in both prestoSQl-322 and trino-353 versions.
Which version of the improvements you mentioned was done? I can do the corresponding test.

[sopel39]

Actually it will be in 359, see #8170 (comment)

[ic4y]

@sopel39 Thanks . this solves the problem