Elasticsearch queries with scroll stopped working

[crspl]

Elasticsearch version: 7.6.2
Plugins installed: -
JVM version: shipped with ES (openjdk version "13.0.2" 2020-01-14)
OS: Centos 7 (kernel 3.10.0-862.14.4.el7.x86_64)

Bug description:
Queries with 'scroll' feature stopped working even on very small indices and with v.high value for search.max_open_scroll_context (5000000). Getting error 500 'Trying to create too many scroll contexts' even if node stats doesn't show any open contexts or "scroll current".

Steps to reproduce:
No idea unfortunately. But it started when several heavy searches using scroll were running at the end of month. Initially it worked well but then it stopped and doesn't work anymore for any index/query.

ES_BUG_DEBUG.txt

[cbuescher]

Hi @crspl! Your question is better suited to the support forums over at https://discuss.elastic.co. We prefer to use Github issues only for bug reports and feature requests, and we think it's more likely this is a question than a bug report. I'm closing this as an issue here, but encourage you to ask this question in the forum instead. Many thanks.

[crspl]

Hi @cbuescher . I believe it's a bug and not lack of knowledge on our side. As I've written all was working fine and stopped suddenly after some time and now even very simple queries with "scroll" on idle cluster return 500. Problem is I cannot give you step by step instruction how to reproduce. But before I raised issue here I've asked on forum. Here's my thread -> https://discuss.elastic.co/t/did-i-hit-bug-related-to-scroll-and-too-many-scroll-contexts/230954 Please consider reopen this. I'm happy to give you more debug info.

[crspl]

one more update... this morning I've restarted nodes in our cluster one by one. Checked "scroll" query every few and ending up it started working (after 6 node out of 8) and no more 500 'Trying to create too many scroll contexts' exception

[wushuaiwangyin]

This is a bug. Openscrollcontexts is used to control the maximum scroll context. But by accessing the "_nodes/ data:true/stats/indices/search" get open_context does not reach the limit, which can be modified“ search.max_open_scroll_context "to the maximum value of int.

[sourcelliu]

This is a bug. Encountered this problem in our production environment.
Elasticsearch version: 7.6.2.

_Openscrollcontexts is used to control the maximum scroll context. But by accessing the "nodes/ data:true/stats/indices/search" get open_context does not reach the limit, which can be modified“ search.max_open_scroll_context "to the maximum value of int.

[doowonee]

Is this bug has been fixed? I'm in the exactly same situation. So I wonder which version should I use then?

[jerryjune]

After some source code research and debug. This is a confirmed bug and fixed after 7.7.0 release. This commit #53449 fixed this problem. And the walk around solution is set search.max_open_scroll_context to the maximum value of int like @mantuliu did.

[jerryjune]

This is a bug. Encountered this problem in our production environment.
Elasticsearch version: 7.6.2.

_Openscrollcontexts is used to control the maximum scroll context. But by accessing the "nodes/ data:true/stats/indices/search" get open_context does not reach the limit, which can be modified“ search.max_open_scroll_context "to the maximum value of int.

After source code analysis, because of the bug, AtomicInteger openScrollContexts will be decreased to negative and finally overflow to Int_Max, this will let openScrollContexts > maxOpenScrollContext always true and cannot create scroll anymore. This is why set search.max_open_scroll_context to the maximum value of int can work.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[dnhatn]

Thanks everyone for the input.

This bug was introduced in #53449 and fixed in 7.7+ in fe2f6b3. I've opened #71354 to fix it in 6.8.

[jimczi]

Fixed by #71354