Multi-index searches goes out of heap space if running with a big search size

[magnhaug]

To reproduce:

• Create multiple indexes
• Search on all indexes, with a search size of 2_000_000_000
• Run test with a normal, low -Xmx

I expect to be able to set a high search size to return all total values, even if all the indexes are small.

Happens in v. 1.1.2. I haven't tested other versions yet.
I can create a Java Unit test sometime later if you need me to.

[dadoonet]

It's not caused by multi index search but by the crazy number of document IDs each shard needs to allocate.

If you really need to extract a big amount of docs, please use scan and scroll API.

Closing. Feel free to reopen if you think it's an issue.

[clintongormley]

Duplicate of #4026

[magnhaug]

I get that this behavior is not something you'd typically want to do. But is there any reason for a shard of size x documents to allocate a docid container of size y, where y > x?

In the more general case: We do most of our (normal-sized) searches against aliases that point to a few big indexes and a couple dozen smaller indexes. Statically allocating memory for all potential result document IDs on every shard sounds like it's wasting precious memory, especially if not all indexes will produce any hits.

[clintongormley]

See #8080