Add ways to force-merge down to 1 segment

[jpountz]

Lucene just added a change to the TieredMergePolicy (default merge policy) which prevents it from force-merging indices down to 1 segment if this wouldn't meet the requirement that all segments must be less than the maximum segment size, 5GB by default. For instance if you have a 23GB shard and force-merge it down to 1 segment, you would en up with something like 5 or 6 segments.

There has been feedback that the ability to force-merge to 1 segment is important for time-series data, so we discussed several ideas:

• override Lucene's merge policy to revert to the previous behavior
• add a new separate API to freeze indices (wrong name because it currently means something else, but you get the idea) and make clear that it should only be called on read-only data
• keep Lucene's new behaviour, document the maximum size parameter (index.merge.policy.max_merged_segment), and tell users that if they actually want to merge down to 1 segment then they first need to update that setting. Maybe we could try to detect when an index becomes read-only and automatically update that setting.

In any case there is still the option to implement the first option in 6.x and only introduce the change in master so that the break is introduced in a major release.

[elasticmachine]

Pinging @elastic/es-distributed

[jpountz]

This was discussed in the Search team and in Fixit Friday.

• The problem is easier for index lifecycle management and curator, which we manage and could be fixed eg. to set the max merged segment size before calling _forcemerge.
• Detecting that the index is read-only based on the timestamp of the last write is not safe.
• @bleskes suggested to respect the max number of segments parameter if writes are blocked, and use Lucene's new default behavior otherwise (and document it). This would play well with ILM which needs to block writes anyway for the shrink operation. It isn't perfect as it is possible to enable writing again later, but might be good enough, especially as some upcoming changes in Lucene might make it better reclaim deletes on large segments in the future.

[pcsanwald]

@jpountz can you comment on if you think this is still a blocker for the 6.x series? it seems like either the blocker label is incorrect, or, the 6.6 label

[jpountz]

Thanks @pcsanwald, this issue is no longer relevant indeed, it was addressed via #32291.