Make Stored Compression Configurable

[kimchy]

Allow to make the stored compression configurable on the index level (index.store.compression.level), and use that when constructing the codec to use.

If possible (will be tricky), allow to set it on a live index, so for example, for time base indices, older indices can move to use higher compression mode and optimized (which should probably be done over a different issue once the setting is implemented).

[avleen]

This would be a huge plus to us right now - we're getting ready to set up a very large cluster (~30Tb+ data), and the current compression (which is designed to be a somewhat compressive, but mostly fast) is painful.

Being able to set a high compression rate, and accept that it will make things slightly slower, is a really big win.

[kimchy]

Indeed, note, this will require some changes in Lucene as well, which we are working on to make it happen. We obviously will keep this issue updated with progress.

[s1monw]

pushing out to 1.2

[avleen]

Hi folks, we're into 1.2 now. Is this still on the map? I would still gladly give up some Elasticsearch CPU time for better compression :-)

[kimchy]

@avleen this is tricky to implement, effectively, it means that we would somehow change Lucene configuration (codec) to have high LZ4 compression on the fly, which is not possible today. We are investigating how to properly expose it in Lucene safely, which proves to be a bit tricky. The hope is to address it soon, but no concrete timeframe. /cc @jpountz

[avleen]

Thanks Shay!

On Sat, Jul 5, 2014 at 1:15 PM, Shay Banon [email protected] wrote:

@avleen https://github.com/avleen this is tricky to implement,
effectively, it means that we would somehow change Lucene configuration
(codec) to have high LZ4 compression on the fly, which is not possible
today. We are investigating how to properly expose it in Lucene safely,
which proves to be a bit tricky. The hope is to address it soon, but no
concrete timeframe. /cc @jpountz https://github.com/jpountz

Reply to this email directly or view it on GitHub
#4160 (comment)
.

[jpountz]

FYI there is now an open issue on the Lucene side: https://issues.apache.org/jira/browse/LUCENE-5914

[kimchy]

closed in favor of #8863