upgrading a read-only index causes ClusterBlockException

[jeremybmerrill]

I had been running Elasticsearch 1.3.2. A few of my indexes were set to read-only -- that is, blocks.read_only was set to true.

I upgraded to 1.5.2. Various end-points, like /_aliases and /_cat/indices all returned a ClusterBlockException like ClusterBlockException[blocked by: [FORBIDDEN/5/index read-only (api)];].

I was able to fix the problem by unlocking the indices individually. However, there was -- to my admittedly-limited knowledge -- no way to list all my indices. It was only because I kept a separate list of indices that I was able to determine which indices to unlock. Conceivably, a user could get into a problematic state where they can't determine which index to unlock.

My guess is that the upgrade process attempted to upgrade the read-only indexes, which failed because they were set to read-only. Two possible solutions: one is that upgrades disregard the read-only flag; the other is that the _aliases etc. endpoints don't fail when some indexes are upgraded to 1.5.2 and others are still an older version.

I should note that this was a one-box, one-cluster install of Elasticsearch. Both 1.3.2 and 1.5.2 were installed via the .deb. My elasticsearch.yml is quite boring (just a few special HTTP settings).

This was my production box, so I don't have it available for testing anymore. Let me know if you have questions about replicability, etc.

[clintongormley]

@tlrx I think these issues have already been solved by #9203, no?

[tlrx]

@clintongormley @jeremybmerrill yes, this issue has been solved by #9203.

In 1.4 some cluster blocks were added to various endpoints (see #7740 and #7763) in order to have more control of what operation is allowed or rejected. Theses cluster blocks did not exist in 1.3.2 and that's why you saw a different behavior, the ClusterBlockException, in 1.5.2. Unfortunately some cluster blocks were not fine grain enough and block some endpoints like /_aliases, /_settings... #9203 add more precise cluster blocks level and fixed that.

I'm closing this issue since it will be resolved by the upcoming 1.6 release.