Deprecated feature use in 7.x index fails upgrade to 8.x but cannot be downgraded. #84199
Comments
|
Pinging @elastic/es-core-infra (Team:Core/Infra) |
|
I am facing the same issue, and am looking forward to any updates! 😄 |
|
I was experiencing the same issue. I hope that this may help someone. |
This is likely to be harmful to your cluster and we very strongly recommend not to do this. See for instance these docs:
You can do whatever you want to your own data of course, but it is irresponsible to suggest this path to anyone else. |
|
Any solution for this? I don't remember having any critical warnings regarding this in the Update Assistant. For my case it would be ok even if this particular index would be deleted, doesn't really matter unless it will affect other indexes Edit: Somehow managed to fix it. In my case I didn't need those broken indexes, they were leftovers from the previous config, so it was probably a bit easier than trying to fix everything. What I did was building elastic from source changing the place where the exception is thrown to just |
This was really helpful. Thank you! For reference, I checked out the v8.2.0 branch, then changed line 198-201 of the IndexMetadataVerifier.java file as follows: from: } catch (Exception ex) {
// Wrap the inner exception so we have the index name in the exception message
throw new IllegalStateException("Failed to parse mappings for index [" + indexMetadata.getIndex() + "]", ex);
}to: } catch (Exception ignore) {
// Wrap the inner exception so we have the index name in the exception message
// throw new IllegalStateException("Failed to parse mappings for index [" + indexMetadata.getIndex() + "]", ex);
}setting the catch block to The code then compiled cleanly, and I was able to build a docker image to run in my cluster, then delete the bad indices using "curl": # Check on the unassigned shards:
$ curl -k -u $ELASTIC_USERNAME:$ELASTIC_PASSWORD https://localhost:9200/_cat/shards?h=index,shards,state,prirep,unassigned.reason | grep UNASSIGNED
elastalert_status_status UNASSIGNED p ALLOCATION_FAILED
elastalert_status_status UNASSIGNED r CLUSTER_RECOVERED
elastalert_status_error UNASSIGNED p ALLOCATION_FAILED
elastalert_status_error UNASSIGNED r CLUSTER_RECOVERED
elastalert_status UNASSIGNED p ALLOCATION_FAILED
elastalert_status UNASSIGNED r CLUSTER_RECOVERED
elastalert_status_past UNASSIGNED p ALLOCATION_FAILED
elastalert_status_past UNASSIGNED r CLUSTER_RECOVERED
elastalert_status_silence UNASSIGNED p ALLOCATION_FAILED
elastalert_status_silence UNASSIGNED r CLUSTER_RECOVERED
# If you are ready to delete all of the indices related to these shards, run:
curl -k -u $ELASTIC_USERNAME:$ELASTIC_PASSWORD https://localhost:9200/_cat/shards | \
grep UNASSIGNED | awk {'print $1'} | \
xargs -i curl -k -u $ELASTIC_USERNAME:$ELASTIC_PASSWORD -XDELETE "https://localhost:9200/{}" |
|
Just sorted this out with compiling from sources for my 4VM cluster because I had elastalert deprecated indices Here are the commands I executed to compile fix and go back to the official release `git clone https://github.com/elastic/elasticsearch.git apt-get install -y unzip zip ########compress your SNAPSHOT to distribute it across your cluster fleet scp the SNAPSHOT to all the servers. ##########make a local backup cp -R /tmp/elasticsearch-8.2.0-SNAPSHOT/ /usr/share/elasticsearch/ vim /etc/passwd chmod +x /etc/default/elasticsearch #########clean indices from UI ###########GO BACK TO THE ELASTIC RELEASE go back toroot againmv /usr/share/elasticsearch /usr/share/patch vim /etc/passwd |
|
hi @jgregmac Do you have the patched docker image with you that I can use? I am currently facing the same problem as described here. |
|
Thanks, I have solved the problem with the pointers above. Thanks! |
|
@yenonn , sorry for the delay in reply, but I am glad to hear that you were able to resolve your problem. I did not keep my docker image, but I will note that it is easy enough to build your own (for a given value of "easy"). @kaisbaccour provides a good starting point. In addition to cloning the elasticsearch repo and installing the JDK and Gradle prerequisites, you also will need Docker (docker-ce) and have configured permissions for the current user to run docker without sudo. I was hoping that "podman" with the "podman-docker" packages would be adequate, but they were not. Once all the prerequisites are in place, you cd to the elasticsearch repo root and simply run:
elasticsearch should compile and build a container image using the Dockerfile in the repo. The resulting container image can be accessed using |
|
Anyone having proxy/ssl issues should add their proxy to the |
|
I also faced that problem with elastalert indices when was upgrading Elastic stack from 7.17 to 8.x (running ECK k8s, with 4 replicas of elasticsearch) Firstly, there is a small how-to about it on elastalert2 docs Second, even I deleted indices before, one of my node was failing with My workaround was: I manually decreased Statefulset replicas count from 4 to 3, then ECK applied back 4 replicas and all of them green. |
|
Thanks to #88914 you can now go straight from 7.17.7 to 8.5.0 without having this issue (you'll still have to recreate the indices as stated in https://elastalert2.readthedocs.io/en/latest/recipes/faq.html?highlight=create-index#does-elastalert-2-support-elasticsearch-8 ) |
Elasticsearch Version
8.x
Installed Plugins
N/A
Java Version
bundled
OS Version
N/A
Problem Description
In #81326 we added some protection against starting up Elasticsearch 8.x where the cluster held indices created in 6.x, allowing the user to downgrade back to 7.x and address the problem.
A user reported a related but still-extant problem: they were using a deprecated feature in their 7.x indices and missed the deprecation log message so they couldn't start the 8.x node, but Elasticsearch didn't detect the problem until after rearranging the data path (#42489) so they could not downgrade to fix it either.
Steps to Reproduce
Create an index in 7.x using a deprecated feature. In this case the deprecation warning was as follows:
Try and upgrade to 8.x and observe that the node won't start.
Downgrade back to 7.x and observe that this node won't start either.
Logs (if relevant)
From the 8.x node when attempting to upgrade
From the 7.x node when attempting to downgrade
The text was updated successfully, but these errors were encountered: