Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New directory layout does not play nicely with upgrades/downgrades #52414

Closed
DaveCTurner opened this issue Feb 17, 2020 · 2 comments · Fixed by #78586 or #78638
Closed

New directory layout does not play nicely with upgrades/downgrades #52414

DaveCTurner opened this issue Feb 17, 2020 · 2 comments · Fixed by #78586 or #78638
Labels
blocker :Distributed Coordination/Cluster Coordination Cluster formation and cluster state publication, including cluster membership and fault detection. Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. v8.0.0-beta1

Comments

@DaveCTurner
Copy link
Contributor

DaveCTurner commented Feb 17, 2020

In #42489 we introduced a new layout for the data directory, but this does not today play nicely with some corner cases of upgrades and downgrades.

Firstly, as noted in #44624 (comment), we reorganise the data directory before checking that we can load and upgrade the cluster metadata; if it turns out that the cluster metadata is unreadable and needs fixing in 7.x then the user is stuck since a downgrade to 7.x will not find the cluster metadata since was already moved. I think we should not reorganise the data directory until we are sure that the upgraded node is viable.

Secondly, if a user does try and downgrade an 8.x node to 7.x then the node will successfully start up but will appear to be completely empty since the upgrade to 8.x already reorganised the data directory. I think we should protect against this by, for instance, leaving a copy of the node metadata in place in nodes/0/_state to indicate that this node has been upgraded.

@DaveCTurner DaveCTurner added the :Distributed Coordination/Cluster Coordination Cluster formation and cluster state publication, including cluster membership and fault detection. label Feb 17, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (:Distributed/Cluster Coordination)

@DaveCTurner DaveCTurner reopened this Feb 17, 2020
@DaveCTurner DaveCTurner changed the title New directory layout does not play nicely with downgrades New directory layout does not play nicely with upgrades/downgrades Feb 17, 2020
@rjernst rjernst added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label May 4, 2020
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Oct 2, 2021
Users sometimes attempt to downgrade a node in place, but downgrades are
totally untested and unsupported and generally don't work. We protect
against this by recording the node version in the metadata and refusing
to start if we encounter metadata written by a future version.

However in 8.0 (elastic#42489) we changed the directory layout so that a 7.x
node won't find the upgraded metadata, or indeed any other data, and
will proceed as if it's a fresh node. That's almost certainly not what
the user wants, so with this commit we create a file at
`${path.data}/nodes` at each startup, preventing an older node from
starting.

Closes elastic#52414
DaveCTurner added a commit that referenced this issue Oct 4, 2021
Users sometimes attempt to downgrade a node in place, but downgrades are
totally untested and unsupported and generally don't work. We protect
against this by recording the node version in the metadata and refusing
to start if we encounter metadata written by a future version.

However in 8.0 (#42489) we changed the directory layout so that a 7.x
node won't find the upgraded metadata, or indeed any other data, and
will proceed as if it's a fresh node. That's almost certainly not what
the user wants, so with this commit we create a file at
`${path.data}/nodes` at each startup, preventing an older node from
starting.

Closes #52414
@DaveCTurner
Copy link
Contributor Author

Merged the fix too soon, I am reverting the fix and reopening this.

@DaveCTurner DaveCTurner reopened this Oct 4, 2021
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Oct 4, 2021
Users sometimes attempt to downgrade a node in place, but downgrades are
totally untested and unsupported and generally don't work. We protect
against this by recording the node version in the metadata and refusing
to start if we encounter metadata written by a future version.

However in 8.0 (elastic#42489) we changed the directory layout so that a 7.x
node won't find the upgraded metadata, or indeed any other data, and
will proceed as if it's a fresh node. That's almost certainly not what
the user wants, so with this commit we create a file at
`${path.data}/nodes` at each startup, preventing an older node from
starting.

Closes elastic#52414
elasticsearchmachine pushed a commit that referenced this issue Oct 14, 2021
Users sometimes attempt to downgrade a node in place, but downgrades are
totally untested and unsupported and generally don't work. We protect
against this by recording the node version in the metadata and refusing
to start if we encounter metadata written by a future version.

However in 8.0 (#42489) we changed the directory layout so that a 7.x
node won't find the upgraded metadata, or indeed any other data, and
will proceed as if it's a fresh node. That's almost certainly not what
the user wants, so with this commit we create a file at
`${path.data}/nodes` at each startup, preventing an older node from
starting.

Closes #52414
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
blocker :Distributed Coordination/Cluster Coordination Cluster formation and cluster state publication, including cluster membership and fault detection. Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. v8.0.0-beta1
Projects
None yet
5 participants