upgrade from 21.1.11 to 21.2.0 got store error

[kocoten1992]

Describe the problem
I try upgrading cockroachdb from 21.1.11 to 21.2.0 (change binary cockroach), database won't start, revert back to 21.1.11 everything is good.

Nov 17 01:04:26 d5 cockroach[738728]: * ERROR: ERROR: cockroach server exited with error: store <no-attributes>=/opt/cockroachdb/cockroach-data, last used with cockroach version v20.2, is too old for running vers>
Nov 17 01:04:26 d5 cockroach[738728]: *
Nov 17 01:04:26 d5 cockroach[738728]: ERROR: cockroach server exited with error: store <no-attributes>=/opt/cockroachdb/cockroach-data, last used with cockroach version v20.2, is too old for running version v21.2>

But I don't use store attribute anymore (I have use it in the past - but nowadays I use ssd for everything so have drop that attribute from cockroachdb start command long ago).

# command to start cockroachdb
cockroach start --certs-dir=/opt/cockroachdb/certs --host=0.0.0.0 --join=localhost:26257 --locality=region=ap-southeast-1

Please describe the issue you observed, and any steps we can take to reproduce it:

This is an old cluster (I think maybe from 19.x version with repeated upgrade to newer version), last it was running on 21.1, upgrade to 21.2 and get issue above

To Reproduce
I've not try yet, but I think you could try store option on 20.2 then try upgrade to 21.2 and hopefully will get error.

Here is the reproduction
#72849 (comment)

[blathers-crl]

Hello, I am Blathers. I am here to help you get the issue triaged.

Hoot - a bug! Though bugs are the bane of my existence, rest assured the wretched thing will get the best of care here.

I was unable to automatically find someone to ping.

If we have not gotten back to your issue within a few business days, you can try the following:

• Join our community slack channel and ask on #cockroachdb.
• Try find someone from here if you know they worked closely on the area and CC them.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

[kocoten1992]

here is the reproduction

# preparing
cd /tmp/
rm -rf reproduce_72849
mkdir reproduce_72849
cd reproduce_72849
mkdir cockroach-data

# try 20.2
wget https://binaries.cockroachdb.com/cockroach-v20.2.18.linux-amd64.tgz
tar -xvf cockroach-v20.2.18.linux-amd64.tgz
cp cockroach-v20.2.18.linux-amd64/cockroach cockroach_20_2_18
./cockroach_20_2_18 start-single-node --insecure --store=cockroach-data

# then Ctrl+C to stop the server and try 21.2
wget https://binaries.cockroachdb.com/cockroach-v21.2.0.linux-amd64.tgz
tar -xvf cockroach-v21.2.0.linux-amd64.tgz
cp cockroach-v21.2.0.linux-amd64/cockroach cockroach_21_2_0
./cockroach_21_2_0 start-single-node --insecure

[kocoten1992]

Oke, I've figured it out, you need to upgrade to latest cockroach binary first, only then upgrade to 21.2, in this case it is 21.1.12, I thought minor version wasn't important, but it does!

P/s: you should let your production rest a few days if you gonna upgrade to 21.2 (ie: from 21.1.11 upgrade to 21.1.12, then rest, then continue to 21.2).

[knz]

Thanks for explaining how you solved the issue!

[igremmerlb]

Just chiming in that we are experiencing the same issue. Currently on 21.1.13. Can't upgrade to 21.2.4. The database was upgraded from 20.2 to 21.1 a few days ago. I'll try again next week and see if it works, as @kocoten1992 mentioned it might be better to wait a few days.

E220204 22:27:30.458947 1 1@cli/clierror/check.go:35 ⋮ [-] 2 ‹ERROR›: cockroach server exited with error: store ‹=/data›, last used with cockroach version v20.2, is too old for running version v21.2 (which requires data from v21.1 or later

[kocoten1992]

@igremmerlb waiting doesn't work (the comment above I made only work on testing 🥲 , on production with running database it does not).

Not reporting that because I'm surprised why no one is having the same problem, what I end up doing is backup and restore, that work for me.

Now there are more people having this issue, is it worth consider open again @knz ?

[knz]

The database was upgraded from 20.2 to 21.1 a few days ago

You have probably not finalized the upgrade to 21.1.

This is documented here:

• https://www.cockroachlabs.com/docs/stable/upgrade-cockroach-version.html#step-3-decide-how-the-upgrade-will-be-finalized
• https://www.cockroachlabs.com/docs/stable/upgrade-cockroach-version.html#step-5-finish-the-upgrade

[kocoten1992]

I'm not sure if that the case, it fail right at binary replace step.

After replace binary, node can no longer be start (result in the error above), only then re-replace with old binary node (21.1.x) node can be start again.

[igremmerlb]

I appreciate the help. This is what I am seeing when I try to run the command.
cockroach sql

Welcome to the CockroachDB SQL shell.

All statements must be terminated by a semicolon.

To exit, type: \q.

Server version: CockroachDB CCL v21.1.13 (x86_64-unknown-linux-gnu, built 2022/01/10 17:44:25, go1.15.14) (same version as client)

Cluster ID: e2e129fd-391d-4b8f-9c14-c78f6d8fce54

Enter ? for a brief introduction.

root@:26257/defaultdb> SET CLUSTER SETTING cluster.preserve_downgrade_option = '21.1';
ERROR: cannot set cluster.preserve_downgrade_option to 21.1 (cluster version is 20.2)

[igremmerlb]

It looks like two of my older nodes are stuck in decommissioning state even though they should be long gone. I bet that is the reason. I'll see if I can get those fully decommissioned.

[igremmerlb]

I stopped the full cluster and then started all the nodes back up. Nodes #1 and #3 are still in decommissioning state even though the cluster was restarted. These particular nodes have been long gone - maybe 6 months or a year. The UI shows everything is fine. Do you have any suggestions on how to force out the stuck decommissioning nodes?

[igremmerlb]

I was able to run:
cockroach node decommission 1 3
and that worked. So mine is good now.