Etcd version should be easier to access when performing snapshot/restore

[serathius]

Background

Etcd allows for executing remote snapshot command that streams the snapshot data to the client. Client can upload the received data to any cloud storage of their preference (S3, GCS etc). Snapshot can be created by any 3.* compatible client, however restoring requires server to match the minor 3.X version of server that created the snapshot (for example 3.4 snapshot can be only restored into 3.4 server).

Restoring a snapshot is usually done in case of emergency mitigation when cluster is down. In such case it's preferred to restore cluster behavior before root causing. If downtime is caused by cluster upgrade or upgrade has happened since the snapshot, it is required to switch Etcd binary.

When working with larger fleets of clusters it is much harder to correlate information about snapshots and upgrades. Providing a fast response to restore critical clusters necessitates pairing etcd snapshots data with information server version.

Problem

Information about etcd version of particular node, nesesery for restoring snapshot, is not easily accessible when creating the snapshot. Version can be retrieved from etcd binary or external system (docker image, node metadata, etc). This undermines the ability to execute snapshots remotely as it either requires access to local node or some external system.

Proposal

I propose to implemented following improvements:

• Extend SnapshotResponse proto to include information about etcd version ([Version in Snapshot] SnapshotResponse includes local etcd version #13073)
• Store etcd file format version in the backend/bbolt itself, therefore making it available within the snapshot ([Version in Snapshot] Preserve etcd version in backend allowing etcdutl to read it from snapshot #13094)
• Have etcdutl snapshot status print information about etcd server that created it. ([Version in Snapshot] Preserve etcd version in backend allowing etcdutl to read it from snapshot #13094)

First change would allow to remove dependence on inspecting etcd binary thus making snapshoting fully remote. Remaining two changes would improve the process of restoring the snapshot, by allowing to verify that snapshot version matches the expected one.

cc @ptabor

[ptabor]

LGTM

Store information about local etcd version within the database itself, therefore making it available within the snapshot.

Nit: I would fraze it: 'Store etcd file format version in the backend/bbolt itself, therefore making it available within the snapshot' .
Technically it means the version of etcd that wrote the file. For bbolt file in running etcd (not backup snapshot written from scratch), the version should get upgraded when properties of new format has been upgraded.

I.e. etcd-3.5 added new field: meta/term
etcd-3.4 does not have it.

When we run etcd-3.5 for the first time on the '3.4' file, some logic should trigger that the meta/term field is 'populated' and only if its fulfiled the file should store to be in version-3.5.

[serathius]

ping @gyuho

[serathius]

WIth both PRs merged this issue can be closed.