[Testing] QA validate embedded etcd

[davidnuzik]

master currently has embedded etcd functionality (replacing dqlite). This needs QA testing before our v1.19.0 release, which is when we currently plan to pull in from master (thus have embedded etcd support).

It's important that we also test upgrades. Older k3s versions come with dqlite - we should start with a 3-node cluster running dqlite and upgrade to v1.19.0. This won't work -- but we need to understand and document why and how it does not work so this can be called out in v1.19.0 release notes.

[davidnuzik]

PR: #1770

[ShylajaDevadiga]

Validated using k3s version v1.19.1+k3s-a08e998b.(master build)
Create first node using

sudo curl -sfL https://get.k3s.io | INSTALL_K3S_COMMIT=a08e998bc5dd93ba3696cb6f08ffc363e6a753d8  INSTALL_K3S_EXEC="--cluster-init" sh -

Join subsequent nodes using

sudo curl -sfL https://get.k3s.io | INSTALL_K3S_COMMIT=a08e998bc5dd93ba3696cb6f08ffc363e6a753d8  sh -s - server --server https://MASTERIP:6443 --token TOKEN

Verified cluster-reset flag taking down two nodes in a 3 node cluster, leading to quorum loss
Steps:

1. Create 3 node cluster
2. Deploy some workloads
3. Shutdown two nodes
4. Update k3s unit file to include --cluster-reset flag
5. Reboot the node
6. Remove cluster-reset from the unit flag (See this msg in logs)
7. Restart k3s
8. Cluster becomes available.

To find the leader and etcd status

1. Install etcdctl using

apt install etcd-client
sudo ETCDCTL_API=3 etcdctl --cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --endpoints https://172.31.10.86:2379,https://172.31.5.93:2379,https://172.31.8.203:2379 endpoint status  -w table