[release-1.26] Multiple issues with temporary bootstrap etcd config

[brandond]

• Backport Multiple issues with temporary bootstrap etcd config #7368

[fmoral2]

Validated on Version:

- k3s version v1.26.4+k3s-e8d9ed79 (e8d9ed79)

Environment Details

Infrastructure
Cloud EC2 instance

Node(s) CPU architecture, OS, and Version:
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP4"

Cluster Configuration:
1 node

Config.yaml:

cat /etc/rancher/k3s/config,yaml
write-kubeconfig-mode: 644
token: test

Steps to validate the fix

1. Install k3s in commit version
2. Stop server
3. Start server with additional etcd args
4. Look in the logs for the etcd args
5. Check if they are set

Validation Results:

~$ k3s --version

k3s version v1.26.4+k3s-e8d9ed79 (e8d9ed79)



$ sudo systemctl stop k3s

$ sudo k3s server --etcd-arg='--quota-backend-bytes=5000000000'

Found in logs! Value set.




INFO[0006] Defragmenting etcd database                  
{"level":"info","ts":"2023-05-16T14:21:29.822Z","caller":"v3rpc/maintenance.go:90","msg":"starting defragment"}
{"level":"info","ts":"2023-05-16T14:21:29.827Z","caller":"backend/backend.go:497","msg":"defragmenting","path":"/var/lib/rancher/k3s/server/db/etcd-tmp/member/snap/db","current-db-size-bytes":4128768,"current-db-size":"4.1 MB","current-db-size-in-use-bytes":4128768,"current-db-size-in-use":"4.1 MB"}
{"level":"info","ts":"2023-05-16T14:21:29.900Z","caller":"backend/backend.go:549","msg":"finished defragmenting directory","path":"/var/lib/rancher/k3s/server/db/etcd-tmp/member/snap/db","current-db-size-bytes-diff":-86016,"current-db-size-bytes":4042752,"current-db-size":"4.0 MB","current-db-size-in-use-bytes-diff":-94208,"current-db-size-in-use-bytes":4034560,"current-db-size-in-use":"4.0 MB","took":"77.920135ms"}
{"level":"info","ts":"2023-05-16T14:21:29.900Z","caller":"v3rpc/maintenance.go:96","msg":"finished defragment"}




INFO[0009] Starting etcd for existing cluster member    

{"level":"info","ts":"2023-05-16T14:21:29.921Z","caller":"embed/etcd.go:306","msg":"starting an etcd server","etcd-version":"3.5.7","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.19.8","go-os":"linux","go-arch":"amd64","max-cpu-set":2,"max-cpu-available":2,"member-initialized":true,"name":"ip-172-31-10-109-fd3bdf3b","data-dir":"/var/lib/rancher/k3s/server/db/etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/rancher/k3s/server/db/etcd/member","force-new-cluster":false,"heartbeat-interval":"500ms","election-timeout":"5s","initial-election-tick-advance":true,"snapshot-count":10000,"max-wals":5,"max-snapshots":5,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://localhost:2380"],"listen-peer-urls":["https://127.0.0.1:2380","https://172.31.10.109:2380"],"advertise-client-urls":["https://172.31.10.109:2379"],"listen-client-urls":["https://127.0.0.1:2379","https://172.31.10.109:2379"],"listen-metrics-urls":["http://127.0.0.1:2381"],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-backend-bytes":5000000000,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":true,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}


sudo ETCDCTL_API=3 etcdctl --cert=/var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key=/var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt endpoint status  -w table
+----------------+------------------+---------+---------+-----------+-----------+------------+
|    ENDPOINT    |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+----------------+------------------+---------+---------+-----------+-----------+------------+
| 127.0.0.1:2379 | aecf4d7a27026eac |   3.5.7 |  4.7 MB |      true |         5 |       4319 |
+----------------+------------------+---------+---------+-----------+-----------+------------+