Implement etcd_bootstrap_script.sh in Golang

[timuthy]

What would you like to be added:
The current bootstrap script etcd_bootstrap_script.sh should be replaced by a Golang based implementation.

Why is this needed:
The etcd_bootstrap_script.sh has several dependencies to OS utilities like bash, curl, etc. This makes it very hard to change to Distroless container base image in the future. The etcd project moved to Distroless for 3.6.x (etcd-io/etcd#13556 reduce the attack surface and we should do the same. This will also eliminate some tedious maintenance tasks like updating base images and utilities because of reported vulnerabilities.

Tasks

Project Structure

• Create project structure
• Implement bootstrap script logic
• Add signal handler
• Add command and flags
• Add readiness endpoint
• Add TLS support
• Add Makefile
• Add Dockerfile
• Add vendor directory
• Add unit tests
  • app
  • bootstrap
  • brclient
  • util
  • types
    - [x] Use 3.5.6 version of etcd for etcd-wrapper.
• Use 3.4.26 of etcd in etcd-wrapper (See Upgrade to etcd v3.4.26 to fix vulnerabilities from go runtime #33). Move to the latest version of etcd in another increment (another PR). Backup-Restore continues to use the current client version.
• Add hack/local-dev scripts to help ease local KIND + skaffold based tests.
• CI/CD
  • Pipeline definition
  • Scripts
• Documentation
  • Ops guide
  • Operator docs
  • Developer docs
• Security requirements:
  • Change base image to distroless for both etcd-wrapper and etcd-backup-restore
  • Add an init container to change the ownership of /var/etcd/data from root to nonroot user (required for existing clusters)
  • Add pod security context to run all containers by default as nonroot.
• Add Dockerfile for ephemeral container which will be used for all diagnostics, this container will also be run as nonroot.
• Adapt etcd-druid and run druid integration and e2e
• Run g/g e2e tests
  • Run additional test where one of the pods is using etcd-wrapper and other two continue to use etcd-custom-image. This is possible during a stuck software update or a partially successful software update of the statefulset. Ensure that both can co-exists together.
  • Run additional test where one tries to roll-back from pods using etcd-wrapper to pods using etcd-custom-image.