Investigate a solution for the local testing of the operator

[sdudoladov]

The current way of testing is to run minikube, either manually or with some tooling around it like /run-operator_locally.sh or Vagrant. This has at least three problems:

1. minikube is a single node cluster => unsuitable for testing vital functions such as pod migration between nodes.
2. minikube starts slowly; that prolongs local testing.
3. every contributor needs to come up with their own solution for local testing

We need to come up with a better option which will enable us to conveniently and uniformly run e2e tests locally / potentially in Travis CI.

A promising option is kind

[sdudoladov]

On the 1st look through the docs kind from k8s' own SIGTEST seems OK, or at least more stable/with better prospects than the alternatives

I will implement a single operator e2e test that uses operator-on-kind to create a PG DB to evaluate:

1. start up costs
2. ability to add/label nodes (needed to test pod migration, affinity etc.)
3. necessary tooling. Ideally e2e tests should not require long custom bash scripts as minikube-based testing currently does.

[sdudoladov]

@Jan-M @alexeyklyukin @erthalion @redbaron do you have more ideas ?

[Jan-M]

kind looks interesting to me if it can start 3 nodes locally and we can deploy/run our tests against this api similar to what we do internally.

[alexeyklyukin]

I used to test the docker for Mac internal Kubernetes instead of minikube (on a Mac), as minikube was not particularly stable there, but kind looks promising indeed to test a multi-node migration.

[sdudoladov]

FYI actual k8s e2e tests' infra overkill for our use case, but we might borrow ideas from it

[sdudoladov]

one other dimension to consider is to add some mock of the S3 API; that may enable testing backups/clones etc. locally

[FxKu]

So, #548 got merged by now. We should add a section in dev docs, though. Closing.