KEP test cluster framework #2178
Conversation
k8s-ci-robot
added
sig/testing
|
@hoegaarden: GitHub didn't allow me to request PR reviews from the following users: mariantalla. Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
k8s-ci-robot
added
cncf-cla: yes
|
This is definitely on my todo list, but right now I'm booked trying to get 1.11 out the door. |
k8s-ci-robot
added
the
needs-rebase
k8s-ci-robot
removed
the
needs-rebase
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: Assign the PR to them by writing The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
|
||
| ### Goals | ||
|
|
||
| 1. To design an abstraction of “standing up a kubernetes cluster” which is: |
There was a problem hiding this comment.
How does this differ from what kubetest is trying to accomplish?
There was a problem hiding this comment.
We didn't know about kubetest before. The intent seems indeed very similar! From an (admittedly very quick) first try there might be room for some UX improvements, specifically for the use case of (new-ish) contributors that want to run it as part of their unit/integration test suite (and would be after fast feedback and low setup overhead).
We'll keep looking at kubetest and come back with more specific suggestions.
Is there additional documentation besides the README in the test-infra repo and the godocs?
cc @mariantalla
There was a problem hiding this comment.
Yeah kubetest is hard to use on its own right now, but I'd be interested in trying to fix that.
The other documentation is: https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-tests.md (hack/e2e.go is just a wrapper to call kubetest)
There was a problem hiding this comment.
We're trying to standardize the client tooling abstraction for re-use. I think there is some overlap of concerns, but there are other constraints we are trying to solve here.
| ## Motivation | ||
|
|
||
| With this proposal we would like to address these three points of friction: | ||
| - For Carol, we introduce an abstraction which is intended to unify existing |
There was a problem hiding this comment.
is there something special about a test cluster versus any other cluster?
There was a problem hiding this comment.
One example: A partial cluster (running only the apiserver & etcd) might be perfectly fine for a certain type of tests.
There was a problem hiding this comment.
@derekwaynecarr - Also (N) nodes on a single host is another example. Similar to other DIND clusters.
There was a problem hiding this comment.
Ability to specify cluster and node type for a suite of tests with specific requirements
is a huge advantage. If the abstraction is provider agnostic even better.
| methods of standing up clusters | ||
| - equally easy to use interactively and in CI | ||
| 2. To build a lightweight testing framework for CLI use which | ||
| - is not dependant on `k8s.io/kubernetes` |
There was a problem hiding this comment.
this implies that k8.io/kubernetes vendors this test framework?
There was a problem hiding this comment.
Yes. It should be possible for k/k to vendor this framework.
| - To wrap any existing “ways of standing up a cluster” in our abstraction | ||
| - Of course, wrapping existing technology may well be the most efficient way | ||
| to meet goals 2 and 3 (building CLI and API/Controller testing frameworks). | ||
| - To port large numbers of existing tests to use our abstraction. |
There was a problem hiding this comment.
absent this, are we just creating a new standard?
There was a problem hiding this comment.
To understand if this framework is useful we were thinking of porting a couple of tests for each (or most) of the test suites. An example of how such a port could look can be found here, where we experimented with porting different tests from test-cmd.sh. And I believe we should port not only tests from test-cmd.sh but also other tests.
But porting all the tests is not in the scope of this KEP.
|
How does this relate to the cluster api? AFAIK having a consistent way to spin up clusters is a goal already, more broadly than just test clusters. |
|
cc @krousey ^^ |
|
@BenTheElder Ultimately, users of the test framework would need to be able to describe the shape of clusters they need in their tests, i.e. number of worker nodes, number of master nodes. In this context, we're not clear yet what the relationship would be between Cluster API and the testing framework. Would using the cluster api become an implementation detail of a specific test cluster implementation - put differently, would some cluster types leverage the cluster API and others not? Or, would the testing framework implement the cluster API itself? cc @mariantalla |
|
I'd be concerned that test clusters are being provisioned differently from
actual end user clusters if we have a special test cluster framework, we
try to avoid this today by just having EG kubetest call upstream
provisioners (like cluster/* scripts).
My understanding is that the cluster API will provide portable and
configurable provisioning of this type and the KEP mentions the aim is to
cover all major existing providers / provisioners. I'm not sure why future
tests and test tools (and any other tools!) shouldn't just integrate with
this directly..?
…On Thu, Jun 21, 2018, 08:08 Hannes Hörl ***@***.***> wrote:
@BenTheElder <https://github.com/BenTheElder>
That's a good point. I think we need to have a close look on the work that
sig-cluster-api doing.
Ultimately, users of the test framework would need to be able to describe
the shape of clusters they need in their tests, i.e. number of worker
nodes, number of master nodes. In this context, we're not clear yet what
the relationship would be between Cluster API and the testing framework.
Would using the cluster api become an implementation detail of a specific
test cluster implementation - put differently, would some cluster types
leverage the cluster API and others not? Or, would the testing framework
implement the cluster API itself?
cc @mariantalla <https://github.com/mariantalla>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#2178 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AA4Bqy_drPUNxQiXZDr-IDTyWjzlH6rYks5t-7bggaJpZM4UKuEQ>
.
|
| ### Goals | ||
|
|
||
| 1. To design an abstraction of “standing up a kubernetes cluster” which is: | ||
| - expressive enough that most or all e2e tests ought to be able to use it, |
There was a problem hiding this comment.
our e2e tests do not stand up clusters directly today? they expect a cluster to be available in $KUBECONFIG, tooling above that does standup.
There was a problem hiding this comment.
Additionally, this is necessary for say, conformance test e2es, which must not be aware of the underlying provider at all, just talking to "a cluster" of some sorts, provisioned elsewhere.
| There is a clear tension between Alice and Carol’s frustrations. Left to their | ||
| own devices, Alice is likely to contribute to the proliferation of test | ||
| infrastructure that frustrates Carol, and Carol may find herself advocating | ||
| that Alice be prevented from acting to resolve her own frustration with the |
There was a problem hiding this comment.
Perhaps I've misread this persona section, but it reads to me as somewhat hostile to Alice, which I don't approve of.
I don't think anything we check in as a KEP should suggest preventing others from developing anything, let alone test infrastructure...? Why can't "Carol" let "Alice" build whatever tooling they want?
There was a problem hiding this comment.
This whole section beyond "We propose an abstraction so ..." (anything after the first paragraph) seems fairly unnecessary. I'd suggest a motivation more along the lines of "we'd like a standard framework for spinning up clusters for testing with varying requirements and backing implementations" or something along those lines.
|
I would second Ben's comments that I think it would be good for this to sync with cluster API as that approach (declarative k8s-style objects backed by controllers) seems to be a very powerful approach and unless there is some architectural reason blocking "clusters that tests run against" from being stood up using it, it seems like converging on one approach for "clusters that tests run against" and production clusters would be good. I concur that the concept of a "test cluster' is a little problematic as it necessarily divides that from "cluster". If the current cluster API controllers don't allow specific cluster configurations, why not create custom controllers on those objects to achieve what would be necessary for testing? |
k8s-ci-robot
added
the
needs-rebase
|
@hoegaarden: PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
This came up today at sig-testing, I think I might understand the goals a bit better now... See https://github.com/kubernetes/test-infra/tree/master/kind for some early tooling that may relate. |
|
@BenTheElder Will take a look at cc @hoegaarden |
|
/kind kep |
|
REMINDER: KEPs are moving to k/enhancements on November 30. Please attempt to merge this KEP before then to signal consensus. Any questions regarding this move should be directed to that thread and not asked on GitHub. |
|
KEPs have moved to k/enhancements. Any questions regarding this move should be directed to that thread and not asked on GitHub. |
|
@justaugustus: Closed this PR. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Proposal
See also: https://docs.google.com/document/d/13bMjmWpsdkgbY-JayrcU-e_QNwRJCP-rHjtqdeeoQHo/edit#
/sig testing
/cc @totherme
/cc @mariantalla
/cc @timothysc
/cc @liztio
/cc @marun