KEP test cluster framework #2178
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| 15 |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,169 @@ | ||
| --- | ||
| kep-number: 14 | ||
| title: Test Cluster Framework | ||
| authors: | ||
| - "@timothysc" # Timothy St. Clair | ||
| - "@marun" # Maru Newby | ||
| - "@liztio" # Liz Frost | ||
| - "@frankgreco" # Frank B Greco Jr | ||
| - "@hoegaarden" # Hannes Hoerl | ||
| - "@totherme" # Gareth Smith | ||
| - "@mariantalla" # Maria Ntalla | ||
| owning-sig: sig-testing | ||
| participating-sigs: | ||
| reviewers: | ||
| - TBD | ||
| approvers: | ||
| - TBD | ||
| editor: TBD | ||
| creation-date: 2018-05-23 | ||
| last-updated: 2018-06-19 | ||
| status: provisional | ||
| see-also: | ||
| replaces: | ||
| superseded-by: | ||
| --- | ||
|
|
||
| # Test Cluster Framework | ||
|
|
||
| ## Table of Contents | ||
|
|
||
| * [Test Cluster Framework](#test-cluster-framework) | ||
| * [Table of Contents](#table-of-contents) | ||
| * [Summary](#summary) | ||
| * [Motivation](#motivation) | ||
| * [Goals](#goals) | ||
| * [Non-Goals](#non-goals) | ||
| * [Proposal](#proposal) | ||
| * [User Stories](#user-stories) | ||
| * [Implementation Details/Notes/Constraints](#implementation-detailsnotesconstraints) | ||
| * [Risks and Mitigations](#risks-and-mitigations) | ||
| * [Graduation Criteria](#graduation-criteria) | ||
| * [Implementation History](#implementation-history) | ||
|
|
||
| <!-- | ||
| [Tools for generating]: https://github.com/ekalinin/github-markdown-toc | ||
| --> | ||
|
|
||
| ## Summary | ||
|
|
||
| > Making testing with clusters consistent | ||
|
|
||
| We propose an abstraction so that users and tools (e.g. CI) have a consistent | ||
| way to spin up and configure clusters they can test against. This abstraction | ||
| is targeted mainly to the needs of the following personas and should work | ||
| equally well used interactively and non-interactively / automated. | ||
|
|
||
| Alice, an experienced CLI Dev (see [Key Persona Alice][alice]), finds that she can’t | ||
| easily onboard new contributors like Bob (see [Key Persona Bob][bob]), in part | ||
| because her CLI and tests are too tightly coupled with heavyweight | ||
| infrastructure in `k8s.io/kubernetes`. To solve this problem, Alice sponsors | ||
| the development of a new lightweight testing infrastructure outside | ||
| `k8s.io/kubernetes`. | ||
|
|
||
| Meanwhile, Carol, an API/Controller dev (see [Key Persona Carol][carol]), is | ||
| frustrated that there are already too many ways to spin up test clusters. | ||
|
|
||
| There is a clear tension between Alice and Carol’s frustrations. Left to their | ||
| own devices, Alice is likely to contribute to the proliferation of test | ||
| infrastructure that frustrates Carol, and Carol may find herself advocating | ||
| that Alice be prevented from acting to resolve her own frustration with the | ||
| status quo. | ||
|
|
||
| Most of the content of this KEP has already [been worked on and fleshed | ||
| out][the_doc] mostly by people from sig-testing, more specifically people | ||
| working on the [testing-commons subproject][testing_commons_notes]. | ||
|
|
||
| [key_personas]: https://docs.google.com/document/d/13bMjmWpsdkgbY-JayrcU-e_QNwRJCP-rHjtqdeeoQHo/edit#heading=h.czu8o1uor9qk | ||
| [alice]: https://docs.google.com/document/d/13bMjmWpsdkgbY-JayrcU-e_QNwRJCP-rHjtqdeeoQHo/edit#heading=h.17sl3t5a8nu | ||
| [bob]: https://docs.google.com/document/d/13bMjmWpsdkgbY-JayrcU-e_QNwRJCP-rHjtqdeeoQHo/edit#heading=h.mecveaw12b4d | ||
| [carol]: https://docs.google.com/document/d/13bMjmWpsdkgbY-JayrcU-e_QNwRJCP-rHjtqdeeoQHo/edit#heading=h.4i5imqvwxv3a | ||
| [testing_commons_notes]: https://docs.google.com/document/d/1TOC8vnmlkWw6HRNHoe5xSv5-qv7LelX6XK3UVCHuwb0/edit# | ||
| [the_doc]: https://docs.google.com/document/d/13bMjmWpsdkgbY-JayrcU-e_QNwRJCP-rHjtqdeeoQHo/edit# | ||
|
|
||
| ## Motivation | ||
|
|
||
| With this proposal we would like to address these three points of friction: | ||
| - For Carol, we introduce an abstraction which is intended to unify existing | ||
|
There was a problem hiding this comment. is there something special about a test cluster versus any other cluster? There was a problem hiding this comment. One example: A partial cluster (running only the apiserver & etcd) might be perfectly fine for a certain type of tests. There was a problem hiding this comment. @derekwaynecarr - Also (N) nodes on a single host is another example. Similar to other DIND clusters. There was a problem hiding this comment. Ability to specify cluster and node type for a suite of tests with specific requirements |
||
| and future methods of spinning up test clusters. | ||
| - For Alice’s frustration, we build a lightweight testing framework with no | ||
| dependency on `k8s.io/kubernetes` | ||
| - For the tension between Alice and Carol, we ensure that Alice’s framework is | ||
| fully accessible through Carol’s abstraction. | ||
|
|
||
|
|
||
| <!-- | ||
| TODO: Add note that we did interviews and talked to different people, noted in | ||
| the "Validation" section of the personas? | ||
|
|
||
| [experience reports]: https://github.com/golang/go/wiki/ExperienceReports | ||
| --> | ||
|
|
||
| ### Goals | ||
|
|
||
| 1. To design an abstraction of “standing up a kubernetes cluster” which is: | ||
|
There was a problem hiding this comment. How does this differ from what kubetest is trying to accomplish? There was a problem hiding this comment. We didn't know about We'll keep looking at Is there additional documentation besides the README in the cc @mariantalla There was a problem hiding this comment. Yeah The other documentation is: https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-tests.md ( There was a problem hiding this comment. We're trying to standardize the client tooling abstraction for re-use. I think there is some overlap of concerns, but there are other constraints we are trying to solve here. |
||
| - expressive enough that most or all e2e tests ought to be able to use it, | ||
|
There was a problem hiding this comment. our e2e tests do not stand up clusters directly today? they expect a cluster to be available in $KUBECONFIG, tooling above that does standup. There was a problem hiding this comment. Additionally, this is necessary for say, conformance test e2es, which must not be aware of the underlying provider at all, just talking to "a cluster" of some sorts, provisioned elsewhere. |
||
| rather than standing things up directly | ||
| - abstract enough that it could in principle be wrapped around many existing | ||
| methods of standing up clusters | ||
| - equally easy to use interactively and in CI | ||
| 2. To build a lightweight testing framework for CLI use which | ||
| - is not dependant on `k8s.io/kubernetes` | ||
|
There was a problem hiding this comment. this implies that k8.io/kubernetes vendors this test framework? There was a problem hiding this comment. Yes. It should be possible for |
||
| - is easy for newcomers to use | ||
| - makes use of the abstraction above | ||
| 3. To build a heavier weight testing framework, for API/Controller use which | ||
| - is fully functional | ||
| - makes use of the abstraction above | ||
|
|
||
|
|
||
| ### Non-Goals | ||
|
|
||
| - To wrap any existing “ways of standing up a cluster” in our abstraction | ||
| - Of course, wrapping existing technology may well be the most efficient way | ||
| to meet goals 2 and 3 (building CLI and API/Controller testing frameworks). | ||
| - To port large numbers of existing tests to use our abstraction. | ||
|
There was a problem hiding this comment. absent this, are we just creating a new standard? There was a problem hiding this comment. To understand if this framework is useful we were thinking of porting a couple of tests for each (or most) of the test suites. An example of how such a port could look can be found here, where we experimented with porting different tests from But porting all the tests is not in the scope of this KEP. |
||
|
|
||
| ## Proposal | ||
|
|
||
| ### User Stories | ||
|
|
||
| - [A lightweight framework](https://docs.google.com/document/d/13bMjmWpsdkgbY-JayrcU-e_QNwRJCP-rHjtqdeeoQHo/edit#heading=h.pwzs2vde6z0h) | ||
| - [A heavier weight framework](https://docs.google.com/document/d/13bMjmWpsdkgbY-JayrcU-e_QNwRJCP-rHjtqdeeoQHo/edit#heading=h.jk3ttegjtexg) | ||
| - [Bridging the two frameworks](https://docs.google.com/document/d/13bMjmWpsdkgbY-JayrcU-e_QNwRJCP-rHjtqdeeoQHo/edit#heading=h.g6pbp8y2hrj) | ||
| - [Newcomer-friendly lightweight bootstrapping](https://docs.google.com/document/d/13bMjmWpsdkgbY-JayrcU-e_QNwRJCP-rHjtqdeeoQHo/edit#heading=h.19lu9pyt334a) | ||
| - [Proper debugging support](https://docs.google.com/document/d/13bMjmWpsdkgbY-JayrcU-e_QNwRJCP-rHjtqdeeoQHo/edit#heading=h.mzdpiav3x5tz) | ||
|
|
||
| There are more [user stories][additional_userstories] and | ||
| [personas][additional_personas] which are probably out of scope for this KEP | ||
| but paint a potential future direction we could imagine for this test cluster | ||
| framework. | ||
|
|
||
| [additional_personas]: https://docs.google.com/document/d/13bMjmWpsdkgbY-JayrcU-e_QNwRJCP-rHjtqdeeoQHo/edit#heading=h.7k68eja3mgiu | ||
| [additional_userstories]: https://docs.google.com/document/d/13bMjmWpsdkgbY-JayrcU-e_QNwRJCP-rHjtqdeeoQHo/edit#heading=h.egbobak5q21m | ||
|
|
||
| ### Implementation Details/Notes/Constraints | ||
|
|
||
| TBD | ||
|
|
||
| ### Risks and Mitigations | ||
|
|
||
| - [Increasing the number of ways to stand up clusters for testing/investigation](https://docs.google.com/document/d/13bMjmWpsdkgbY-JayrcU-e_QNwRJCP-rHjtqdeeoQHo/edit#heading=h.89dtjdkjrit0) | ||
| - [Over-specialising our abstraction](https://docs.google.com/document/d/13bMjmWpsdkgbY-JayrcU-e_QNwRJCP-rHjtqdeeoQHo/edit#heading=h.5gypeylotxzj) | ||
|
|
||
| ## Graduation Criteria | ||
|
|
||
| TBD | ||
|
|
||
| ## Implementation History | ||
|
|
||
| <!-- | ||
| Major milestones in the life cycle of a KEP should be tracked in `Implementation History`. | ||
| Major milestones might include | ||
|
|
||
| - the `Summary` and `Motivation` sections being merged signaling SIG acceptance | ||
| - the `Proposal` section being merged signaling agreement on a proposed design | ||
| - the date implementation started | ||
| - the first Kubernetes release where an initial version of the KEP was available | ||
| - the version of Kubernetes where the KEP graduated to general availability | ||
| - when the KEP was retired or superseded | ||
| --> | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps I've misread this persona section, but it reads to me as somewhat hostile to Alice, which I don't approve of.
I don't think anything we check in as a KEP should suggest preventing others from developing anything, let alone test infrastructure...? Why can't "Carol" let "Alice" build whatever tooling they want?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This whole section beyond "We propose an abstraction so ..." (anything after the first paragraph) seems fairly unnecessary. I'd suggest a motivation more along the lines of "we'd like a standard framework for spinning up clusters for testing with varying requirements and backing implementations" or something along those lines.