diff --git a/keps/NEXT_KEP_NUMBER b/keps/NEXT_KEP_NUMBER index 8351c19397f..60d3b2f4a4c 100644 --- a/keps/NEXT_KEP_NUMBER +++ b/keps/NEXT_KEP_NUMBER @@ -1 +1 @@ -14 +15 diff --git a/keps/sig-testing/0014-20180523-test-cluster-framework.md b/keps/sig-testing/0014-20180523-test-cluster-framework.md new file mode 100644 index 00000000000..e0dfe233fe1 --- /dev/null +++ b/keps/sig-testing/0014-20180523-test-cluster-framework.md @@ -0,0 +1,169 @@ +--- +kep-number: 14 +title: Test Cluster Framework +authors: + - "@timothysc" # Timothy St. Clair + - "@marun" # Maru Newby + - "@liztio" # Liz Frost + - "@frankgreco" # Frank B Greco Jr + - "@hoegaarden" # Hannes Hoerl + - "@totherme" # Gareth Smith + - "@mariantalla" # Maria Ntalla +owning-sig: sig-testing +participating-sigs: +reviewers: + - TBD +approvers: + - TBD +editor: TBD +creation-date: 2018-05-23 +last-updated: 2018-06-19 +status: provisional +see-also: +replaces: +superseded-by: +--- + +# Test Cluster Framework + +## Table of Contents + +* [Test Cluster Framework](#test-cluster-framework) + * [Table of Contents](#table-of-contents) + * [Summary](#summary) + * [Motivation](#motivation) + * [Goals](#goals) + * [Non-Goals](#non-goals) + * [Proposal](#proposal) + * [User Stories](#user-stories) + * [Implementation Details/Notes/Constraints](#implementation-detailsnotesconstraints) + * [Risks and Mitigations](#risks-and-mitigations) + * [Graduation Criteria](#graduation-criteria) + * [Implementation History](#implementation-history) + + + +## Summary + +> Making testing with clusters consistent + +We propose an abstraction so that users and tools (e.g. CI) have a consistent +way to spin up and configure clusters they can test against. This abstraction +is targeted mainly to the needs of the following personas and should work +equally well used interactively and non-interactively / automated. + +Alice, an experienced CLI Dev (see [Key Persona Alice][alice]), finds that she can’t +easily onboard new contributors like Bob (see [Key Persona Bob][bob]), in part +because her CLI and tests are too tightly coupled with heavyweight +infrastructure in `k8s.io/kubernetes`. To solve this problem, Alice sponsors +the development of a new lightweight testing infrastructure outside +`k8s.io/kubernetes`. + +Meanwhile, Carol, an API/Controller dev (see [Key Persona Carol][carol]), is +frustrated that there are already too many ways to spin up test clusters. + +There is a clear tension between Alice and Carol’s frustrations. Left to their +own devices, Alice is likely to contribute to the proliferation of test +infrastructure that frustrates Carol, and Carol may find herself advocating +that Alice be prevented from acting to resolve her own frustration with the +status quo. + +Most of the content of this KEP has already [been worked on and fleshed +out][the_doc] mostly by people from sig-testing, more specifically people +working on the [testing-commons subproject][testing_commons_notes]. + +[key_personas]: https://docs.google.com/document/d/13bMjmWpsdkgbY-JayrcU-e_QNwRJCP-rHjtqdeeoQHo/edit#heading=h.czu8o1uor9qk +[alice]: https://docs.google.com/document/d/13bMjmWpsdkgbY-JayrcU-e_QNwRJCP-rHjtqdeeoQHo/edit#heading=h.17sl3t5a8nu +[bob]: https://docs.google.com/document/d/13bMjmWpsdkgbY-JayrcU-e_QNwRJCP-rHjtqdeeoQHo/edit#heading=h.mecveaw12b4d +[carol]: https://docs.google.com/document/d/13bMjmWpsdkgbY-JayrcU-e_QNwRJCP-rHjtqdeeoQHo/edit#heading=h.4i5imqvwxv3a +[testing_commons_notes]: https://docs.google.com/document/d/1TOC8vnmlkWw6HRNHoe5xSv5-qv7LelX6XK3UVCHuwb0/edit# +[the_doc]: https://docs.google.com/document/d/13bMjmWpsdkgbY-JayrcU-e_QNwRJCP-rHjtqdeeoQHo/edit# + +## Motivation + +With this proposal we would like to address these three points of friction: +- For Carol, we introduce an abstraction which is intended to unify existing + and future methods of spinning up test clusters. +- For Alice’s frustration, we build a lightweight testing framework with no + dependency on `k8s.io/kubernetes` +- For the tension between Alice and Carol, we ensure that Alice’s framework is + fully accessible through Carol’s abstraction. + + + + +### Goals + +1. To design an abstraction of “standing up a kubernetes cluster” which is: + - expressive enough that most or all e2e tests ought to be able to use it, + rather than standing things up directly + - abstract enough that it could in principle be wrapped around many existing + methods of standing up clusters + - equally easy to use interactively and in CI +2. To build a lightweight testing framework for CLI use which + - is not dependant on `k8s.io/kubernetes` + - is easy for newcomers to use + - makes use of the abstraction above +3. To build a heavier weight testing framework, for API/Controller use which + - is fully functional + - makes use of the abstraction above + + +### Non-Goals + +- To wrap any existing “ways of standing up a cluster” in our abstraction + - Of course, wrapping existing technology may well be the most efficient way + to meet goals 2 and 3 (building CLI and API/Controller testing frameworks). +- To port large numbers of existing tests to use our abstraction. + +## Proposal + +### User Stories + +- [A lightweight framework](https://docs.google.com/document/d/13bMjmWpsdkgbY-JayrcU-e_QNwRJCP-rHjtqdeeoQHo/edit#heading=h.pwzs2vde6z0h) +- [A heavier weight framework](https://docs.google.com/document/d/13bMjmWpsdkgbY-JayrcU-e_QNwRJCP-rHjtqdeeoQHo/edit#heading=h.jk3ttegjtexg) +- [Bridging the two frameworks](https://docs.google.com/document/d/13bMjmWpsdkgbY-JayrcU-e_QNwRJCP-rHjtqdeeoQHo/edit#heading=h.g6pbp8y2hrj) +- [Newcomer-friendly lightweight bootstrapping](https://docs.google.com/document/d/13bMjmWpsdkgbY-JayrcU-e_QNwRJCP-rHjtqdeeoQHo/edit#heading=h.19lu9pyt334a) +- [Proper debugging support](https://docs.google.com/document/d/13bMjmWpsdkgbY-JayrcU-e_QNwRJCP-rHjtqdeeoQHo/edit#heading=h.mzdpiav3x5tz) + +There are more [user stories][additional_userstories] and +[personas][additional_personas] which are probably out of scope for this KEP +but paint a potential future direction we could imagine for this test cluster +framework. + +[additional_personas]: https://docs.google.com/document/d/13bMjmWpsdkgbY-JayrcU-e_QNwRJCP-rHjtqdeeoQHo/edit#heading=h.7k68eja3mgiu +[additional_userstories]: https://docs.google.com/document/d/13bMjmWpsdkgbY-JayrcU-e_QNwRJCP-rHjtqdeeoQHo/edit#heading=h.egbobak5q21m + +### Implementation Details/Notes/Constraints + +TBD + +### Risks and Mitigations + +- [Increasing the number of ways to stand up clusters for testing/investigation](https://docs.google.com/document/d/13bMjmWpsdkgbY-JayrcU-e_QNwRJCP-rHjtqdeeoQHo/edit#heading=h.89dtjdkjrit0) +- [Over-specialising our abstraction](https://docs.google.com/document/d/13bMjmWpsdkgbY-JayrcU-e_QNwRJCP-rHjtqdeeoQHo/edit#heading=h.5gypeylotxzj) + +## Graduation Criteria + +TBD + +## Implementation History + +