Determine process for keeping Conformance Tests current with API additions

[bobcatfish]

Expected Behavior

As folks make changes to the Elafros API, e.g. updating labels, we should make sure the conformance tests cover this behavior. At the moment I need to keep up with these changes and update the conformance tests myself. It may be better to have the folks who make the changes update the tests, however it's not 100% clear when contributors would need to do that, so if we want contributors to make the changes, we need to clearly define when they should do so.

Actual Behavior

I keep up with these changes and update the tests usually, though sometimes folks are pro-active about it (thanks @tcnghia !).

[bobcatfish]

After talking with @evankanderson we have decided that any changes to:

• The spec doc
• The API definition

Should result in a change in the conformance tests.

I will update the docs and also created #780 to automate the process decision.

[evankanderson]

An additional item which would help (I'm not quite sure how to do this) would be some sort of test coverage map of the conformance tests, e.g. 1. For each field, a. A count of how many times that field is set in the tests. b. A count of how many different values the field is set to. 2. As a possible addition, it might make sense to indicate in either the API or the conformance tests that certain fields are "critical" vs incidental to the test. I'm wondering if #1 could be done via an interposing webhook which simply records all the different states set.

…

On Sun, Apr 29, 2018 at 3:24 PM Christie Wilson ***@***.***> wrote: After talking with @evankanderson <https://github.com/evankanderson> we have decided that any changes to: - The spec doc <https://github.com/elafros/elafros/blob/master/docs/spec/spec.md> - The API definition <https://github.com/elafros/elafros/tree/master/pkg/apis/ela> Should result in a change in the conformance tests. I will update the docs and also created #780 <#780> to automate the process decision. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#592 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHlyN7_OzsIvX_O99NkjM9egznBH4ra_ks5ttj2XgaJpZM4TJI-r> .

-- Evan Anderson <[email protected]>