Testing Best-Practices

[alexlafroscia]

With interest in this addon growing, it's becoming clear that there are some inconsistencies in the testing situation that would be good to resolve. This has arisen due to the fact that different people have implemented each of the components, and brought their own favorite practices with them. There's nothing wrong with that necessarily, but it's causing confusion and friction for new contributors who are unsure of what patterns they should be following.

Currently, things are inconsistent around

• Whether the components as-authored should include data-test- attributes
• Whether testing utilities should be part of the test module file or a separate file
• Probably other things?

It would be good to sort out the desired state around testing best-practices and define those guidelines in the contribution documentation, so that we can have a consistent approach to testing across all of the components in this addon.

[alexlafroscia]

Here are my own thoughts on this:

• I would like to avoid data-test- helpers within the authored components

  I see this as an implementation detail of our tests, rather than part of the components' public API. If we add them to the DOM, they immediately become public API, locking us into whatever names, conventions, etc. we start with. If we were, for example, to use accessibility-based selectors in the future -- something that feels reasonable for this project, given the goals -- then we need a semver "major" to remove the data-test- selectors. I'd rather we avoid them in the first place.

  This does mean that we need to be explicit about adding them to the test templates that need them for our own testing purposes. However, I personally don't find that extra line of code to be too much of a nuisance. Lines of testing code are free, and making the attributes explicit in the tests often aids in readability.

  An additional benefit is not needing to depend at all on ember-test-selectors, which we will need a dependency of the addon tree includes the attributes directly. Even if it's a popular package, it would be even better to not need it at all. If we add the attributes explicitly in our tests, where needed, we do not need the dependency at all (even as a dev dependency) since it only serves to strip the attributes out of production builds.

• I would like to reorganize our test helpers so that

  1. There is an addon-test-support tree at the root of the repo
  2. There is a file in that directory for each component, which exports the test helpers that we use
  3. All of these are considered to be public API

  This way, whatever test helpers we need for internal purposes can be re-used by consumers of this addon.

  Specifically what those helpers are is not something I have a deep concern over. I have a preference toward it just being helper functions or custom assertions, since those typically are easier for end-users to integrate with, and there isn't a dominant page object library in the Ember community. Choosing a page object implementation ourselves makes the test helpers less useful for other consumers. I am a little skeptical that our testing needs in this repo are so complex that they really need full page objects for each component, over simple helper functions, but I am open to an argument for supporting page objects specifically, especially if there's a way we can do so in a general enough way that they work with multiple page object libraries

[roomman]

Can you give a basic example of what your ideal test would look like?

More generally, it would be useful to know what is the end goal for this repository. To be the best implementation of the headlessui concept in Ember, or the one most aligned with the React / Vue implementations? I know these are not mutually exclusive but, for me, they do characterise the differences between how tests have been written. Some are pragmatic about using an Ember "style", while others try to mirror the React / Vue code.

Personally, and while I believe that the public API for this library should mirror headlessui, I feel like writing tests with a structure and style that's more familiar to the Ember community will help with adoption.

If we are to take that route, I wonder if we should consider adding ember-a11y-testing so we can audit for accessibility as a part of the testing workflow?