Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

conformance: make it easier to isolate a single test failure #416

Open
rogpeppe opened this issue May 22, 2023 · 0 comments
Open

conformance: make it easier to isolate a single test failure #416

rogpeppe opened this issue May 22, 2023 · 0 comments

Comments

@rogpeppe
Copy link
Contributor

The conformance tests are not all independent of one another. That is, when there's a failure, isolating that test (for example by changing Specify to FSpecify) and re-running the test suite will not yield the same test failure, because the tests have interdependencies.

To take one example, I am currently seeing this failure:

/home/rogpeppe/other/oci-distribution-spec/conformance/00_conformance_suite_test.go:12
  Push
  /home/rogpeppe/other/oci-distribution-spec/conformance/02_push_test.go:15
    Blob Upload Chunked
    /home/rogpeppe/other/oci-distribution-spec/conformance/02_push_test.go:152
      Get on stale blob upload should return 204 with a range and location [It]
      /home/rogpeppe/other/oci-distribution-spec/conformance/02_push_test.go:215

      Expected
          <int>: 405
      to equal
          <int>: 204

      /home/rogpeppe/other/oci-distribution-spec/conformance/02_push_test.go:220

The location of the failure is here.

If I try to run that test on its own to see exactly what HTTP requests are being sent and received, so to try to understand why this failure is happening and how to fix it, I'd expect to be able to focus on the test by changing line 215 to use FSpecify.

However, that doesn't work because on line 217, the test refers to prevResponse, a variable outside of that test, that has been populated by this test.

This pattern is common throughout the conformance tests.

In general, it's a good idea for tests to be self-contained. Failing that, it would be nice if tests could at least document their dependencies so running a test on its own doesn't require error-prone tracing of dependencies.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant