Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tests requiring the conftest binary fail with newer versions of the utility #2600

Closed
tpickett66 opened this issue Oct 19, 2022 · 1 comment · Fixed by #2851 or #2966
Closed

Tests requiring the conftest binary fail with newer versions of the utility #2600

tpickett66 opened this issue Oct 19, 2022 · 1 comment · Fixed by #2851 or #2966
Labels
bug Something isn't working
Milestone

Comments

@tpickett66
Copy link
Contributor

When running the tests on a fresh system the following failure occurs:

--- FAIL: TestGitHubWorkflowWithPolicyCheck (0.03s)
    events_controller_e2e_test.go:1324: conftest >= 0.25.0 must be installed to run this test
FAIL
FAIL	github.com/runatlantis/atlantis/server/controllers/events	131.108s
FAIL

When the latest version of conftest (0.34.0) is installed using homebrew this failure persists despite the binary being on $PATH. Digging into the code I found that the tests are looking for a binary named conftest0.25.0, adding a symlink with this name allowed the tests to find the binary but they still fail with a different error.

» go test ./server/controllers/events
--- FAIL: TestGitHubWorkflowWithPolicyCheck (0.13s)
    events_controller_e2e_test.go:1335: could not parse contest version from Conftest: 0.34.0
        OPA: 0.43.0
FAIL
FAIL	github.com/runatlantis/atlantis/server/controllers/events	138.859s
FAIL

It seems like there are two problems here.

  1. We're looking for a binary with a version in the name rather than the generic binary name
  2. The version detection logic checking the program output is not handling the new output correctly.
@tpickett66 tpickett66 added the bug Something isn't working label Oct 19, 2022
@nitrocode
Copy link
Member

Hmm then perhaps we need to update this version or change the logic so we can avoid this check. I'm unsure how much value a version check is doing here.

version, _ := version.NewVersion("0.25.0")

But perhaps it's simply testing downloading custom versions of conftest? Would you be able to look at this closer @tpickett66 ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
2 participants