Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Better determine number of expected comments and webhooks in e2e test #3907

Merged

Conversation

lukemassa
Copy link
Contributor

what

Determine how many expected comments and webhooks there are in the events controller e2e test by making use of a variable that tracks whether or not plans need to be discarded.

why

This was a very subtle bug that I noticed when working on #2992. Right now we use the ExpParseFailedCount flag to decrement not only the number of webhooks we expect to have fired, but also the number of comments that are expected. When the controller fails to parse the command, it does not fire a webhook (https://github.com/runatlantis/atlantis/blob/main/server/controllers/events/events_controller.go#L577), however it does add a comment. We then unconditionally add 1 to the expected number of comments to handle the comment for the cleanup of locks, but this does not happen in this case. Therefore two wrongs made a right, and the tests passed :)

In the process of removing --disable-apply for #2992, I had the need to tweak these tests in such a way that we had a parse error, but still had locks to cleanup. In this case there was no right setting to make both number of webhooks and number of comments happy, thus necessitating another parameter, ExpNoLocksToDelete that can vary independently from ExpParseFailedCount.

tests

All the changes are in unit tests, so unit tests passing should be sufficient.

The most straightforward test to demonstrate the old code has a bug is to add this simple test in main (which I ended up including in my PR)

atlantis % git diff
diff --git a/server/controllers/events/events_controller_e2e_test.go b/server/controllers/events/events_controller_e2e_test.go
index 93c63df3..1c6b2b6c 100644
--- a/server/controllers/events/events_controller_e2e_test.go
+++ b/server/controllers/events/events_controller_e2e_test.go
@@ -112,6 +112,13 @@ func TestGitHubWorkflow(t *testing.T) {
                // ExpParseFailedCount represents how many times test sends invalid commands
                ExpParseFailedCount int
        }{
+               {
+                       Description:   "no comment or change",
+                       RepoDir:       "simple",
+                       ModifiedFiles: []string{},
+                       Comments:      []string{},
+                       ExpReplies:    [][]string{},
+               },
                {
                        Description:   "simple",
                        RepoDir:       "simple",

This fails with this message:

            Mock invocation count for CreateComment(Any(models.Repo), Any(int), Any(string), Any(string)) does not match expectation.
            
            	Expected: 1; but got: 0

This is because we're unconditionally adding 1 with the expectation that every PR will need a comment to clean up, when it doesn't.

The only Comment that simple_with_allow_commands creates is the response about the incorrect command. Adding 1 to handle the cleanup then is not correct.

    --- FAIL: TestGitHubWorkflow/simple_with_allow_commands (0.11s)
...
            	CreateComment(models.Repo{FullName:"runatlantis/atlantis-tests", Owner:"runatlantis", Name:"atlantis-tests", CloneURL:"https://github-user:[email protected]/runatlantis/atlantis-tests.git", SanitizedCloneURL:"https://github-user:<redacted>@github.com/runatlantis/atlantis-tests.git", VCSHost:models.VCSHost{Hostname:"github.com", Type:0}}, 1, "```\nError: unknown command \"import\".\nRun 'atlantis --help' for usage.\nAvailable commands(--allow-commands): plan, apply\n```", "")
...

references

@lukemassa lukemassa requested a review from a team as a code owner November 1, 2023 04:34
@github-actions github-actions bot added the go Pull requests that update Go code label Nov 1, 2023
@jamengual
Copy link
Contributor

thanks @lukemassa for the contribution

@jamengual jamengual merged commit 8a87d2e into runatlantis:main Nov 1, 2023
12 checks passed
ijames-gc pushed a commit to gocardless/atlantis that referenced this pull request Feb 13, 2024
… test (runatlantis#3907)

* fix: Better determine number of expected comments and webhooks in e2e test

* Cleanup
ijames-gc pushed a commit to gocardless/atlantis that referenced this pull request Feb 13, 2024
… test (runatlantis#3907)

* fix: Better determine number of expected comments and webhooks in e2e test

* Cleanup
terakoya76 pushed a commit to terakoya76/atlantis that referenced this pull request Dec 31, 2024
… test (runatlantis#3907)

* fix: Better determine number of expected comments and webhooks in e2e test

* Cleanup
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
go Pull requests that update Go code
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants