fix: implement backoff + retry when GitLab SetCommitStatus returns 409

[jippi]

what

GitLab returns a 409 Conflict status when the commit pipeline status is being changed/locked by another request, which is likely to happen if you use [--parallel-pool-size > 1] and [parallel-plan|apply].

The likelihood of this happening is increased when the number of parallel apply jobs is increased.

why

Returning the [err] without retrying will permanently leave the GitLab commit status in a "running" state, which would prevent Atlantis from merging the merge request on [apply]. GitLab does not allow merge requests to be merged when the pipeline status is "running."

tests

We've been running these changes in our environment for months and it has eliminated this type of error entirely

[jamengual]

@lukemassa since you are a Gitlab user.

[X-Guardian]

Can you add a test for this?

[jippi]

@X-Guardian I can certainly give it a try again; I came up short figuring out how to make it work in my original implementation. Do Atlantis provide docs on how testing should be done, and the libraries being used?

[X-Guardian]

The tests for this change need adding to the gitlab_client_test.go file. If you look at the patterns used in that, you should be able to add a new test for this scenario following the same patterns.

[lukemassa]

I agree about the tests, otherwise this looks good to me!

Needs Tests

[chenrui333]

lgtm, would be good to also have some tests.

[chenrui333]

lgtm, would be good to also have some tests.

I guess, I just echo what my fellow maintainers said :)

[jippi]

I have a short work week (just 2 days this week); so will probably not get it done this week - but will prioritize it next week , regardless of this is merged in with the future promise of tests or not :)

[chenrui333]

yeah, that works. thanks @jippi!

[jippi]

The promised tests are now up at #4503 :)