Feature request: allow apply on merge #2172
Comments
chenrui333
added
feature
|
will setting |
No it won't work. For atlantis to respect GitHub branch protection rules we need to use |
|
Hey all,
There can be an alternate atlantis requirement for Thanks. |
|
Would one of the maintainers be able to access above plan form @satyamz ? We're considering contributing to the project by adding this functionality but would like to know if the plan looks sound and whether a PR would be accepted. |
|
I'm interested in this due to the GitLab API bug mentioned in #1174 that is preventing Atlantis' |
|
I'd like to take this issue one step further: automatically plan on any change to the PR - failing on other parallel PRs is okay - but maybe make that optional for obvious reasons: do NOT respond to and finally, apply on merge this relies on the plan being okay and the git provider approvals/results on the PR being acceptable and merge possible - so no smarts needed there, just dump plan output on every push and do apply and dump output on merge. This tool is almost perfect on the face of it but the above work flow is the only one that makes sense to me for any serious use. |
I believe plans are regenerated on PR changes already, there is no need to
Yes absolutely, it's a good point that in this scenario
Yes exactly, this is exactly the workflow I'd like to use as well!
It's good to see that there are more people who have the same requirements! |
|
We would love something along these lines as feature, as this is something that happens on a regular basis with engineers new to our team, or for engineers from outside our organisation. The ability to require the |
|
we would love to see this feature implemented for our team as well, many people forgot to comment |
|
Speaking as an interested observer, I'm curious if there's something about Atlantis' design that requires it to only run during the PR phase. I ask because I recently came across a notice that said a failure occurring post-merge "would need" to be fixed by a subsequent PR. Is this really mandatory? If I have a GitHub Action that fails post-merge, I have the option to just re-run it. Granted, if the fault is something within the repository that should never have been merged I'll most likely fix it in a new PR. But if the issue lies outside the repo, if it's something like an expired token or an intermittent network outage, I just need a button. Is there something fundamental to Atlantis that prevents this sort of capability? |
|
is this still an issue with |
As far as I can tell yes this is still a problem. |
|
Add Set https://www.runatlantis.io/docs/server-configuration.html#gh-allow-mergeable-bypass-apply flag |
|
@nitrocode regardless of the potential workaround I still think this feature would be very useful. Based on thumbs up there are at least 44 others who agree. We actually added this capability to forked atlantis, we should be in a position to submit a PR soon so that everyone in the community can benefit from it. Could this issue be reopened? |
|
this can be reopened but keep in mind that people tried to do this before in different ways and a lot of logic was added and broke other functionality since this part of the code is very tricky due to workarounds and limitations on VCSs. PRs are welcome but please give it a try to the gh-allow-mergeable-bypass-apply that @nitrocode mentioned. |
|
I don't understand what is wrong with the current solution or workaround of using https://www.runatlantis.io/docs/server-configuration.html#gh-allow-mergeable-bypass-apply. Please explain why this method is not what you prefer @jacekn |
|
@nitrocode Adding this flag still doesn't work, unless there's some other setting I'm missing. I've setup the atlantis apply check and adding the flag, but after getting an approval I get this message. Once I remove the atlantis/apply check, I'm able to apply normally. Setting the flag This is what we have for our apply_requirements Apply Failed: Pull request must be approved by at least one person other than the author before running apply. |
You have |
|
I did have that which is why I'm asking
|
Here are the reasons I can think of:
|
|
@nitrocode quick question about the |
|
@jacekn number 4 exactly - terraform cloud works quite nicely in this regard, though it'd be nice to have some alternative to it that isn't proprietary and pay per view. This stuff is trivial to achieve in something like Jenkins. You need some sort of call back event or webhook on new updates to the default branch ref spec in the remote - eg direct push or PR merge of any kind. The comment in the docs about lack of status checks in the branch protection rules seems bizarre at first - you can configure them to require any/all status checks and then when some kind of CI/CD registers one against it, suddenly that becomes required, for example. PR created = plan How hard can it be? I don't understand how that option helps at all, @nitrocode / @jamengual ? Seems to be orthogonal to the request here. |
|
we are not opposed to add a new functionality that can solve this problem. there is many ways mentioned that try in one way or another to solve this issue but it looks like at the end of the day something like - - apply-on-merge flag is necessary for those admins that do not like to apply their changes before merging. |
|
disclaimer: i did not yet install or use atlantis, rather waiting for this feature to be implemented before trying out. question: could apply-on-merge be somehow achieved with existing atlantis version, if we would trigger some github action on "push to master" which would then do an api call to atlantis? in this case it could be possible to say that merge condition is "atlantis plan passed without errors, and codeowner gave approval to merge" (that would be taken care of completely at github side) and then after merge, push to master would happen triggering "something" via atlantis http api. I can imagine that main problem could be to somehow infer which exactly prepared plan needs to be run at atlantis side, and then it would just boild down to streaming atlantis api output as github action output for real-time progress report and merge result outcome (pass|errored) |
|
yes, that is possible.
…On Thu, Feb 23, 2023, 12:39 a.m. Ervin Weber ***@***.***> wrote:
disclaimer: i did not yet install or use atlantis, rather waiting for this
feature to be implemented before trying out.
question: could apply-on-merge be somehow achieved with existing atlantis
version, if we would trigger some github action on "push to master" which
would then do an api call to atlantis? in this case it could be possible to
say that merge condition is "atlantis plan passed without errors, and
codeowner gave approval to merge" (that would be taken care of completely
at github side) and then after merge, push to master would happen
triggering "something" via atlantis http api. I can imagine that main
problem could be to somehow infer which exactly prepared plan needs to be
run at atlantis side, and then it would just boild down to streaming
atlantis api output as github action output for real-time progress report
and merge result outcome (pass|errored)
—
Reply to this email directly, view it on GitHub
<#2172 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAQ3ERHFBI574ZE2BFHESU3WY4O5ZANCNFSM5SCAIYHA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
still an issue with @jacekn I tried to answer your questions here: #2172 (comment) (i.e have a required atlantis/apply status check + atlantis having ATLANTIS_GH_ALLOW_MERGEABLE_BYPASS_APPLY set to true + apply_requirements: [approved, mergeable, undiverged]), and it didn't respect the fact that there was no approving PR review in the branch protection rule :/ cc @nitrocode |
|
@dimisjim I think you need to open a separate bug for your case and attach your branch protection configuration. Note that currently that flag in Atlantis only works with branch protection but not repository rulesets and also it only takes into account review state and required status checks and contains a couple of bugs, but there is an open PR to fix them. |
|
Indeed: #4193. Thanks for the tip |
|
In GitHub there is feature that can be enabled per repo - merge queue, which allows running commands just in the middle between PR schedulled for merge and before the actual merging. |
|
Having a big green merge button is not actually what you want - the PR is NOT mergable until the state has been applied, if you want the state of the repo to match reality. Ideally, merges would be locked while TF running, and the PR is merged immediately upon success. In theory it would be nice if Atlantis could optionally rerun the apply from
Github teams and Gihub branch protection rules cover this use case completely as far as I can tell. Have your PR only be mergable if approved by a specific team. Have a specific team be the only ones allowed to run apply (or plan, because plan is dangerous and allows arbitrary code execution). And that's it. Your PR will never be mergable if not approved by your designated team(s) and no one except your configured teams can run plan/apply. Note branch protection rules are configured in GHE, and the Atlantis server side config
Hashicorps recommendations are questionable IMO. I've created some pretty large Terraform setups and in no way would I ever recommend any team starting with Atlantis to start with one module per repo. At a certain point it makes sense, but trying to put one module per repo and versioning everything individually from the beginning is incredibly painful and completely unnecessary. Hashicorp clearly designed Terraform to run with manual intervention. Running it fully via automation is counter to the workflow Hashicorp requires. The main reason applying on merge is a bad idea is simple - what if your Terraform apply fails for any reason? If it ran as a server side per-merge hook, I could see that. If the apply fails, do not merge the commit. But any other situation will leave you in a bad position. A stale plan (which can happen for a number of reasons), a name is too long for some research, your AWS IAM policy is too large, duplicate name on s3 bucket, any number of reasons. There is a lot of complexity and if this feature was implemented it would become very clear for most people why it isn't supported. There are too many undesired behaviors that lead to bad outcomes. |
|
I can't stress enough how much I agree with post above! |
This was explained in this issues somewhere but tl;dr; is that you can still end up with out of sync situation even with the
It's possible that things have changed on GitHub side but historically there was no API to confirm all branch protection rules are met. Approvals are easy but branches might require other things than just approvals (for example you may enforce commit signing at branch protection rule level)
While I agree that some recommendations might be questionable they also are what they are and they come from the same people who created terraform. Not allowing your users to follow them seems like overstepping on the atlantis project side. Having said that I think there is a place for challenging hashicorp's recommendations in the terraform project.
To answer in short - similar thing will happen if you run |
I'd like to add my 2c too. I can totally understand that this feature might not be a priority, or, perhaps, is hard to implement due to some limitations of the architecture. However, I can't really accept the argument that this feature is a bad idea cause Terraform apply might fail after the code is merged, at least without some extra arguments. IMO, there's no major difference between the Terraform code as applied to infrastructure by Atlantis/Terraform and any application code as deployed by CI/CD pipelines. Companies that follow continuous deployment GitHub flow (single main branch, pull requests, changes merged to main branch deployed immediately) face the same issue: code changes merged into main branch could break when deployed to production due to migration failures, intermittent networking issues, etc; I would argue that those companies trade the chance of those issues (and the need of manual intervention in that case) for the simplicity and agility of the process when everything works fine.
From a formal point of view, repository state can never match real state 100% of the time. It is either occasionally ahead of the real state (if the PR is merged and then applied), or it is occasionally behind the real state (if the PR is applied and then merged). |
|
It's worth mentioning that Atlantis is "Terraform Pull Request Automation", not a "Terraform git automation", so it makes sense that atlantis is scoped to a Pull request.
Yes, and it's the whole point that it will be possible to see result directly in PR and either easily retry/fix whatever is broken in PR. To add to all things here, apply on merge will require some other communication/alerting mechanism. In PR you can at least have email/slack notifications for comments.
Yes, GitHub still doesn't have a single endpoint to check mergeability status unfortunately. However, there is a PR that tries to check various rules and decide if PR is mergeable or not. We haven't implemented signed commits check, but I guess it can added afterwards as any other check. It sucks that we need to re-implement github, but it is what it is.
Most of the beauty and power of Atlantis, at least how I see it, is to be able to do planning and applying via comments/commands in addition to what standard PR experience provides. If the desire is to have apply on merge (which we for example used to have with Jenkins), I personally don't see much reason to use Atlantis at all. It's pretty trivial to create github workflow for plan/apply, that will output plan as comment and rely on standard PR experience. It seems overkill to use Atlantis for this flow. You can actually come up with a better DX for yourself by using GH workflows.
When we are talking about cloud native, I think there is a huge difference between terraform and k8s/argo/flux/whatever. In k8s case it's safe to push a change that will break deployment, because old, working, deployment will still be there and while there will be a difference between git and actual state, it's not that important. In terraform case, it may make sense to consider doing more "risky" operation first - which would be applying terraform - and do merge after, because that one almost never fails.
This seems to be a strange thing to say about an open-source project. I think that tools should be picked according to your needs, not the other way around and there are other terraform/infrastructure automation tools out there that work with applying terraform after merge. |
|
In our R&D department, we use Terraform extensively to manage our resources, which works well for us. However, given the nature of R&D, there are times when we need to quickly delete and recreate resources, like when an EC2 instance is misconfigured by an engineer. Currently, the workaround is to create an empty pull request and run Any suggestions or improvements on this workflow would be greatly appreciated. |
I cannot disagree with this enough. Terraform applies are not atomic. Rarely does an apply fail without changing internal state and your environment. Therefore an apply failing after merge, leaves your trunk/main branch intact with the latest changes. That's much more desirable and reliable than a pull request failing to apply, without merging. That leaves your state in a transitional period where your main branch doesn't reflect the true nature of your environment. |
|
What if terraform apply fails after merge? You answered it yourself. 🍭
Both approaches have its own pros and cons :) |
You're misquoting me. I explicitly said, if your apply fails after merge, your trunk branch is still intact with the latest changes since applies are not atomic. Your quote is me talking about what happens if your apply fails before merge. |
|
When apply fails after the merge, your trunk branch is NOT intact with the state of your infrastructure. Both approaches (before and after merge) have the same drawback, because terraform does not have reconciliation/drift detection built in. So the question here is which situation is easier to recover from - failed to apply on PR or on the trunk. Since PRs do have a "user interface" in the form of comments, it's easier to understand an error and try to fix/retry quickly. But then again, it heavily depends on who you are trying to optimize the experience for. For regular developers, I would argue that a "nice" UI is more important than trying to dig through logs of terraform. For infra/platform people, sacred trunk and using API to trigger atlantis can be more natural and important. |
|
@stasostrovskyi not understanding how your trunk branch is not intact. If your apply fails after merge, then your code is already committed to trunk. Your trunk reflects what's in your environment because applies are not atomic. If your apply fails and creates X resources but fails Y, your trunk branch reflects those X resources. This is inherently more reliable than a pull request applying X resources and failing on Y, with your main branch not reflecting on any applied resources. |
|
It doesn't matter if the trunk branch reflects your environment 99% or 1%. Both of those cases need to be resolved ASAP and it's a matter of which way would be easier to perform recovery. There are also cases when you don't really know how to configure a specific resource because documentation is meh or you don't even know what it is you want to achieve and applying from PRs gives you a much faster feedback loop without polluting the trunk with random commits. Sure there is a case for applying after merge and we also do for some things that atlantis cannot manage (ex. atlantis deployment itself), but for that we are more than happy with a simple github workflow because in this scenario you are not using all the features of atlantis commands, locks and outputs anyway. |
|
github's merge queues solve all of these issues by backing out upon failure and it would be great to see support for this as an option. |
Interesting. How does that solve the problem? Could you elaborate? Just curious. |
if the apply is a required check, it reruns before merging, and if it fails, it backs out and re-opens the PR |
|
Well, that's not how merge queue would work with terraform. Whole idea of atlantis is that when you have a PR, you approve both the code change and the plan and then apply that specific plan. With merge queue you will get something like that:
|
its possible to get around this by failing and backing out the PR if the branch requires an update. the half applied resources are an issue, but i complex enough workflow could solve for that as well. |
and others
Community Note
Overview of the Issue
Currently when a PR is merged without
atlantis applywe end up in a situation where reality doesn't match desired state described with terraform. There is also no way to apply changes post-merge without raising a dummy PR.At the same time, in GitHub, it's sometimes not possible to prevent people form merging PRs. For example anyone with repo admin access will get implicit push access and thus will be able to merge approved PRs without having to apply first.
In practice this means that there is currently no way to prevent human errors (people merging approved PRs without running
atlantis applyfirst).I'd like to propose optional functionality to allow atlantis to be configured to apply changes on PR merges. This would allow organizations who cannot remove push or admin access from repos to ensure changes are applied on merges without relying on humans to remember to
atlantis applyI realize that there is a possibility that apply fails and we'll end up being out of sync but this is a trade off that some organizations might be willing to accept.
Reproduction Steps
Open a PR and then merge it without running
atlantis applyfirst. This will result in code being out of sync with reality and there is no way to retroactively apply changes.The only workaround I found is to open 2nd dummy PR and ensure it's applied before merging.
Logs
Environment details
I'm on atlantis v 0.18.2 but I believe older/newer versions will also be affected by this.
Additional Context
This feature was requested in 2017 but wasn't implemented back then: #36
There is also discussion about the mergable limitations here: #1316
Apply on merge would allow us to create workflow similar to that hashicorp describe in a docs. This makes me think that the functionality is non-controversial as hashicorp themselves describe it.
The text was updated successfully, but these errors were encountered: