Question: Multiple Atlantis Servers

[devinslick]

We’d like to use multiple Altantis servers, with each running local to its provider’s resources.
The problem with this (design 1) is that both Atlantis servers would respond to the webhooks received by GitHub.

Design 1: Issue, both Altantis Servers react to webhooks from GitHub
├── Github Enterprise
│ └─── Repository
│ ├── Provider A subfolder
│ └── Provider B subfolder
├── Atlantis Server for Provider A
└── Atlantis Server for Provider B

Based on the design above, is there a way to configure Altantis to do one of the following?

• Filter webhooks/actions based on provider
• Whitelist subfolders (our folder structure organizes resources based on provider)
  The closest setting I was able to find was the atlantis.yaml autoplan (when_modified) parameter. https://www.runatlantis.io/docs/repo-level-atlantis-yaml.html#use-cases
  Would this or would any would other native Atlantis functionality meet these design criteria?
  I understand that intercepting / conditionally forwarding these webhooks via an API gateway or load balancer might work, but I would like to avoid adding complexity to the design.
  A second idea I had was using multiple Atlantis servers, using code-owners to permission the folders to different service accounts using code owners.

Design 2: 2 Atlantis Servers, subfolders with CodeOwners
├── Github Enterprise.
│ └─── Repository
│ ├─── Subfolder for Provider 1
│ │ └─── CODEOWNER=atlantis-provider-1
│ └─── Subfolder for Provider 2
│ └─── CODEOWNER=atlantis-provider-2
├── Atlantis Server
│ └─── gh-user=atlantis-provider-1
└── Atlantis Server 2
└─── gh-user=atlantis-provider-2

Lastly, if design 2 isn’t feasible then it seems like multiple repositories might be the simplest way forward:

Design 3 - Multiple Repos
├── Github Enterprise.
│ ├─── Repository for Provider 1
│ └─── Repository for Provider 2
├── Atlantis Server 1
└── Atlantis Server 2

Can you provide any recommendations on a solution?

Thanks!
Devin Slick

[chadasapp]

A custom workflow would be able to call a script you created on the Atlantis server capable of making that distinction. It'd be pretty simple too, basically "if environment execute plan/apply, else exit 0'

[devinslick]

I've been reviewing the workflow/scripting possibilities using atlantis.yaml and repos.yaml. It looks like I would be able to do something like this:

atlantis.yaml (uploaded to repo)

version: 3
projects:

• name: cloudprovider
  dir: cloudprovider
  workflow: cloudprovider
• name: localprovider
  dir: localprovider
  workflow: localprovider

repo.yaml (passed to localprovider's atlantis server with --repo-config)

workflows:
localprovider:
cloudprovider:
plan:
steps:
- run: echo "Cloud plan disabled from localprovider's atlantis server"
apply:
steps:
- run: echo "Cloud apply disabled from localprovider's atlantis server"

Do you see any issues with this solution?
Thanks for the help!

[lkysow]

There's no way built-in to Atlantis so the direction you're working in is on the right track.

I haven't tried this myself so I don't know the downsides. It should be pretty simple to test though.

If you work through it, please report back here for the benefit of other users.

[osterman]

Fwiw, #326 and #310 were to address a similar use-case, however, we have abandoned both. Now we just rely on cloudposse-archives#23.

We currently run multiple atlantis servers, but practice a poly-repo strategy, with a centralized module catalog. We run one atlantis server per AWS account, and each AWS account gets it's own repo / Dockerfile. This strategy has allowed us to run different versions of atlantis per account, and promote stable releases of atlantis as necessary. It also means the atlantis only wakes up when the webhooks for that account are triggered and forces an security paradigm whereby atlantis can only modify one account at a time. We can do all the github repo security controls to control who can commit to which repos (aka accounts).

[osterman]

Also, #249 is related

[devinslick]

Thank you both! It seems clear that we'll need to use a single Atlantis server and GitHub repository.
For anyone following this issue in the future, I was able to use custom workflows to run a 'echo' command, but this still shows warnings in Github that include this output. For a solution with multiple atlantis servers to work, we'd only want responses from one atlantis server.

Some of the resulting warnings that I got were due a related issue. I'll try to describe it below, but please let me know if you'd prefer that I close this issue and open a new one.

I'm now trying to use Atlantis on a nested folder structure with custom workflows to use Terragrunt.

├── Github Enterprise.
│ └─── Repository
│ │ └─── atlantis.yaml: defining projects (dir and workflows)
│ ├─── Subfolder for Provider 1 / Workflow 1
│ │ └─── terraform.tfvars (defining remote state) This is the working directory
│ │ └─── subfolder
│ │ └─── subfolder
│ │ └─── subfolder
│ │ └─── resource
│ │ └─── terraform.tfvars (defining new resource) This should be the working directory
│ └─── Subfolder for Provider 2 / Workflow 2
│ └─── terraform.tfvars (defining remote state)
└── Atlantis Server
└─── gh-user=atlantis-provider-1

Is it expected behavior that use of workflows changes the working directory to Provider 1's subfolder?
It only does this if the terraform.tfvars file exists there. If terraform.tfvars doesn't exist under Provider1 then it runs terragrunt with the correct working directory.

The only workaround we've come up with is to avoid custom workflows and use a wrapper shell script named 'atlantis' to intercept the default terraform command and pass appropriate parameters to terragrunt.

Again, thanks for the help and please let me know if you'd prefer that I open a new issue for this question.

[lkysow]

Hey Devin, your directory diagrams are kinda hard to follow. It looks like your Subfolder for provider 1 is outside your repository. Is that correct?

Do you have those folders on the Atlantis server itself instead of in a Terraform repo?

[devinslick]

I'll see if I can clean up the formatting when I get to my desk.
This might make more sense:
Repo > subfolderA > subfolderB > terraform.tfvars.

If a custom workflow for subfolderA is used to call terragrunt against subfolderB's new resource and a terraform.tfvars files exists under subfolderA then the working directory of the command will be subfolderA.

Since the working directory is incorrect terragrunt will find subfolderA's terraform.tfvars (which defines only remote state) but not the new resource.

[lkysow]

So like:

subfolderA/
  terraform.tfvars
  subfolderB/
    terraform.tfvars

Is it expected behavior that use of workflows changes the working directory to Provider 1's subfolder?
It only does this if the terraform.tfvars file exists there. If terraform.tfvars doesn't exist under Provider1 then it runs terragrunt with the correct working directory.

Atlantis shouldn't be doing this. It will execute the command from within the directory configured by projects:

version: 3
projects:
- dir: subfolderA
  workflow: pwd
- dir: subfolderA/subfolderB
  workflow: pwd
workflows:
  pwd:
    plan:
      steps:
      - run: pwd

What does the above workflow do? I would expect if subfolderA/terraform.tfvars changes, then it prints subfolderA and if subfolderA/subfolderB changes, it prints subfolderA/subfolderB. Are you saying that that is not happening?

[devinslick]

I think the issue is that I'm expecting to retain the pwd of changed tfvars file, regardless of the workflow in use. I'm trying to avoid defining each subfolder.

repository/
└── [cloud]-[account]/
    └── terraform.tfvars
    └── [department]
        └── [resourceType]
           └── [resourceName]
              └── terraform.tfvars

With this directory structure, it's not feasible to define a project for each subfolder.

Right now, for just this cloud account, I'm using something like:

version: 3
projects:
- dir: AWS-Shared
  workflow: aws-shared

Since each new resource is in its own subfolder, we'd have to add projects for each new VM/resource to make this work. Here's an example of what it sounds like workflows would require:

version: 3
projects:
- dir: AWS-Shared\IT\VM\Test-CentOS
  workflow: aws-shared

Is this right?

Might it be possible to do something like this?

version: 3
projects:
- dir: AWS-Shared\**
  workflow: aws-shared

It would, of course, need to be able to call Terragrunt with a working directory of \AWS-Shared\IT\VM.

The odd part I was trying to describe in my previous post is that if terraform.tfvars doesn't exist under the workflow's defined directory, Atlantis does use the working directory of the changed file.

[lkysow]

Okay I see the problem now. There's no support for wildcards in dirs. I think #500 is what you need. Otherwise you'll have to write a very smart workflow that handles things or you'll need to creat an atlantis.yaml that defines all the directories.

The odd part I was trying to describe in my previous post is that if terraform.tfvars doesn't exist under the workflow's defined directory, Atlantis does use the working directory of the changed file.

I still don't understand this. If this is something you think of a bug can you provide me a fully spec'd way to reproduce with an example repo and atlantis.yaml file? As you can see here we're setting the Dir of exec.Command to the directory defined by the dir key so I don't understand how Atlantis is behaving differently based on the existence of a terraform.tfvars file or not. Perhaps this is something in terragrunts behaviour?

[devinslick]

#500 would help, but seems like overkill for this issue. It's also not yet available :).
SubdirectoryB, C, etc, have no differences in workflow. The only reason we're using workflows is to set cloud account information (IAM) and handle authentication to local resources. Nested folders shouldn't all need to be defined in atlantis.yaml.

I'm curious, why does the use of workflows change the working directory?

How to reproduce:
1 - Define terragrunt remote_state backend in a terraform.tfvars file
2 - Create a subdirectory with a new resource terraform.tfvars
3 - The new resource needs to depend on the terraform.tfvars file created in step 1: path = "${find_in_parent_folders()}"

The inconsistency in working directory based on the existence of terraform.tfvars under the workflow dir is more difficult to reproduce. I did it by deleting the file from the cache directory in the Atlantis container. That behavior confused me but isn't really the issue I'm reporting.

Since I'd prefer not to rely on automatically generated atlantis.yaml projects, I think I'm going to have to resort to dropping workflow usage and using a terraform shell script instead.

[lkysow]

Yes I agree that your path forward is your shell script.

Can you provide a repro of "workflows changing the directory" issue without using terragrunt?

[lkysow]

Closing because I haven't heard back.