Bug: Atlantis loses planfiles when running on multiple projects.

[vedantrathore]

Hello,

So I was experimenting with atlantis for my organization. Currently, I'm using a local docker setup to test stuff out and then will move that to ECS/EKS.

I've setup my workflow using terragrunt as specified in the official documentation and I'm generating a new project for each directory that contains a separate terragrunt.hcl file (Basically each project is of the format account_env_module).

While testing out, I sent a PR that changes a module tag ref in all the projects, so atlantis ran plan using terragrunt for around 19 projects and also commented in the plan for each project:
This plan was saved to: /home/atlantis/.atlantis/repos/vedant.rathore/infrastructure-live/15/default/path/to/config/project_name-default.tfplan

According to the output, a separate tfplan should be saved in each project's terragrunt config path but as I exec into the docker container and traverse to the path, there is no such file. Even the locks for each project are automatically deleted once the plan for that project finishes (Since it's 19 projects, it's going sequentially).

Due to this, when I comment atlantis apply -p project_name I get an error that the tfplan does not exist.

Apply output

exit status 1: running "terragrunt apply -no-color $PLANFILE" in "/home/atlantis/.atlantis/repos/vedant.rathore/infrastructure-live/15/default/infra-internal/trm-poc/network": 
[terragrunt] [/home/atlantis/.atlantis/repos/vedant.rathore/infrastructure-live/15/default/infra-internal/trm-poc/network] 2020/08/01 17:50:47 Running command: terraform0.12.28 --version
[terragrunt] 2020/08/01 17:50:47 Terraform version: 0.12.28
[terragrunt] 2020/08/01 17:50:47 Reading Terragrunt config file at /home/atlantis/.atlantis/repos/vedant.rathore/infrastructure-live/15/default/infra-internal/trm-poc/network/terragrunt.hcl
[terragrunt] 2020/08/01 17:50:47 Terraform files in /home/atlantis/.atlantis/repos/vedant.rathore/infrastructure-live/15/default/infra-internal/trm-poc/network/.terragrunt-cache/8xOez0vKC12fUJcCeYQpKiq5wB0/KUM7PUknBwPUDqRegzGmvD8tOOU/infrastructure/network are up to date. Will not download again.
[terragrunt] 2020/08/01 17:50:47 Copying files from /home/atlantis/.atlantis/repos/vedant.rathore/infrastructure-live/15/default/infra-internal/trm-poc/network into /home/atlantis/.atlantis/repos/vedant.rathore/infrastructure-live/15/default/infra-internal/trm-poc/network/.terragrunt-cache/8xOez0vKC12fUJcCeYQpKiq5wB0/KUM7PUknBwPUDqRegzGmvD8tOOU/infrastructure/network
[terragrunt] 2020/08/01 17:50:47 Setting working directory to /home/atlantis/.atlantis/repos/vedant.rathore/infrastructure-live/15/default/infra-internal/trm-poc/network/.terragrunt-cache/8xOez0vKC12fUJcCeYQpKiq5wB0/KUM7PUknBwPUDqRegzGmvD8tOOU/infrastructure/network
[terragrunt] 2020/08/01 17:50:47 The file path /home/atlantis/.atlantis/repos/vedant.rathore/infrastructure-live/15/default/infra-internal/trm-poc/network/.terragrunt-cache/8xOez0vKC12fUJcCeYQpKiq5wB0/KUM7PUknBwPUDqRegzGmvD8tOOU/infrastructure/network/main_providers.tf already exists and if_exists for code generation set to "overwrite". Regenerating file.
[terragrunt] 2020/08/01 17:50:47 Generated file /home/atlantis/.atlantis/repos/vedant.rathore/infrastructure-live/15/default/infra-internal/trm-poc/network/.terragrunt-cache/8xOez0vKC12fUJcCeYQpKiq5wB0/KUM7PUknBwPUDqRegzGmvD8tOOU/infrastructure/network/main_providers.tf.
[terragrunt] 2020/08/01 17:50:47 The file path /home/atlantis/.atlantis/repos/vedant.rathore/infrastructure-live/15/default/infra-internal/trm-poc/network/.terragrunt-cache/8xOez0vKC12fUJcCeYQpKiq5wB0/KUM7PUknBwPUDqRegzGmvD8tOOU/infrastructure/network/_backend.tf already exists and if_exists for code generation set to "overwrite". Regenerating file.
[terragrunt] 2020/08/01 17:50:47 Generated file /home/atlantis/.atlantis/repos/vedant.rathore/infrastructure-live/15/default/infra-internal/trm-poc/network/.terragrunt-cache/8xOez0vKC12fUJcCeYQpKiq5wB0/KUM7PUknBwPUDqRegzGmvD8tOOU/infrastructure/network/_backend.tf.
[terragrunt] 2020/08/01 17:50:50 Running command: terraform0.12.28 apply -var-file=/home/atlantis/.atlantis/repos/vedant.rathore/infrastructure-live/15/default/infra-internal/trm-poc/common.tfvars -input=false -no-color /home/atlantis/.atlantis/repos/vedant.rathore/infrastructure-live/15/default/infra-internal/trm-poc/network/infra-internal_trm-poc_network-default.tfplan
stat /home/atlantis/.atlantis/repos/vedant.rathore/infrastructure-live/15/default/infra-internal/trm-poc/network/infra-internal_trm-poc_network-default.tfplan: no such file or directory
[terragrunt] 2020/08/01 17:50:51 Hit multiple errors:
exit status 1

Here is the atlantis.yaml jinja template I'm using. I'm generating this for each commit using a pre-commit hook.

---
version: 3

parallel_plan: false
parallel_apply: false
automerge: true

projects:
  # Project Anchor
  - &terragrunt
    name: template
    dir: '.null'
    workflow: terragrunt
    autoplan:
      enabled: true
      when_modified:
        - "**/*.tf*"
        - "**/terragrunt.hcl"

{% set projects = ('find * -mindepth 2 \( ! -regex ".*/\..*" \) -type f -name "terragrunt.hcl"' | shell(timeout=20) ).split('\n') -%}

{% for project in projects %}
  {%- set project_env = project.split('/')[1] %}
  {%- set project_name = project.split('/')[0:-1] | join('_') %}
  {%- set project_path = project.split('/')[0:-1] | join('/') %}

  - <<: *terragrunt
    {% if project_env == 'prod' %}
    apply_requirements: [approved, mergeable]
    {% else %}
    apply_requirements: [mergeable]
    {% endif %}
    name: {{ project_name }}
    dir: ./{{ project_path }}

{% endfor %}

Please let me know if I'm doing something wrong here and how would the workflow be handling multiple projects. Also what should be the ideal definition of a Project?

[lkysow]

Do you have -out=$PLANFILE as part of the plan command for your workflow?

[vedantrathore]

Yes I do. This is my ATLANTIS_REPO_CONFIG_JSON environment variable. Converted from json to yaml. This is the same specified in the official documentation.

---
repos:
- id: "/.*/"
  apply_requirements:
  - mergeable
  allowed_overrides:
  - apply_requirements
  - workflow
  workflow: terragrunt
workflows:
  terragrunt:
    plan:
      steps:
      - env:
          name: TERRAGRUNT_TFPATH
          command: echo "terraform${ATLANTIS_TERRAFORM_VERSION}"
      - run: terragrunt plan -no-color -out=$PLANFILE
    apply:
      steps:
      - env:
          name: TERRAGRUNT_TFPATH
          command: echo "terraform${ATLANTIS_TERRAFORM_VERSION}"
      - run: terragrunt apply -no-color $PLANFILE

[vedantrathore]

Hey @lkysow, Don't want to keep bugging but is there a way out of this? 😅

[lkysow]

This should be working. I think at this point it looks like more of an issue with your setup than with Atlantis because no one else is reporting this. Is it possible the local Docker setup is losing its data?

Can you try running just on your laptop and debug from there?

[jamengual]

@vedantrathore did you figure it out this?

[jasonrberk]

I think I ran into this same potential bug. I was working to handle #1508 The gist is this....

1. put a a PR that adds a .hcl file at nonprod/some/path/appA/terragrunt.hcl
2. atlantis plans and the PR is updated with the plan output
3. someone approves that plan
4. I, as a malicious / ignorant person, add this comment atlantis plan -d /nonprod/us-east-1/dev/catphotos -- -destroy
5. the catphotos path is on my branch....and there is a .hcl file, so atlantis honors the request, and makes a second plan, and now I can approve and get both merged, because the first approval...... 👎
6. thankfully, my preplan script ran, and dismissed the approvals....so I can't merge 👍
7. but I see this in the logs:

command_runner.go:332 ORG/REPO#62: Deleting plans because there were errors and automerge requires all plans succeed

the atlantis website only shows one lock, for nonprod/some/path/appA

but the PR ends up in this weird state where it sayd "1/2 projects planned successfully" and there's no way to fix your issue, besides "starting over"

not the end of the world, but very confusing......