Support Git Submodules

[osterman]

what

• Support git submodules

why

• It appears that clones are not recursive:

  atlantis/server/events/working_dir.go

  Line 113 in f057d6d

  cloneCmd := exec.Command("git", "clone", cloneURL, cloneDir) // #nosec

• Support importing terraform root modules without needing terragrunt

use-case

• We're trying to adapt our deployment methodology to work well with Atlantis
• We keep all of our "root" modules in one repo called terraform-root-modules
• We then import these modules into various stages (aka environments or accounts)
• We have been using docker multi-stage, but that won't play well with atlantis detecting changes, so we're willing to compromise
• We want to use git submodules instead, with the expectation that atlantis would be able to detect changes for autoplans

---

@lkysow do you think this would work?

Technically, looks very easy to implement here and we'd be happy contribute the capability.

[lkysow]

Yeah that looks good! Please push a PR. Can you also look into if there are any unintended consequences?

[lkysow]

We should version detect git though and make sure we use the right command for the right version.

[tomer-1]

@osterman a more "backward compatible" approach to this would be:

1. "cd" into the repo
2. run "git submodules update --init --recursive"
   that'll do for all repos, even ones not containing any submodules.
   but this operation should be coupled with any "git clone" operation.
   i'm not aware of the way atlantis manages the repositories after the first clone.
   but if it's doing any git manipulations to prevent re-clones, then those manipulations should contain the steps presented above to achieve an up to date working copy.

[osterman]

The problem with this approach is we cannot detect changes automatically, which is why we implemented custom support for it in our fork.

[tomer-1]

The problem with this approach is we cannot detect changes automatically, which is why we implemented custom support for it in our fork.

could you please elaborate how did you manage implement this?
we have many environments and many repositories, and there are some things that implementing as a submodules is the best way to go

[osterman]

@tomer-1 we ended up having to compromise. Our change detects if a submodule folder is modified and then executes. We were not able to detect changes inside the submodule due to the current implementation relying on the files changed as returned by the github API and not the files changed using git calculus.

[osterman]

@aknysh can add more details

[tomer-1]

@osterman i understand you had to compromise for the changes in the modules,
what i don't fully understand is how did you manage to make atlantis sync the submodules after the clone?

[osterman]

@tomer-1 did you have a chance to look at the PRs where we implement it?

cloudposse-archives#11
cloudposse-archives#18
cloudposse-archives#19

@aknysh can answer any specific questions about these.

[krishnanandchoudhary]

@osterman HI, I am exploring atlantis to use, While the repo has submodules for terraform vars file, it does not checkout submodules. Wondering how to make it work. Thanks

[goobysnack]

I'm blocked by this too. Can we add extra args for the git clone so that it can --recurse-submodules TIA.

[lkysow]

I'm blocked by this too. Can we add extra args for the git clone so that it can --recurse-submodules TIA.

Does this work? @osterman was saying that that's not enough because the github api doesn't return the list of files changed in the submodule so Atlantis won't know that the files in the submodule have changed and it won't trigger a plan.

[goobysnack]

Are there extra args so that we can configure git in our server config? So we can add --recurse-submodules if we need to.

[lkysow]

Are there extra args so that we can configure git in our server config? So we can add --recurse-submodules if we need to.

No, you can't add extra args to the git clone command.

[goobysnack]

@lkysow I figured this one out too. I added this to my plan workflow:

- run: git submodule update --init --recursive && ...<removed>

[dcatalano-figure]

another submodule usecase to support is forcing a plan b/c of symlinks ... we have common terraform code IE. tfvars, in a directory that also happens to be a git submodule. We are able to check out the submodule with a custom workflow as described above. There are files in the submodule that are shared / accesses via a symlink in each respective directory. Of course the symlink doesn't change so there is nothing to trigger the plan.

I'd be really awesome to be able to force a plan via GitHub regardless if Atlantis thinks there is a change or not.

[ganeshk1928]

Is there any update on this one?

[dgokcin]

• run: git submodule update --init --recursive && ...<removed

@lkysow do you know how to make this work with a github app? any suggestions

[mnhat3896]

To workaround, here is my configuration, I did a test with Atlantis on local and it worked. hope it can help somehow.
--remote will always pull from the submodule repo, so you don't need to run submodule update on your root module and push the change

[dgokcin]

@mnhat3896 did you use a github app or your PAT to authenticate between atlantis and your github account on this test?

[mnhat3896]

@dgokcin Sorry for the response late.
No, I was using the ssh key, you just need to add your ssh key to the personal setting, don't use HTTPS in .gitmodules config file. when running the plan. Atlantis will expose your username/password as the plain-text.
==> Atlantis does not work well with HTTPS protocol because of the security risk above

[dgokcin]

@dgokcin Sorry for the response late.

No, I was using the ssh key, you just need to add your ssh key to the personal setting, don't use HTTPS in .gitmodules config file. when running the plan. Atlantis will expose your username/password as the plain-text.

==> Atlantis does not work well with HTTPS protocol because of the security risk above

hmm. can i do this with a github app? I want to avoid using a PAT or anything personal.

[ganeshk1928]

@dgokcin Sorry for the response late.
No, I was using the ssh key, you just need to add your ssh key to the personal setting, don't use HTTPS in .gitmodules config file. when running the plan. Atlantis will expose your username/password as the plain-text.
==> Atlantis does not work well with HTTPS protocol because of the security risk above

hmm. can i do this with a github app? I want to avoid using a PAT or anything personal.

@dgokcin Even before you run the plan/init or submodule commands, you have to authenticate with Github and you can do that via a Github app too. Please check this for reference on how to authenticate using Github app.

[mnhat3896]

well, yes indeed, as @ganesh-katakam-T18451 said you have to authenticate with VCS. The thing I mentioned above about submodule. when you run git submodule add ... will decide your protocol for Atlantis run in the plan. for example, if you run git submodule add https://github.com/<YOUR_REPO_URL>. then Atlantis will use HTTPS to run git submodule init and expose your authentication like clone of 'https://userA:[email protected]. therefore using SSH protocol if possible

[FRABUCHI]

@dgokcin
If you are using the github app and using --write-git-creds, .gitconfig and .gitcredential will be created in the /home/atlantis path.
The contents of gitconfig should be in the following path.

[url "https://x-access-token@[your git address]"]
    insteadOf = ssh://git@[your git ssh address]

If I write gitconfig like that code, the request will go to https.

[url "https://x-access-token@[your git address]/"]
    insteadOf = git@[your git ssh address]:

The contents of this file may vary depending on how you use git submodules.

[bml1g12]

In case it's useful, when using git app with --write-git-creds, I found the following allows me to use git submodules:

workflows:
  default:
    plan:
      steps:
        - run: |
            #!/bin/ash
            cat >/home/atlantis/.gitconfig <<EOF
            [credential]
                    helper = store
            [url "https://[email protected]"]
                    insteadOf = ssh://[email protected]
            [url "https://[email protected]/"]
                    insteadOf = [email protected]:
            EOF
        - run: echo "init submodule" && git submodule init
        - run: echo "updating submodule" && git submodule update --init --recursive

[bdellegrazie]

For github apps, see also #4704