feat: atlantis import #2783

krrrr38 · 2022-12-13T00:27:12Z

what

introduce import workflow step
- import step process terraform import and discard plan file, cause existing plan file cannot be used anymore.
import workflow run init and import steps
command runner can handle atlantis import ADDRESS ID
- currently import run on a single project only. this is because we don't run import on multiple projects in the general use-case.
add import_requirements configuration not to edit tfstate without any reviews
introduce atlantis import e2e tests

why

Currently, we need to do manual terraform operations to import unmanaged resources. If support atlantis import, we can handle these operations on vcs operation.

references

debug PR: atlantis import krrrr38/atlantis-debug#3
closes Support atlantis import subcommand #217

discussion

PR is too large
Support remaining atlantis terraform commands #2776
- It's possible to close this PR, if the above feature will be introduced.
pflag command handling
- ~~atlantis import -- ADDR ID~~
  - PROS
    - easy implementation
    - After -- comments can handled as terraform args.
  - CONS
    - pflag cannot handle it directory
- ~~atlantis import -a ADDR -i ID~~
  - PROS:
    - intuitive
    - pflag can show help
  - CONS:
    - CommentCommand need to handle command-specific parameters.
- atlantis import ADDR ID <- choosed
  - PROS: more intuitive
  - CONS:
    - ~~we need to manage extraArgs and newArgs in flags.~~
    - ~~pflag cannot show ADDR ID usage in help like this?~~
    - append args into extraArgs

runatlantis.io/docs/using-atlantis.md

server/core/config/valid/global_cfg.go

server/core/runtime/import_step_runner.go

server/events/comment_parser.go

server/events/import_command_runner.go

server/events/project_command_runner.go

nitrocode · 2022-12-13T06:00:39Z

Wow. This looks very promising @krrrr38 !

I don't see how this can close the terraform pass through command #2776 . As amazing of a contribution as this is, it implements the import subcommand but not the state or validate or other terraform subcommands that #2776 refers to. Please don't see that as criticism. Keep going!

This is a very large PR so it will take time to review. Thank you for your efforts.

nitrocode · 2022-12-19T20:52:36Z

Please give an initial review when time permits @runatlantis/maintainers

cc: @albertollamaso @Fabianoshz @Fauzyy @SSKLCP (for visibility from our more recent golang expert contributors)

Fabianoshz

Aside from the metrics, only minor/nitpicks from my part.

Great PR, really nice work!

server/server.go

server/events/comment_parser.go

runatlantis.io/docs/using-atlantis.md

server/server.go

albertollamaso · 2022-12-20T20:33:18Z

LGTM

jamengual · 2022-12-20T21:27:17Z

@krrrr38 Amazing work, thanks so much for this PR.
As long as you address the comments it looks good to me

nitrocode · 2022-12-20T23:04:45Z

Looks good to me too. Please fix the conflicts and we can get this merged.

runatlantis.io/docs/using-atlantis.md

runatlantis.io/docs/command-requirements.md

nitrocode · 2022-12-21T06:46:24Z

runatlantis.io/docs/command-requirements.md

@@ -1,9 +1,9 @@
-# Apply Requirements
+# Command Requirements


Hm the renaming of this file will remove any existing links that point to this page.

There's a way to alias pages. If you keep this file rename in place, could you add a link for the original page apply-requirements.md to redirect to the new page?

I added an empty page. I will try redirect plugin later.

https://vuepress-community.netlify.app/en/plugins/redirect/

cc: @chenrui333 @lilincmu in case this is what you folks mentioned a couple months ago

runatlantis.io/docs/using-atlantis.md

nitrocode · 2022-12-21T06:51:40Z

runatlantis.io/docs/using-atlantis.md

+```
+### Explanation
+Runs `terraform import` that matches the directory/project/workspace.
+This command discards terraform plan result. Before apply, required `atlantis plan` again.


Suggested change

This command discards terraform plan result. Before apply, required `atlantis plan` again.

This command discards the terraform plan result. After an import and before an apply, another `atlantis plan` must be run again.

Because of the discarding of the plan after an import, should there be another flag to auto plan a project/dir/workspace after a successful import?

Also if an import command fails can we retain the plan instead of discarding it?

Because of the discarding of the plan after an import, should there be another flag to auto plan a project/dir/workspace after a successful import?

Currently, there is not such flag.
I thought that if users want to import one resource, autoplan is useful. But if try to import multiple imports, autoplan make the operation slow.
terraform import do not plan again, so I implements no-autoplan by default. We can provide such flag in the future as optional.

Also if an import command fails can we retain the plan instead of discarding it?

import_step_runner discard planfile if import has no err 👌

out, err := p.TerraformExecutor.RunCommandWithVersion(ctx, filepath.Clean(path), importCmd, envs, tfVersion, ctx.Workspace) // If the import was successful and a plan file exists, delete the plan. planPath := filepath.Join(path, GetPlanFilename(ctx.Workspace, ctx.ProjectName)) if err == nil {

Perhaps we can have an optional subcommand like we do for atlantis apply e.g. --auto-merge-disabled

For this case, perhaps a flag for --autoplan, --autoplan-on-success ? This could be added in a separate PR.

--autoplan looks good for import option. We can do atlantis pllan manually, so it is better to implement this after raised issue got votes.

runatlantis.io/docs/using-atlantis.md

runatlantis.io/docs/server-side-repo-config.md

nitrocode · 2022-12-21T07:07:33Z

runatlantis.io/docs/using-atlantis.md

@@ -124,6 +124,36 @@ Because Atlantis under the hood is running `terraform apply plan.tfplan`, any Te
 They're ignored because they can't be specified for an already generated planfile.
 If you would like to specify these flags, do it while running `atlantis plan`.

+---
+## atlantis import


Could you also update the tfvars section on the custom workflows page to show how to set it up for the import step? I've run into issues before where i need to pass in a custom tfvars file in order to get the import command to run correctly

https://www.runatlantis.io/docs/custom-workflows.html#tfvars-files

A test with tfvars may be useful too. Ignore me if that is overkill.

same with #2783 (comment)

Co-authored-by: nitrocode <[email protected]>

runatlantis.io/docs/custom-workflows.md

nitrocode · 2022-12-23T00:37:54Z

Thank you @krrrr38 ! This change is going to help a lot!!

jamengual · 2022-12-23T00:54:31Z

What an example of teamwork!!!! thanks @krrrr38 @nitrocode @Fabianoshz @albertollamaso

Dilergore · 2023-02-02T10:24:50Z

Hi @krrrr38,

I recently started to enable this feature in our environment. For most of the workflow steps, we have customizations (plan, apply). I've been trying to figure out how I could run terraform import from a script instead of the pre-built step.

We might have a missing feature here for custom workflows, that there is no environment var holding the ADDRESS and the ID to be able to customize this. Am I right? If so, I will raise a feature request.

krrrr38 · 2023-02-02T10:35:44Z

@Dilergore Thank you for using new feature.

ADDRESS ID are contains in COMMENT_ARGS in run step. But COMMENT_ARGS have 2 problems.

Values are contained with comma.
All characters are escaped by \.

This is also discussed in the Slack and there is a dirty workaround.. (slack invitation is existed in the repository README)
If you want to use more good solution, I'd be happy if you open a feature request or suggests good any ideas. Thanks.

Dilergore · 2023-02-02T10:58:07Z

@Dilergore Thank you for using new feature.

ADDRESS ID are contains in COMMENT_ARGS in run step. But COMMENT_ARGS have 2 problems.

Values are contained with comma.

All characters are escaped by \.

This is also discussed in the Slack and there is a dirty workaround.. (slack invitation is existed in the repository README) If you want to use more good solution, I'd be happy if you open a feature request or suggests good any ideas. Thanks.

Hi,

Thanks for the quick response, much appreciated.

I can confirm that the above is working, I was able to implement it as a custom step in the past 20 mins.

That said, I still think this should have its own dedicated variable. I will open a feature request.

krrrr38 force-pushed the feat-atlantis-import branch from 8a6ad94 to 65c4963 Compare December 13, 2022 00:34

krrrr38 marked this pull request as ready for review December 13, 2022 00:37

krrrr38 requested a review from a team as a code owner December 13, 2022 00:37

krrrr38 commented Dec 13, 2022

View reviewed changes

krrrr38 marked this pull request as draft December 13, 2022 01:09

krrrr38 force-pushed the feat-atlantis-import branch from 65c4963 to af56f82 Compare December 13, 2022 02:20

krrrr38 marked this pull request as ready for review December 13, 2022 02:20

krrrr38 force-pushed the feat-atlantis-import branch 2 times, most recently from d53c1a4 to c3f51e1 Compare December 13, 2022 04:48

krrrr38 force-pushed the feat-atlantis-import branch from c3f51e1 to ec0bdf6 Compare December 13, 2022 21:35

nitrocode added the waiting-on-review Waiting for a review from a maintainer label Dec 17, 2022

Fabianoshz suggested changes Dec 20, 2022

View reviewed changes

server/server.go Outdated Show resolved Hide resolved

server/events/comment_parser.go Show resolved Hide resolved

runatlantis.io/docs/using-atlantis.md Outdated Show resolved Hide resolved

server/server.go Show resolved Hide resolved

jamengual previously approved these changes Dec 20, 2022

View reviewed changes

krrrr38 force-pushed the feat-atlantis-import branch from ec0bdf6 to 39158d4 Compare December 21, 2022 05:50

krrrr38 commented Dec 21, 2022

View reviewed changes

runatlantis.io/docs/using-atlantis.md Show resolved Hide resolved

krrrr38 requested a review from Fabianoshz December 21, 2022 05:57

nitrocode reviewed Dec 21, 2022

View reviewed changes

runatlantis.io/docs/command-requirements.md Outdated Show resolved Hide resolved

nitrocode reviewed Dec 21, 2022

View reviewed changes

runatlantis.io/docs/using-atlantis.md Outdated Show resolved Hide resolved

nitrocode reviewed Dec 21, 2022

View reviewed changes

runatlantis.io/docs/using-atlantis.md Outdated Show resolved Hide resolved

nitrocode reviewed Dec 21, 2022

View reviewed changes

runatlantis.io/docs/using-atlantis.md Outdated Show resolved Hide resolved

nitrocode reviewed Dec 21, 2022

View reviewed changes

runatlantis.io/docs/server-side-repo-config.md Outdated Show resolved Hide resolved

nitrocode reviewed Dec 21, 2022

View reviewed changes

runatlantis.io/docs/server-side-repo-config.md Outdated Show resolved Hide resolved

nitrocode reviewed Dec 21, 2022

View reviewed changes

krrrr38 and others added 6 commits December 22, 2022 16:48

Update runatlantis.io/docs/command-requirements.md

3baf52e

Co-authored-by: nitrocode <[email protected]>

Update server/events/command_requirement_handler.go

1456da3

Co-authored-by: nitrocode <[email protected]>

Update server/events/command_requirement_handler_test.go

728c405

Co-authored-by: nitrocode <[email protected]>

fix test import usage

7ccde7c

fix e2e expected txt

9556879

fix doc link

7797de7

krrrr38 requested review from jamengual and nitrocode and removed request for Fabianoshz, jamengual and nitrocode December 22, 2022 08:57

nitrocode approved these changes Dec 22, 2022

View reviewed changes

nitrocode reviewed Dec 22, 2022

View reviewed changes

runatlantis.io/docs/custom-workflows.md Show resolved Hide resolved

nitrocode mentioned this pull request Dec 22, 2022

plan_requirements (similar to apply_requirements) #2073

Closed

krrrr38 added 4 commits December 23, 2022 03:48

docs: workflow import stage/step

79ba39e

Merge branch 'main' into feat-atlantis-import

0f42124

docs fixup

cd29e46

Merge branch 'main' into feat-atlantis-import

dee6b3c

nitrocode merged commit 4e0d4ec into runatlantis:main Dec 23, 2022

krrrr38 deleted the feat-atlantis-import branch December 23, 2022 01:37

nitrocode added this to the 0.22.0 milestone Dec 23, 2022

krrrr38 mentioned this pull request Dec 24, 2022

feat: --allow-commands restricts available atlantis commands #2877

Merged

Dilergore mentioned this pull request Feb 2, 2023

Feat: Dedicated variable for import commands #3094

Open

1 task

albertorm95 mentioned this pull request May 19, 2023

fix(metrics): add missing stat metric scopes #3416

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: atlantis import #2783

feat: atlantis import #2783

krrrr38 commented Dec 13, 2022 •

edited

Loading

nitrocode commented Dec 13, 2022

nitrocode commented Dec 19, 2022

Fabianoshz left a comment

albertollamaso commented Dec 20, 2022

jamengual commented Dec 20, 2022

nitrocode commented Dec 20, 2022

nitrocode Dec 21, 2022

krrrr38 Dec 21, 2022 •

edited

Loading

nitrocode Dec 22, 2022

nitrocode Dec 21, 2022

nitrocode Dec 21, 2022

krrrr38 Dec 21, 2022

nitrocode Dec 21, 2022

krrrr38 Dec 21, 2022

nitrocode Dec 21, 2022

nitrocode Dec 21, 2022

krrrr38 Dec 21, 2022

nitrocode commented Dec 23, 2022

jamengual commented Dec 23, 2022

Dilergore commented Feb 2, 2023 •

edited

Loading

krrrr38 commented Feb 2, 2023 •

edited

Loading

Dilergore commented Feb 2, 2023

	This command discards terraform plan result. Before apply, required `atlantis plan` again.
	This command discards the terraform plan result. After an import and before an apply, another `atlantis plan` must be run again.

feat: atlantis import #2783

feat: atlantis import #2783

Conversation

krrrr38 commented Dec 13, 2022 • edited Loading

what

why

references

discussion

nitrocode commented Dec 13, 2022

nitrocode commented Dec 19, 2022

Fabianoshz left a comment

Choose a reason for hiding this comment

albertollamaso commented Dec 20, 2022

jamengual commented Dec 20, 2022

nitrocode commented Dec 20, 2022

Choose a reason for hiding this comment

krrrr38 Dec 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nitrocode commented Dec 23, 2022

jamengual commented Dec 23, 2022

Dilergore commented Feb 2, 2023 • edited Loading

krrrr38 commented Feb 2, 2023 • edited Loading

Dilergore commented Feb 2, 2023

krrrr38 commented Dec 13, 2022 •

edited

Loading

krrrr38 Dec 21, 2022 •

edited

Loading

Dilergore commented Feb 2, 2023 •

edited

Loading

krrrr38 commented Feb 2, 2023 •

edited

Loading