Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support parallel import #260

Merged
merged 12 commits into from
Oct 28, 2022
Merged

Support parallel import #260

merged 12 commits into from
Oct 28, 2022

Conversation

magodo
Copy link
Collaborator

@magodo magodo commented Oct 14, 2022

This PR adds support for parallel import.

Currently, the resources are imported one by one. This is because aztfy runs terraform import under the hood, which by default locks the state file during importing a single resource to avoid inference to the state file at the same time.

Since terraform v1.1.0, the terraform state mv command can be used to move resources among different state file. With that, we can import resources in parallel to different working directories. After that, using terraform state mv to move them into a single state file. This applies to both local and remote backends. This logic is implemented in https://github.com/magodo/tfmerge.

The process is now like below:

  • Pull state (original state)
  • Import resources in parallel
  • Pull state again to ensure it is the same as the original state. This is to mimic the terraoform lock
  • Push state

Current implementation introduced a new option --parallel-import, which controls whether to import in paralle or sequential. Meanwhile, the --parallelism controls the level of parallelism, which defaults to 10. I'm not sure whether we shall always make --parallel-import to be true (hencen remove it). The concern is that there is still a small un-protected window between pull and push state.

Fix #206.

@magodo magodo added the enhancement New feature or request label Oct 14, 2022
@magodo magodo requested review from ms-henglu and stemaMSFT October 14, 2022 03:32
Copy link
Member

@stemaMSFT stemaMSFT left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you help explain for me @magodo why a user might want to turn off parallelism and use less than 10 parallel operations? If this is working functionally, it seems like it's only best to run with as many parallel operations as you can, thus I'd imagine we could implement this in the backend but make the feature invisible to the end user to simplify the already growing list of flags.

@magodo
Copy link
Collaborator Author

magodo commented Oct 28, 2022

@stemaMSFT Agree to remove the flag and makes 10 parallelism to be the default. The reason why users might want to explicitly set the parallelism to 1 can be avoiding the Azure throttling issue. I'll go on removing the flag and merging the PR then.

@magodo magodo merged commit 1cf5ef9 into Azure:main Oct 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Import resources in parallel
3 participants