-
Notifications
You must be signed in to change notification settings - Fork 192
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support parallel import #260
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you help explain for me @magodo why a user might want to turn off parallelism and use less than 10 parallel operations? If this is working functionally, it seems like it's only best to run with as many parallel operations as you can, thus I'd imagine we could implement this in the backend but make the feature invisible to the end user to simplify the already growing list of flags.
@stemaMSFT Agree to remove the flag and makes 10 parallelism to be the default. The reason why users might want to explicitly set the parallelism to 1 can be avoiding the Azure throttling issue. I'll go on removing the flag and merging the PR then. |
This PR adds support for parallel import.
Currently, the resources are imported one by one. This is because
aztfy
runsterraform import
under the hood, which by default locks the state file during importing a single resource to avoid inference to the state file at the same time.Since terraform v1.1.0, the
terraform state mv
command can be used to move resources among different state file. With that, we can import resources in parallel to different working directories. After that, usingterraform state mv
to move them into a single state file. This applies to both local and remote backends. This logic is implemented in https://github.com/magodo/tfmerge.The process is now like below:
Current implementation introduced a new option
--parallel-import
, which controls whether to import in paralle or sequential. Meanwhile, the--parallelism
controls the level of parallelism, which defaults to 10. I'm not sure whether we shall always make--parallel-import
to be true (hencen remove it). The concern is that there is still a small un-protected window between pull and push state.Fix #206.