New subcommand: query to support customized set of resources
#231
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
TODO: - Cross resource dependencies of non-parent-child relationship - Resources needs to be ignored but now listed (e.g. OS disk of vm) - Resources needs to be listed but ignored (e.g. nsr, subnet, etc)
…id is used for import When providing the resource mapping file, it is matched case sensitvely before. Changing it to insensitive is to mitigate a bug that ARG will return its first resource id with RG name uppercased, which will cause the mismatch if compares case sensitively. Additionally, the resource id that is recorded in the mapping file will be used as the tf id for importing (case sensitively). This is useful when aztft has bugs that returns the incorrect casing for certain resources, users can tune the resource mapping file to get the correct casing. Whilst there might be cases that the resource id is not the same as tf id not only in casing. In that case, we will need to further modify the format of resource mapping file to accomodate it.
…he parent-child dependency late prior to HCL generation
- Introduce a hidden flag `--plain-ui`, which is used for e2e test to stream the test output - Add more stage related log during e2e test - Introduce delay in e2e test right before apply, to wait for the created resources be recorded in ARG
There is another option that discovers the dependencies on API models, just like ARM does. This might save us the dependency for cases like VM disk depends on VM, which is now missing because that dependency is introduced by the attachment resource. However, that can only save a very small amount of cases, but there are still a big part of dependencies can't be discovered by simply looking into API properties, which are heavily depending on the business (e.g. sentinel resources depends on an operation insight resource). For catching up the gap of the previous VM depends on disk, we shall consider introducing the association resources in the azure resource set hack code. This is the great place for holding such kind of code, as we are already using it to populate the missing managed disk in this case. If we go through this direction, then discovering reference dependencies based on HCL is the right choice here, as it will then be able to add on the dependencies for the association resource on the associated resources.
…ent` This commit introduce a new field `PesudoResourceInfo` in the `AzureResource` to record TF pesudo resources, which will later be used when converting into `TFResource`. In this case, rather than using aztft to look for the TF resource type based on the concrete azure resource, it directly consume the info recorded in the `PesudoResourceInfo`. With above, now resourceset hack not only populates the managed disk, but also the disk attachement resource.
…ery mode to support both scope/specific predicates
stemaMSFT
reviewed
Sep 21, 2022
There was a problem hiding this comment.
Great work Zhaoting! Can't understate how much benefit this is going to bring to our users. Customers will be very excited to try this out :) I'll leave it to @ms-henglu to check the technical coding aspects of this PR and do the final approval, but in terms of features/functionality, I have no problems with it.
E.g. `aztfy res <vm id>` now might terrafies three resources: vm, managed disk and disk attachement.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces a 3rd mode (among the
rgmode andresmode):querymode. It asks users to specify an Azure Resource Graphwherepredicate, and terrafying those resources. Optionally, when--recursive(newly introduced for this mode) is set, all the child resources of the queried resources will be included.Additionally, in order to improve the performance of listing resource, we also introduced a common option:
--parallelismto allow parallelized resource list. This can extend for other scenarios, e.g. parallel import, in the future.The resource dependency is now resolved by HCL.
The VM is now correctly populates the disk attachemnt resource (previously, only the managed data disk is populated). This is also the case for single resource mode, i.e. if you terrafies a single VM resource that has one data disk attached, it will results into three TF resources: vm, managed disk and the disk attachment resource, with correct dependencies added. This will be extended to support other AzureRM provider association resources, where the full list can be found at: https://github.com/magodo/aztft/blob/3dfe6a12ed5ee515b6f9281ed307c54211ee35b3/tool/aztft-import/main.go#L45.
Internal
Internally, we removed the dependency on ARM template export for the
rgmode. Instead, it uses the ARG querywhere resourceGroup =~ "my-rg"in recursive mode, with the rg itself added on. The recursively listing child resource logic is implemented in another depending module: https://github.com/magodo/azlist. That is basically how ARM template export lists resources.Another benefit that is brought by ARM template is the cross resource dependency. ARM discovers the dependencies by walking through the API model of each resource.
aztfynow discovers the dependency at two dimensions:"Discover by API model" vs "Discover by HCL model"
The API model might save us the dependency for cases like VM disk depends on VM, which will be missing if discovering by HCL model, because that dependency is introduced by the attachment resource. However, that can only save a very small amount of cases, but there are still a big part of dependencies can't be discovered by simply looking into API properties, which are heavily depending on the business (e.g. sentinel resources depends on an operation insight resource).
For catching up the gap of the previous VM depends on disk, we introduced the association resources in the azure resource set hack code. This is the great place for holding such kind of code, as we are already using it to populate the missing managed disk in this case. If we go through this direction, then discovering reference dependencies based on HCL is the right choice here, as it will then be able to add on the dependencies for the association resource on the associated resources (the dependency of vm depends on managed disk in the previous version is actually not so correct).
Closed Issues