Significant performance decrease v0.11.0

[sofbkeller]

Since upgrading from the V0.09.0 release to V0.11.0 I have noticed a significant decrease in the performance of AzTFExport and a increase in the amount of system resources a single concurrent export requires. Is this an expected change?
Previously on windows using an i7-1185G7 I was able to have a export running while still being able to utilize the system for other tasks. Under this latest update it consistently fully utilizes the CPU and the overall progress of the action takes at least an hour to pull from Azure.

Any troubleshooting tips or workarounds that can be used?

[magodo]

@sofbkeller There should be anything related to performance being introduced between v0.9.0 to v0.10.0 (since v0.11.0 only contains changes for renaming).

Some suspicious bits:

• Telemetry collection is introduced in New command config [get|set|show] and collecting telemetry #360, which you can opt out by aztfy config set telemetry_enabled false. But this shouldn't cause resource penalty
• The terraform binary being used is the one in your PATH, while previously if there isn't such a terraform found, aztfy downloads one for you
• The provider being used is now v3.46.0, while v0.9.0 was using v3.31.0

May I ask how many resources are being exported in your case? 1hr sounds to be really long to me. Also, are you using any remote backend for storing state?

FYI, if you are only interested to exporting the HCL - and ignoring the state - then you might be interested in #381 (merged but not released yet).

[sofbkeller]

@magodo - I am not sure how to narrow it down but it is definitely working better when I have rolled back to V0.9.0.

Generally, the RGs I am exporting are pretty large with around 500-4000 objects being exported. I was exporting all the RGs in a sub yesterday and it only got through about 20 RGs over 2.5 hours of sequential runs. I am doing an import all and skip anything that errors.

The backend I am storing the state is just the local system. We are not using this to TFy existing resources but more to take an indirect output of what exists.
The ability to not keep state would be desirable for sure since all we do care about IS the HCL. So, I will be looking forward to that release.

Another potentially unrelated issue I am noticing is that when the import generates a RG folder and pulls down the AzureRM provider into the .terraform folder, it also does it on the fly in the %TEMP% directory (Windows) and for each interaction it also seems to pull down the AzureRM provider there as well leading to a bunch of unnecessary I/O.

[stemaMSFT]

@sofbkeller —hcl-only is currently available to use in any of the versions you currently use. The flag magodo mentioned is purely for optimization of that import/export.

[magodo]

@sofbkeller Downloading the provider in the temp directory is as expected, since those temp directories are meant to parallelize the importing, which means they also need a reference to the provider. One way to speed it up is you can setup the provider cache, so that the provider will only be downloaded for the 1st time.

Regarding your scenario, you can pre-download the provider to somewhere, then run aztfexport <subcommand> -n -hcl-only -tfclient-plugin-path=<path-to-provider> ARGS.... This should speed up a lot.

[magodo]

@sofbkeller As mentioned above, with v0.12.0 you can use the new -tfclient-plugin-path in HCL mode, which boost up the performance a lot, and consumes much lower resources. With this, I'm going to close this issue for now. Please feel free to reopen it if the issue remains.