How to debug terraform

[d-apurva]

Is there any better way to debug terraform provider other than using TF_LOG=DEBUG.
Please list the steps required to reproduce the log which i did.

set TF_LOG=DEBUG
set TF_TF_LOG_PATH=/tmp/log
terraform apply
observe TRACE level logs in the file /tmp/log

Please let me know if there is better way than this.

[tombuildsstuff]

hey @d-apurva

Thanks for opening this issue

Unfortunately debugging a Terraform Provider is complicated due to the fact that Plugins are run in separate processes - that said there's two approaches to debugging Providers:

1. Using the environment variables mentioned above to set the log level and parsing them. This generally contains the HTTP logs - but also contains the output from Terraform Core

2. Using an HTTP Proxy such as Charles.app or Fiddler (or any other Proxy) to intercept the HTTP(S) traffic via http_proxy=http://localhost:8888 https_proxy=http://localhost:8888 terraform plan - which will show the HTTP/HTTPS traffic from the Provider to the API.

From there you should be able to determine the necessary changes to the Providers (sometimes outputting logs via log.Printf() and the first approach can come in handy) and re-compile as outlined in the BUILDING.md file which should allow you to debug further :)

Thanks!

[domdom82]

@tombuildsstuff thanks for shedding some light on this. I am desperately trying to fix something in the AWS provider but haven't even found my logs in stdout yet :)

IMHO this should be documented in either the main "Debugging Terraform" page here https://www.terraform.io/docs/internals/debugging.html or at some new page specific to plugin development.

[zlesnr]

I too am trying to write a terraform provider and I'm not able to get the information out of my code. I've got the log level set to DEBUG, and I've got plenty of log.Println() lines, but don't see any of them.

[techtonik]

So no standard way to inspect stdout/stderr from plugins?

[Didainius]

Is there a way to get access to some resource type name inside terraform provider or something like "request ID"?

I do write logs inside provider, but the problem is that if I provision many things it is becoming almost impossible to read the log as there is too much going on in parallel. Log messages are missing something like correlation_id or whatever to filter it out and see trace for one operation rather than trying to understand this manully.
I could add them to logs if I had access to them.

[bcampoli]

Any update on this? Not having a straight forward way to collect provider logs makes debugging very difficult. Even a blog post on how to set up the proxy mentioned above for debugging would help.

[eedwards-sk]

I'm bit by this right now trying to debug the AWS provider when doing iterative IAM permissions development.

It's very common that you need to keep running terraform over and over again to figure out what permissions are needed, so getting detailed information from the provider (aws) is highly valuable.

[JBirdVegas]

Any update on this? I'm debugging my custom plugin and just realized the first meaningful line of output from my plugin started on line 17,277 of the LF_LOG... Scrolling through 17k+ output lines to debug is a horrible experience.

[tamsky]

@tombuildsstuff I have a few questions about what you described:

Unfortunately debugging a Terraform Provider is complicated due to the fact that Plugins are run in separate processes - that said there's two approaches to debugging Providers:

1. Using the environment variables mentioned above to set the log level and parsing them. This generally contains the HTTP logs - but also contains the output from Terraform Core

My TF_LOGs contain Terraform Core log output interspersed with Provider log output...

2019/07/24 03:09:09 [TRACE] [walkPlan] Entering eva2019-07-24T03:09:09.896-0700 [DEBUG] plugin.terraform-provider-aws_v2.20.0_x4: 2019/07/24 03:09:09 [DEBUG] Canonical definitions are not equal.

Notice the truncated TRACE output.

Is there any way around this?

It'd be really nice if one could simply disable the Terrafrom Core TF_LOG, but leave it enabled for all Providers.

[tombuildsstuff]

@tamsky

Is there any way around this?

Not at this time that I'm aware of unfortunately, the TF_LOG environment variable is passed between Terraform Core and the Providers at the moment.

---

To give an update on the original "how to debug a Terraform Provider" question: I'm aware of some folks using VSCode's debugger to do this - by placing a breakpoint in the Resource's Create/Read method - and then debugging an Acceptance Test, which will end up calling into the Create/Read methods of the Resource. I've not tried this personally - but since other folks have mentioned this approach works for them I felt it's worth noting.

[tamsky]

Not at this time that I'm aware of unfortunately, the TF_LOG environment variable is passed between Terraform Core and the Providers at the moment.

Which approach might be preferred toward improving the current situation?

Create:

1. a TF_LOG_PROVIDER env var, or
2. a TF_LOG_ATOMIC env var flag/feature?

TF_LOG_PROVIDER could set the TF_LOG env var seen by all Providers.
Perhaps even as granular as: TF_LOG_PROVIDER_aws.

TF_LOG_ATOMIC could alter the logging library's behavior to buffer all log messages until a newline -- after which it would perform a synchronous write() + fflush().

[asubmani]

@tamsky
I'm aware of some folks using VSCode's debugger to do this - by placing a breakpoint in the Resource's Create/Read method - and then debugging an Acceptance Test, which will end up calling into the Create/Read methods of the Resource. I've not tried this personally - but since other folks have mentioned this approach works for them I felt it's worth noting.

Hi @tombuildsstuff Would you know any repositories or documentation on how to use vscode to debug terraform?

[PSanetra]

I have submitted a PR, which would make it easier to filter for log lines of certain terraform providers: hashicorp/terraform#25086

[paddycarver]

👋 hey y'all, so I took a look into some of the concerns behind this.

First, good news: improved support for debugging providers using tools like delve is planned for the v2 release of the SDK, which we're working on pushing out the door now. You'll need Terraform 0.12.26 or higher to take advantage of it, and your provider will need to be using v2 of the SDK, but ti should allow debugging with delve in tests and with production binaries. We will publish documentation on how to do this when it lands.

Second, bad news: this doesn't fix the logging situation. As a println-based debugger myself, I recognise the current logging situation isn't ideal. I've had success in my own provider development with logging to a file, using unique strings in log messages, and searching for those strings, but that's obviously a... rough experience that we could improve on. Unfortunately, Terraform's logging situation could stand some reworking, and that work isn't planned in the immediate future. I had a fruitful conversation about a design direction for logging just a provider's output to a specific file, however, and that may be an incremental improvement we can deliver before Terraform's logging situation gets updated. Given that we're busy with getting v2 of the SDK out the door, however, and that the core team is busy getting 0.13 out the door, this incremental improvement is likely going to need to wait until the dust clears on those releases before it can be undertaken. And even then, it would require coordination between the core team and the SDK team, and we both have lengthy backlogs, so I can't promise a timeline, just that someone has looked at it and found a potential path forward.

[PSanetra]

@paddycarver did you have a look at the PR hashicorp/terraform#25086? I think it would greatly improve the logging situation without any changes in the terraform-plugin-sdk and with minimal changes in terraform-core.

[paddycarver]

Hey @PSanetra! I took a look at the PR in question, and I think that's a question for the Core team to answer. Right now, the logging infrastructure in Terraform is just fiddly, and they want to overhaul it entirely, because it can be hard to reason about after years of accreting layers of patches. So even though your change is small in terms of code, it still poses risk in terms of unknown interactions. How big a risk and how willing they are to take it is a question that I can't answer for them, unfortunately. I imagine after 0.13 gets wrapped up and delivered, you'll get some more engagement there.

[ScottSuarez]

First, good news: improved support for debugging providers using tools like delve is planned for the v2 release of the SDK, which we're working on pushing out the door now. You'll need Terraform 0.12.26 or higher to take advantage of it, and your provider will need to be using v2 of the SDK, but ti should allow debugging with delve in tests and with production binaries. We will publish documentation on how to do this when it lands.

Hey @paddycarver, was wondering if there was any update on possibly getting some documentation for working with Delve and Terraform.

[cemckenzie]

To anyone reading this thread, please use TF_LOG=TRACE and not DEBUG, TRACE has additional internal information. Also to avoid issues of multithreaded log messages, you can set -parallelism=1 on the command line and TFE_PARALLELISM=1 in cloud/enterprise to single thread the run.

[paddycarver]

I've opened hashicorp/terraform-website#1584 to add some docs on the state of debugging as of 0.15.0. I'll make sure these documents stay updated as we continue to build out support for more advanced and powerful debugging capabilities.

[paddycarver]

Our docs are now live on the website: https://www.terraform.io/docs/extend/debugging.html

There's also #695 available now for comment. It's a proposal for the solution I think will close out this issue. While making Terraform providers more debuggable is something we'll always be interested in and working towards, I think that represents the end of a focused project on the subject. We're very interested in hearing your thoughts and feedback. :)

[satyamuralidhar]

Is there any better way to debug terraform provider other than using TF_LOG=DEBUG.
Please list the steps required to reproduce the log which i did.

set TF_LOG=DEBUG
set TF_TF_LOG_PATH=/tmp/log
terraform apply
observe TRACE level logs in the file /tmp/log

Please let me know if there is better way than this.

is that we use logs set path log.sh file and call it from main.tf using provisioners. is this the right approach or any different kind of approach for logs

[Feliksas]

First, good news: improved support for debugging providers using tools like delve is planned for the v2 release of the SDK, which we're working on pushing out the door now. You'll need Terraform 0.12.26 or higher to take advantage of it, and your provider will need to be using v2 of the SDK, but ti should allow debugging with delve in tests and with production binaries. We will publish documentation on how to do this when it lands.

Looks like it's not entirely true - debugging doesn't work with any version between 0.12.26 and 0.13.0, seems like it's broken in the 0.12.x branch

[paddycarver]

I'd love a separate issue open, with information on how to reproduce and what "doesn't work" means in this context/what provider code is set up. :)

[Feliksas]

I don't know if it is really worth a separate issue, since v0.12.x branch is long deprecated, and probably only weirdos like me still tinker with it :) But basically, if you follow the manual and set up the TF_REATTACH_PROVIDERS environment variable, Terraform seems to ignore it and still looks for the provider binary in ~/.terraform.d/ Doing the very same with v0.13.0 seems to do the thing just right

[bflad]

Hi folks 👋

There is information about how to setup recent versions of the SDK with additional logging and debuggers such as delve available at https://www.terraform.io/plugin/sdkv2/debugging. If you have a question about this functionality, please start a new topic in the HashiCorp Discuss forums. If you have specific bug report or feature request, please open a new issue and we can take a fresh look. Thanks!

[hablutzel1]

To debug old Terraform providers (pre Terraform Plugin SDK versions 2.0.0 if I'm not wrong), you can use the technique described in https://groups.google.com/g/delve-dev/c/RTtmwkJe5Jw/m/sD1RylCWGwAJ.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.