Authentication issue

[hussainaphroj]

Describe the bug

I am getting NSXT different authentication errors during the plan and apply phase.

Getting "Error: Failed to retrieve NSX version (403 ). Please check connectivity and authentication settings of the provider" during terraform plan

incase of plane phase succeed without any issues then getting error during terraform apply but a different error that is "Error: The credentials were incorrect or the account specified has been locked. (code 403)"

I tried downgrading the nsxt provider to 3.2.6, 3.2.5 and vcd to 3.5.0 but still no luck.

My Environment:

Terraform v0.13.7
vmware/nsxt 3.2.6/3.2.5/3.2.7
NSX-T Manager: 3.1.2.0.0.17883596

The provider configuration looks as following:

terraform {
required_version = ">= 0.13"
required_providers {
nsxt = {
source = "vmware/nsxt"
version = "3.2.7"
}
}
}

provider "nsxt" {
host = "192.0.2.1"
username = "admin"
password = "default"
allow_unverified_ssl = true
max_retries = 10
retry_min_delay = 500
retry_max_delay = 3000
retry_on_status_codes = [429]
}

Logs:

2022-06-07T14:34:04.597+0400 [DEBUG] plugin: waiting for RPC address: path=.terraform/plugins/registry.terraform.io/vmware/vcd/3.5.0/linux_amd64/terraform-provider-vcd_v3.5.0
2022-06-07T14:34:04.610+0400 [INFO] plugin.terraform-provider-vcd_v3.5.0: configuring server automatic mTLS: timestamp=2022-06-07T14:34:04.610+0400
2022-06-07T14:34:04.625+0400 [DEBUG] plugin.terraform-provider-vcd_v3.5.0: plugin address: address=/tmp/plugin1776914615 network=unix timestamp=2022-06-07T14:34:04.625+0400
2022-06-07T14:34:04.625+0400 [DEBUG] plugin: using plugin: version=5
2022/06/07 14:34:05 [DEBUG] Resource state not found for node "data.vcd_external_network_v2.nsxt-ext-net", instance data.vcd_external_network_v2.nsxt-ext-net
2022/06/07 14:34:05 [WARN] ReferenceTransformer: reference not found: "var.edge_external_network_name"
2022/06/07 14:34:05 [DEBUG] ReferenceTransformer: "data.vcd_external_network_v2.nsxt-ext-net" references: []
2022-06-07T14:34:05.132+0400 [INFO] plugin.terraform-provider-vcd_v3.5.0: 2022/06/07 14:34:05 [TRACE] external network V2 data source read initiated: timestamp=2022-06-07T14:34:05.132+0400
2022-06-07T14:34:05.189+0400 [WARN] plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2022-06-07T14:34:05.191+0400 [DEBUG] plugin: plugin process exited: path=.terraform/plugins/registry.terraform.io/vmware/vcd/3.5.0/linux_amd64/terraform-provider-vcd_v3.5.0 pid=4036526
2022-06-07T14:34:05.191+0400 [DEBUG] plugin: plugin exited
2022/06/07 14:34:20 [ERROR] eval: *terraform.EvalConfigProvider, err: Failed to retrieve NSX version (403 ). Please check connectivity and authentication settings of the provider
2022/06/07 14:34:20 [ERROR] eval: *terraform.EvalSequence, err: Failed to retrieve NSX version (403 ). Please check connectivity and authentication settings of the provider
2022/06/07 14:34:20 [ERROR] eval: *terraform.EvalOpFilter, err: Failed to retrieve NSX version (403 ). Please check connectivity and authentication settings of the provider
2022/06/07 14:34:20 [ERROR] eval: *terraform.EvalSequence, err: Failed to retrieve NSX version (403 ). Please check connectivity and authentication settings of the provider
2022-06-07T14:34:20.090+0400 [WARN] plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2022-06-07T14:34:20.092+0400 [DEBUG] plugin: plugin process exited: path=.terraform/plugins/registry.terraform.io/vmware/nsxt/3.2.5/linux_amd64/terraform-provider-nsxt_v3.2.5 pid=4036517

Reproduction steps

terraform {
  required_version = ">= 0.13"
  required_providers {
    nsxt = {
      source  = "vmware/nsxt"
      version = "3.2.7"
    }
  }
}

provider "nsxt" {
  host                  = "192.0.2.1"
  username              = "admin"
  password              = "default"
  allow_unverified_ssl  = true
  max_retries           = 10
  retry_min_delay       = 500
  retry_max_delay       = 3000
  retry_on_status_codes = [429]
}

Expected behavior

NSXT authentication should succeed

Additional context

No response

[salv-orlando]

Hello - thanks for reporting this issue!

The routine were we are having this failure is https://github.com/vmware/terraform-provider-nsxt/blob/master/nsxt/utils.go#L545, which is called when the NSX client for the provider is being initialized.
The provider tries to read the NSX version by querying the endpoint /api/v1/node and fetching NSX version from the response. The Terraform provider needs to be aware of the NSX version so that all the features for that specific version are enabled.

Unfortunately, in your case NSX is not allowing access to this URI endpoint. The admin user is always expected to be able to access api/v1/node. Therefore what probably needs to be understood here is why NSX is rejecting this operation.

There does not seem to be any issue with the provider, as it is simply reporting the response it receives from NSX.

Can you please verify whether curl -ku admin:default https://<nsx_mgr>/api/v1/node returns a 403 error? Also it might be worth checking instead that operations on "standard" API endpoints like /policy/api/v1/infra/segments are still working correctly.

[sneal]

I ran into the same error because of a special character in my password, my password was something like mypass@&word where &word was getting truncated off in my bash script I was running via SSH before getting passed to Terraform.

[olivierHa]

Hello, I think I have the same issue here.

curl to the /api/v1/node works , but after some activity (plan/apply), it gives 403 error.

{"module_name":"common-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403}

Scenario :

curl ok

terraform plan/apply with TF_LOG=debug gives :

`
2023-01-19T09:33:26.077Z [DEBUG] created provider logger: level=debug
2023-01-19T09:33:26.078Z [INFO] provider: configuring client automatic mTLS
2023-01-19T09:33:26.095Z [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/vmware/nsxt/3.2.9/linux_amd64/terraform-provider-nsxt_v3.2.9 args=[.terraform/providers/registry.terraform.io/vmware/nsxt/3.2.9/linux_amd64/terraform-provider-nsxt_v3.2.9]
2023-01-19T09:33:26.096Z [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/vmware/nsxt/3.2.9/linux_amd64/terraform-provider-nsxt_v3.2.9 pid=46
2023-01-19T09:33:26.096Z [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/vmware/nsxt/3.2.9/linux_amd64/terraform-provider-nsxt_v3.2.9
2023-01-19T09:33:26.103Z [INFO] provider.terraform-provider-nsxt_v3.2.9: configuring server automatic mTLS: timestamp=2023-01-19T09:33:26.103Z
2023-01-19T09:33:26.121Z [DEBUG] provider: using plugin: version=5
2023-01-19T09:33:26.121Z [DEBUG] provider.terraform-provider-nsxt_v3.2.9: plugin address: address=/tmp/plugin2439669977 network=unix timestamp=2023-01-19T09:33:26.121Z
2023-01-19T09:33:26.180Z [WARN] ValidateProviderConfig from "provider["registry.terraform.io/vmware/nsxt"]" changed the config value, but that value is unused

2023-01-19T09:33:26.344Z [INFO] provider.terraform-provider-nsxt_v3.2.9: 2023/01/19 09:33:26 Warning: Failed to create session: status code 403: timestamp=2023-01-19T09:33:26.344Z

2023-01-19T09:33:26.486Z [ERROR] vertex "provider["registry.terraform.io/vmware/nsxt"]" error: Failed to retrieve NSX version (403 ). Please check connectivity and authentication settings of the provider
╷
│ Error: Failed to retrieve NSX version (403 ). Please check connectivity and authentication settings of the provider
│
│ with provider["registry.terraform.io/vmware/nsxt"],
│ on terraform.tf line 18, in provider "nsxt":
│ 18: provider "nsxt" {
│
╵
`

curl nok

[github-actions]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[DominicMagnus]

Have the same problem on NSX 4.1.2

terraform plan

Planning failed. Terraform encountered an error while generating this plan.

│ Error: Failed to retrieve NSX version (com.vmware.vapi.std.errors.service_unavailable). Please check connectivity and authentication settings of the provider
│
│ with provider["registry.terraform.io/vmware/nsxt"],
│ on nsx.tf line 10, in provider "nsxt":
│ 10: provider "nsxt" {
│

[ksamoray]

Have the same problem on NSX 4.1.2

terraform plan

Planning failed. Terraform encountered an error while generating this plan.

│ Error: Failed to retrieve NSX version (com.vmware.vapi.std.errors.service_unavailable). Please check connectivity and authentication settings of the provider │ │ with provider["registry.terraform.io/vmware/nsxt"], │ on nsx.tf line 10, in provider "nsxt": │ 10: provider "nsxt" { │

@DominicMagnus this usually indicates a failure with connecting to NSX manager.
Can you retry while exporting the environment variable TF_LOG=debug?

[ShipInB]

Same issue here which spawn randomly

│ Error: Failed to retrieve NSX version, please check connectivity and authentication settings of the provider: The credentials were incorrect or the account specified has been locked. (code 403)
│
│ with provider["registry.terraform.io/vmware/nsxt"],
│ on 01_data.tf line 22, in provider "nsxt":
│ 22: provider "nsxt" {
│

Terraform v1.5.5
vmware/nsxt v3.6.0
NSX v4.1.1

[salv-orlando]

We recently fixed an issue with could lead to intermittent authentication failures due to incorrect escaping of passwords. (this could be the same issue mentioned by @sneal )

this was fixed with PR #1216, the fix has been released with v3.6.2.

Please let us know if you are still seeing intermittent authentication issues.

[nickliuibm]

Hello,

We are still seeing this issue and our password does not include the special characters. Sometimes a few retries can workaround this failure. We are using
Terraform v1.5.5
vmware/nsxt v3.2.8
NSX v4.1.1

[ Terraform ] -> �[31m╷�[0m�[0m
[ Terraform ] -> �[31m│�[0m �[0m�[1m�[31mError: �[0m�[0m�[1mFailed to retrieve NSX version (403 Forbidden). Please check connectivity and authentication settings of the provider�[0m
[ Terraform ] -> �[31m│�[0m �[0m
[ Terraform ] -> �[31m│�[0m �[0m�[0m with provider["registry.terraform.io/terraform-provider-nsxt/nsxt"],

[annakhm]

Could you please reproduce this with the following variables set in the environment, and share the logs:
TF_LOG=debug
TF_LOG_PROVIDER_NSX_HTTP=debug

This would allows us to see full HTTP request/response parameters (with sensitive info being omitted)