net/http: TLS handshake timeout when using multiple providers in Terraform0.12

[varmax2511]

Terraform Version

Terraform v0.12.6

Terraform Configuration Files

# providers for all regions

provider "oci" {
  tenancy_ocid     = "${var.tenancy_ocid}"
  user_ocid        = "${var.user_ocid}"
  fingerprint      = "${var.fingerprint}"
  private_key_path = "${var.private_key_path}"
  region           = "us-phoenix-1"
  alias            = "phx"
}

provider "oci" {
  tenancy_ocid     = "${var.tenancy_ocid}"
  user_ocid        = "${var.user_ocid}"
  fingerprint      = "${var.fingerprint}"
  private_key_path = "${var.private_key_path}"
  region           = "us-ashburn-1"
  alias            = "iad"
}

provider "oci" {
  tenancy_ocid     = "${var.tenancy_ocid}"
  user_ocid        = "${var.user_ocid}"
  fingerprint      = "${var.fingerprint}"
  private_key_path = "${var.private_key_path}"
  region           = "eu-frankfurt-1"
  alias            = "fra"
}

provider "oci" {
  tenancy_ocid     = "${var.tenancy_ocid}"
  user_ocid        = "${var.user_ocid}"
  fingerprint      = "${var.fingerprint}"
  private_key_path = "${var.private_key_path}"
  region           = "uk-london-1"
  alias            = "lhr"
}

provider "oci" {
  tenancy_ocid     = "${var.tenancy_ocid}"
  user_ocid        = "${var.user_ocid}"
  fingerprint      = "${var.fingerprint}"
  private_key_path = "${var.private_key_path}"
  region           = "ca-toronto-1"
  alias            = "yyz"
}

provider "oci" {
  tenancy_ocid     = "${var.tenancy_ocid}"
  user_ocid        = "${var.user_ocid}"
  fingerprint      = "${var.fingerprint}"
  private_key_path = "${var.private_key_path}"
  region           = "ap-tokyo-1"
  alias            = "nrt"
}

provider "oci" {
  tenancy_ocid     = "${var.tenancy_ocid}"
  user_ocid        = "${var.user_ocid}"
  fingerprint      = "${var.fingerprint}"
  private_key_path = "${var.private_key_path}"
  region           = "ap-seoul-1"
  alias            = "icn"
}

provider "oci" {
  tenancy_ocid     = "${var.tenancy_ocid}"
  user_ocid        = "${var.user_ocid}"
  fingerprint      = "${var.fingerprint}"
  private_key_path = "${var.private_key_path}"
  region           = "ap-mumbai-1"
  alias            = "bom"
}

provider "oci" {
  tenancy_ocid     = "${var.tenancy_ocid}"
  user_ocid        = "${var.user_ocid}"
  fingerprint      = "${var.fingerprint}"
  private_key_path = "${var.private_key_path}"
  region           = "eu-zurich-1"
  alias            = "zrh"
}

provider "oci" {
  tenancy_ocid     = "${var.tenancy_ocid}"
  user_ocid        = "${var.user_ocid}"
  fingerprint      = "${var.fingerprint}"
  private_key_path = "${var.private_key_path}"
  region           = "sa-saopaulo-1"
  alias            = "gru"
}

# data source for all DRGs

data "oci_core_vcns" "phx_drgs" {
  provider       = "oci.phx"
  compartment_id = "${var.compartment_ocid}"
}

data "oci_core_vcns" "iad_drgs" {
  provider       = "oci.iad"
  compartment_id = "${var.compartment_ocid}"
}

data "oci_core_vcns" "fra_drgs" {
  provider       = "oci.fra"
  compartment_id = "${var.compartment_ocid}"
}

data "oci_core_vcns" "lhr_drgs" {
  provider       = "oci.lhr"
  compartment_id = "${var.compartment_ocid}"
}

data "oci_core_vcns" "yyz_drgs" {
  provider       = "oci.yyz"
  compartment_id = "${var.compartment_ocid}"
}

data "oci_core_vcns" "nrt_drgs" {
  provider       = "oci.nrt"
  compartment_id = "${var.compartment_ocid}"
}

data "oci_core_vcns" "icn_drgs" {
  provider       = "oci.icn"
  compartment_id = "${var.compartment_ocid}"
}

data "oci_core_vcns" "bom_drgs" {
  provider       = "oci.bom"
  compartment_id = "${var.compartment_ocid}"
}

data "oci_core_vcns" "zrh_drgs" {
  provider       = "oci.zrh"
  compartment_id = "${var.compartment_ocid}"
}

data "oci_core_vcns" "gru_drgs" {
  provider       = "oci.gru"
  compartment_id = "${var.compartment_ocid}"
}

Debug Output

2019-08-13T16:12:31.384-0700 [DEBUG] plugin.terraform-provider-oci_v3.37.0_x4: DEBUG 2019/08/13 16:12:31.384344 log.go:168: Get https://.........: net/http: TLS handshake timeout

Expected Behavior

Plan should have succeeded.

Actual Behavior

Plan fails with TLS handshake error consistently when using Terraform0.12.6. The plan succeeds when using Terraform v0.11.11 or when using Terraform 0.12.6 with terraform plan -parallelism=1

Steps to Reproduce

1. terraform init
2. terraform plan

Additional Context

The issue is reproduced consistently in the above script which has 10 Terraform OCI provider, for 10 different regions and has 10 data sources, one per provider.
When Terraform plan is executed on this script using Terraform 0.11.11 , the script always executes successfully. However, when the script is executed using Terraform 0.12.6, some of the datasources always return net/http: TLS handshake timeout. The exact datasources change based on execution but the issue is always reproduced.
When setting the -parallelism=1 with Terraform 0.12, the script executes successfully. I tried running the script on different terraform versions since 0.12.4 to 0.12.6 and different OCI provider versions am able to reproduce the issue.

[ghost]

This issue has been automatically migrated to terraform-providers/terraform-provider-oci#852 because it looks like an issue with that provider. If you believe this is not an issue with the provider, please reply to terraform-providers/terraform-provider-oci#852.

[alexng-canuck]

This should be treated as a Terraform issue because the behaviors we're seeing are different between Terraform v0.11 vs. Terraform v0.12.

[teamterraform]

Hi @alexng-canuck,

The Terraform SDK does not itself make any HTTPS requests, so we're not sure what exactly we could change on the SDK side to improve this situation. Do you know what exactly the OCI provider is doing when it encounters this problem?

When running under Terraform 0.12, there will be one provider process running per provider block, whereas Terraform 0.11 (due to a different plugin design) would sometimes share the same process for multiple instances. Could it be that the target server here doesn't have enough capacity to handle 10 concurrent requests?

[alexng-canuck]

@varmax2511 should be able to provide more details.

[varmax2511]

Hi -- The above script is making 10 GET requests one per region for getting VCN resources and the provider fails with the above error when establishing a connection.
I tried making the same 10 requests concurrently via client other than Terraform and didn't run into any TLS handshake issue. This concludes that the target server can handle 10 concurrent requests.

[simon-said]

We also receiving this error for 1 or 2 weeks. And its really worse since few days.

....
Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...

• terraform.io/builtin/terraform is built in to Terraform
• Reusing previous version of hashicorp/null from the dependency lock file
• Reusing previous version of hashicorp/archive from the dependency lock file
• Reusing previous version of hashicorp/random from the dependency lock file
• Reusing previous version of hashicorp/template from the dependency lock file
• Reusing previous version of hashicorp/aws from the dependency lock file
• Installing hashicorp/null v3.1.0...
• Installed hashicorp/null v3.1.0 (signed by HashiCorp)
• Installing hashicorp/archive v2.1.0...
• Installed hashicorp/archive v2.1.0 (signed by HashiCorp)
• Installing hashicorp/random v3.1.0...
• Installed hashicorp/random v3.1.0 (signed by HashiCorp)
• Installing hashicorp/template v2.2.0...
• Installed hashicorp/template v2.2.0 (signed by HashiCorp)
• Installing hashicorp/aws v3.34.0...

**Error: Failed to install provider

Error while installing hashicorp/aws v3.34.0: Get
"https://releases.hashicorp.com/terraform-provider-aws/3.34.0/terraform-provider-aws_3.34.0_windows_amd64.zip":
net/http: TLS handshake timeout**

$ terraform -version
Terraform v0.14.8

• provider registry.terraform.io/hashicorp/archive v2.1.0
• provider registry.terraform.io/hashicorp/aws v3.34.0
• provider registry.terraform.io/hashicorp/null v3.1.0
• provider registry.terraform.io/hashicorp/random v3.1.0
• provider registry.terraform.io/hashicorp/template v2.2.0

Your version of Terraform is out of date! The latest version
is 0.15.0. You can update by downloading from https://www.terraform.io/downloads.html