Breaking change in 1.6.0: s3 backend config rejects hostname for endpoint parameter

[mogul]

Terraform Version

1.6.0

Terraform Configuration Files

terraform {
  required_version = "~> 1.0"
  required_providers {
    cloudfoundry = {
      source  = "cloudfoundry-community/cloudfoundry"
      version = "~>0.51.3"
    }
  }

  backend "s3" {
    # We are using "partial configuration" here. The rest of the backend
    # parameters are provided when you initialize terraform, eg run:
    # 
    #   terraform init \
    #    --backend-config=../shared/config/backend.tfvars \
    #    --backend-config=key=terraform-state-$(basename $(pwd))
    #
    # For more info, see: 
    # https://developer.hashicorp.com/terraform/language/settings/backends/configuration#partial-configuration
    endpoint = "s3-fips.us-gov-west-1.amazonaws.com"
    encrypt = "true"
  }
}

Debug Output

N/A

Expected Behavior

  Initializing the backend...
  
  Successfully configured the backend "s3"! Terraform will automatically
  use this backend unless the backend configuration changes.
  Initializing modules...
  - environments in bootstrap-env
  Downloading git::https://github.com/18f/terraform-cloudgov.git?ref=v0.5.1 for s3-backups...
  - s3-backups in /tmp/terraform-data-dir/modules/s3-backups/s3
  
  Initializing provider plugins...
  - Reusing previous version of integrations/github from the dependency lock file
  - Reusing previous version of hashicorp/local from the dependency lock file
  - Reusing previous version of cloudfoundry-community/cloudfoundry from the dependency lock file
  - Installing integrations/github v5.12.0...
  - Installed integrations/github v5.12.0 (signed by a HashiCorp partner, key ID 38027F80D7FD5FB2)
  - Installing hashicorp/local v2.4.0...
  - Installed hashicorp/local v2.4.0 (signed by HashiCorp)
  - Installing cloudfoundry-community/cloudfoundry v0.51.3...
  - Installed cloudfoundry-community/cloudfoundry v0.51.3 (self-signed, key ID C0E4EB79E9E6A23D)
  
  Partner and community providers are signed by their developers.
  If you'd like to know more about provider signing, you can read about it here:
  https://www.terraform.io/docs/cli/plugins/signing.html
  
  Terraform has been successfully initialized!

Actual Behavior

  Initializing the backend...
  Initializing modules...
  - environments in bootstrap-env
  Downloading git::https://github.com/18f/terraform-cloudgov.git?ref=v0.5.1 for s3-backups...
  - s3-backups in /tmp/terraform-data-dir/modules/s3-backups/s3
  ╷
  │ Warning: Deprecated Parameter
  │ 
  │   on providers.tf line 10, in terraform:
  │   10:   backend "s3" {
  │ 
  │ The parameter "endpoint" is deprecated. Use parameter "endpoints.s3"
  │ instead.
  ╵
  
  ╷
  │ Error: Invalid Value
  │ 
  │   on providers.tf line 10, in terraform:
  │   10:   backend "s3" {
  │ 
  │ The value must be a valid URL containing at least a scheme and hostname.
  │ Had "***"
  ╵
  

Steps to Reproduce

1. terraform init

Additional Context

Previously the endpoint variable in the S3 backend config accepted just a hostname. As of the 1.6.0 release, Terraform requires that value to be a URL. This is how we fixed it in our codebase. It seems like a small thing, but this is a breaking change! You should release a 1.6.x point release that relaxes the value constraint for endpoint to what it was before.

References

No response

[jhoelzel]

using digitalocean it gets worse. once you add https to the endpoint you are greeted with

Error: Retrieving AWS account details: AWS account ID not previously found and failed retrieving via all available methods. See https://www.terraform.io/docs/providers/aws/index.html#skip_requesting_account_id for workaround and implications. Errors: 2 errors occurred:
│ 	* retrieving caller identity from STS: operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: 0da35d8e-5b58-48ca-a43d-1e6e735aedf7, api error InvalidClientTokenId: The security token included in the request is invalid.
│ 	* retrieving account information via iam:ListRoles: operation error IAM: ListRoles, https response error StatusCode: 403, RequestID: e209a1b2-03ce-4ab6-b724-634f3d80aace, api error InvalidClientTokenId: The security token included in the request is invalid.

[MelvinSone]

Same error here i tried it with endpoints = {s3 = "https://myurl.local.com"} getting the same error as @jhoelzel

[mogul]

Same error here i tried it with endpoints = {s3 = "https://myurl.local.com"} getting the same error as @jhoelzel

You're referring to endpoints.s3, which is a new way of specifying configuration in 1.6.0! This issue concerns the existing endpoint variable behavior breaking in that release. Would you mind making a separate issue about problems you're having with endpoints.s3?

[hloeung]

#33983

[MelvinSone]

@

Same error here i tried it with endpoints = {s3 = "https://myurl.local.com"} getting the same error as @jhoelzel

You're referring to endpoints.s3, which is a new way of specifying configuration in 1.6.0! This issue concerns the existing endpoint variable behavior breaking in that release. Would you mind making a separate issue about problems you're having with endpoints.s3?

But the problem is with the endpoint parameter itself is deprecated. Like ur Warning message shows, and that u need to use the new way with endpoints.s3, like i tried because i got the same behavior like u. So why i need to open a new issue if i have the same configs, with the same errors like u?

[samuelarogbonlo]

using digitalocean it gets worse. once you add https to the endpoint you are greeted with

Error: Retrieving AWS account details: AWS account ID not previously found and failed retrieving via all available methods. See https://www.terraform.io/docs/providers/aws/index.html#skip_requesting_account_id for workaround and implications. Errors: 2 errors occurred:
│ 	* retrieving caller identity from STS: operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: 0da35d8e-5b58-48ca-a43d-1e6e735aedf7, api error InvalidClientTokenId: The security token included in the request is invalid.
│ 	* retrieving account information via iam:ListRoles: operation error IAM: ListRoles, https response error StatusCode: 403, RequestID: e209a1b2-03ce-4ab6-b724-634f3d80aace, api error InvalidClientTokenId: The security token included in the request is invalid.

@jhoelzel how were you able to get pass this issue?

[jhoelzel]

@jhoelzel how were you able to get pass this issue?

I have downgraded terraform and am checking out opentofu

[MelvinSone]

using digitalocean it gets worse. once you add https to the endpoint you are greeted with

Error: Retrieving AWS account details: AWS account ID not previously found and failed retrieving via all available methods. See https://www.terraform.io/docs/providers/aws/index.html#skip_requesting_account_id for workaround and implications. Errors: 2 errors occurred:
│ 	* retrieving caller identity from STS: operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: 0da35d8e-5b58-48ca-a43d-1e6e735aedf7, api error InvalidClientTokenId: The security token included in the request is invalid.
│ 	* retrieving account information via iam:ListRoles: operation error IAM: ListRoles, https response error StatusCode: 403, RequestID: e209a1b2-03ce-4ab6-b724-634f3d80aace, api error InvalidClientTokenId: The security token included in the request is invalid.

@jhoelzel how were you able to get pass this issue?

I haven't passed it. i just reverted to Version 1.5.7.

[cameronharriss]

Hi,

I seem to be facing this issue while also using

terraform {
required_version = ">= 1.0.0"

Is this a known issue also?

[mogul]

I seem to be facing this issue while also using

terraform { required_version = ">= 1.0.0"

Is this a known issue also?

By saying >= you're indicating "go ahead and use the most recent 1.x.x version" so you're also hitting this in 1.6.0.

[gdavison]

Thanks for reporting this, @mogul. According to the AWS documentation, the parameters should be full URLs, including the protocol. However, bare hostnames have worked in the past.

Due to internal implementation of the Go url.Parse function, a bare URL is stored in the Path field of the url.URL type. This may cause unexpected behaviour with some services, such as the Instance Metadata Service (IMDS), which overrides the request path.

The IMDS endpoint will still require a full URL and the S3, IAM, DynamoDB, and STS endpoints will allow a bare hostname with a warning.

[gdavison]

@jhoelzel it looks like your error is when accessing the STS endpoint. Could you please open a new issue and include the configuration that you're using and the debug log for the configuration. The debug log configuration is described at https://developer.hashicorp.com/terraform/internals/debugging

[davidmir]

I'm with a similar issue. I'm using DigitalOcean Spaces to store Terraform state, and the backend only works if I set the endpoint parameter (which is deprecated according to the documentation). The backend is ignoring the new way of endpoints.s3 inside the provider? Could be?

[yordis]

@davidmir did you figure out the DO situation?!

[Tchoupinax]

The same situation happened fro Scaleway provider

[lotusnoir]

I got it working replacing

endpoint = "https://s3.fr-par.scw.cloud"

by

    endpoints = { s3 = "https://s3.fr-par.scw.cloud" }
    skip_requesting_account_id = true

then

terraform init -reconfigure

I'm using scaleway s3 backend

[barnabasbusa]

If you also want to write to the s3 bucket (digitalocean), not just read from it, you also have to add

skip_s3_checksum            = true

#34130

Works as of tf version 1.6.3.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.