Support Cloudflare r2 for storing Terraform state

[maggie44]

Terraform Version

1.5.6

Use Cases

Cloudflare r2 storage is S3 API compatible, but the s3 backend for storing Terraform state seems to be only compatible with Amazon AWS.

https://developers.cloudflare.com/r2/api/s3/api/

Attempted Solutions

Tried using the S3 but it's not compatible. Thought changing the endpoint would do it, but the region field is compulsory. When adding in a random region it fails to authenticate with the Cloudflare endpoint.

Proposal

No response

References

No response

[maggie44]

This seems to work:

  backend "s3" {
    bucket         = "bucket-name"
    key            = "tfstate/terraform.tfstate"
    endpoint       = "xxx"
    access_key     = "xxx"
    secret_key     = "xxx"
    skip_credentials_validation = true
    skip_region_validation = true
    region         = "us-east-1"
  }

Yet if I remove region = "us-east-1" it prompts for a region:

Initializing the backend...
region
  AWS region of the S3 Bucket and DynamoDB Table (if used).

  Enter a value:

I had expected skip_region_validation = true to override that.

[crw]

Thanks for this request!

[ioannidesalex]

Didn't find a working solution for Terraform 1.6+

[bestrocker221]

I am having the same problem, i also added

skip_requesting_account_id  = true
skip_metadata_api_check     = true

But now I get:

│ Error: Failed to save state
│ 
│ Error saving state: failed to upload state: operation error S3: PutObject,
│ https response error StatusCode: 501, RequestID: , HostID: , api error
│ NotImplemented: STREAMING-UNSIGNED-PAYLOAD-TRAILER not implemented

I could not find a workaround

[crw]

Terraform 1.6.x included an upgrade to the AWS SDK v2, (see the release notes), and 1.6.3 will add a feature that should help with third-party, S3-"compatible" object stores. It may be worth re-testing with 1.6.3, although I do not specifically recognize the error specified above.

[bestrocker221]

I can confirm the error mentioned above (STREAMING-UNSIGNED-PAYLOAD-TRAILER) appears with many others s3 compatible providers, such as Oracle Cloud buckets for example.
State file and s3 cant really be used anymore at the moment.. will test out with 1.6.3 as soon as its out

[crw]

State file and s3 cant really be used anymore at the moment

To clarify for any future reader of this issue, AWS S3 can be used as a state file backend now. "S3-compatible" services from non-AWS vendors appear to be having trouble keeping up with compatibility, but hopefully the fixes made to the S3 backend by our AWS provider team to workaround these compatibility issues will help alleviate the issue.

[maggie44]

State file and s3 cant really be used anymore at the moment

To clarify for any future reader of this issue, AWS S3 can be used as a state file backend now. "S3-compatible" services from non-AWS vendors appear to be having trouble keeping up with compatibility, but hopefully the fixes made to the S3 backend by our AWS provider team to workaround these compatibility issues will help alleviate the issue.

Doesn't look like the changes have stopped the issue

│ Error saving state: failed to upload state: operation error S3: PutObject,
│ https response error StatusCode: 501, RequestID: , HostID: , api error
│ NotImplemented: STREAMING-UNSIGNED-PAYLOAD-TRAILER not implemented

Version 1.6.5

[ILLISIS]

I am also keen to see a resolution on this

[elasticdotventures]

+1 for resolution

[missinglink]

I was able to use a Cloudflare R2 bucket as a s3 backend with terraform 1.6.6 today.

In order to solve the NotImplemented: STREAMING-UNSIGNED-PAYLOAD-TRAILER error I needed to add skip_s3_checksum = true:

terraform {
  required_providers {
    cloudflare = {
      source  = "cloudflare/cloudflare"
      version = "~> 4.0"
    }
  }

  backend "s3" {
    bucket = "terraform-state"
    key    = "project_name/terraform.tfstate"
    endpoints = { s3 = "https://xxxxx.r2.cloudflarestorage.com" }
    region = "us-east-1"

    access_key = "xxxx"
    secret_key = "xxxxx"
    skip_credentials_validation = true
    skip_region_validation = true
    skip_requesting_account_id  = true
    skip_metadata_api_check     = true
    skip_s3_checksum = true
  }
}

[missinglink]

reference: #34127

[ILLISIS]

I was able to use a Cloudflare R2 bucket as a s3 backend with terraform 1.6.6 today.

In order to solve the NotImplemented: STREAMING-UNSIGNED-PAYLOAD-TRAILER error I needed to add skip_s3_checksum = true:

terraform {
  required_providers {
    cloudflare = {
      source  = "cloudflare/cloudflare"
      version = "~> 4.0"
    }
  }

  backend "s3" {
    bucket = "terraform-state"
    key    = "project_name/terraform.tfstate"
    endpoints = { s3 = "https://xxxxx.r2.cloudflarestorage.com" }
    region = "us-east-1"

    access_key = "xxxx"
    secret_key = "xxxxx"
    skip_credentials_validation = true
    skip_region_validation = true
    skip_requesting_account_id  = true
    skip_metadata_api_check     = true
    skip_s3_checksum = true
  }
}

Thanks! that has worked for me on v1.6.5. I have been able to init, plan and apply.

[maggie44]

I was able to use a Cloudflare R2 bucket as a s3 backend with terraform 1.6.6 today.

In order to solve the NotImplemented: STREAMING-UNSIGNED-PAYLOAD-TRAILER error I needed to add skip_s3_checksum = true:

terraform {
  required_providers {
    cloudflare = {
      source  = "cloudflare/cloudflare"
      version = "~> 4.0"
    }
  }

  backend "s3" {
    bucket = "terraform-state"
    key    = "project_name/terraform.tfstate"
    endpoints = { s3 = "https://xxxxx.r2.cloudflarestorage.com" }
    region = "us-east-1"

    access_key = "xxxx"
    secret_key = "xxxxx"
    skip_credentials_validation = true
    skip_region_validation = true
    skip_requesting_account_id  = true
    skip_metadata_api_check     = true
    skip_s3_checksum = true
  }
}

Works for me too.

Would have thought skip_region_validation would mean we wouldn’t need the us-east entry though, so still potentially a bug that could confuse people? Or is that doing something different?

[alecaugh]

[terraform version 1.6.6]
Unfortunately this workaround doesn't seem to work when wanting to provide the key via environmental variables i.e.

terraform {
  required_version = "1.6.6"

  required_providers {
    cloudflare = {
      source  = "cloudflare/cloudflare"
      version = "~> 4"
    }
  }
}

data "terraform_remote_state" "state" {
  backend = "s3"

  config = {
    bucket    = "terraform-state"
    key       = "project_name/terraform.tfstate"
    endpoints = { s3 = "https://xxxx.r2.cloudflarestorage.com" }
    region    = "us-east-1"

    access_key                  = "${var.CLOUDFLARE_S3_ACCESS_KEY_ID}"
    secret_key                  = "${var.CLOUDFLARE_S3_SECRET_ACCESS_KEY}"
    skip_credentials_validation = true
    skip_region_validation      = true
    skip_requesting_account_id  = true
    skip_metadata_api_check     = true
    skip_s3_checksum            = true
  }
}

It's returning this as an error during the terraform plan:

 Error: Unable to access object "terraform.tfstate" in S3 bucket "terraform": operation error S3: HeadObject, https response error StatusCode: 400, RequestID: , HostID: , api error BadRequest: Bad Request
│```

Has anyone else encountered this issue and found a solution?

[missinglink]

doesn't seem to work when wanting to provide the key via environmental variables

https://stackoverflow.com/a/59491777

[alecaugh]

@missinglink It's not an issue with using the values from environment variables, it is successfuly authenticating to the S3 API.
There's an issue where the HeadObject API call in the terraform plan doesn't succeed.
Sometimes the above error isn't thrown, as if the remote state file has been fetched but it has no content. All of the resources need to be created in the plan results even though they exist and are present in the state file.

[alecaugh]

Fixed my issue. I misunderstood what the terraform_remote_state did.
What I needed was this solution #13022 (comment)

[crw]

I am under the impression that as of the 1.6.6 release, the original issue scenario now is supported or have work-arounds. Is anyone still experiencing the original issue as described? If not I'll close this issue. Thanks!

[missinglink]

@crw it certainly doesn't work 'out of the box' as there is significant configuration required.

Once this issue is closed where will the knowledge gained in this thread live?

[crw]

Good question. I doubt we would document the nuances of configuration for every 3rd party S3-compatible vendor on the primary S3 backend page. Is there any existing documentation on this on the Cloudflare side? A quick google search turns up this user-editable page: https://developers.cloudflare.com/r2/examples/terraform/ -- my gut feeling is that this would be the best place for this documentation.

[jwdeane]

Following confirmed to work on Terraform v1.7.4

  backend "s3" {
    bucket = "whatever"
    key    = "key/goes/here/terraform.tfstate"

    region                      = "auto"
    skip_credentials_validation = true
    skip_metadata_api_check     = true
    skip_region_validation      = true
    skip_requesting_account_id  = true
    skip_s3_checksum            = true
    use_path_style              = true
    # endpoint                  = AWS_ENDPOINT_URL_S3

    /*
      ENVIRONMENT VARIABLES
      ---------------------
      AWS_ACCESS_KEY_ID     - R2 token
      AWS_SECRET_ACCESS_KEY - R2 secret
      AWS_ENDPOINT_URL_S3   - R2 location: https://ACCOUNT_ID.r2.cloudflarestorage.com
    */
  }

[simonostendorf]

@jwdeane

Following confirmed to work on Terraform v1.7.4

You're missing state lock here, correct?

Is there a way to enable r2 to support state locking?

[crw]

Closing this issue as it appears to be resolved. For future viewers, the suggested path forward is to edit the page mentioned in #33847 (comment), possibly using the information provided in #33847 (comment). From my quick examination, it looks like that page has not yet been updated. Thanks!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.