aws_security_group with separate aws_security_group_rule rules for both IPv4 and IPv6 gets confused and can't find the rules, completes with errors, even though the results are correct

[buchs]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

$ tf -version
Terraform v1.0.10
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v3.63.0
+ provider registry.terraform.io/hashicorp/external v2.1.0
+ provider registry.terraform.io/hashicorp/local v2.1.0
+ provider registry.terraform.io/hashicorp/null v3.1.0

This is the apply transcript where some parts I don't want to share, not related to the security group rules, are omitted.

$ tf apply -auto-approve
Terraform used the selected providers to generate the following execution plan. Resource
actions are indicated with the following symbols:
  + create
  ~ update in-place

Terraform will perform the following actions:

  # aws_security_group.tableau_sg will be created
  + resource "aws_security_group" "tableau_sg" {
      + arn                    = (known after apply)
      + description            = "tableau specific sg"
      + egress                 = (known after apply)
      + id                     = (known after apply)
      + ingress                = (known after apply)
      + name                   = "tableau_sg"
      + name_prefix            = (known after apply)
      + owner_id               = (known after apply)
      + revoke_rules_on_delete = false
      + tags                   = {
          + "Name" = "tableau_sg"
        }
      + tags_all               = {
          + "Name" = "tableau_sg"
        }
      + vpc_id                 = "vpc-8e6f54f5"
    }

  # aws_security_group_rule.outbound will be created
  + resource "aws_security_group_rule" "outbound" {
      + cidr_blocks              = [
          + "0.0.0.0/0",
        ]
      + from_port                = 0
      + id                       = (known after apply)
      + ipv6_cidr_blocks         = [
          + "0::0/0",
        ]
      + protocol                 = "-1"
      + security_group_id        = (known after apply)
      + self                     = false
      + source_security_group_id = (known after apply)
      + to_port                  = 0
      + type                     = "egress"
    }

  # aws_security_group_rule.tableau-inbound1 will be created
  + resource "aws_security_group_rule" "tableau-inbound1" {
      + cidr_blocks              = [
          + "0.0.0.0/0",
        ]
      + from_port                = 443
      + id                       = (known after apply)
      + ipv6_cidr_blocks         = [
          + "0::0/0",
        ]
      + protocol                 = "tcp"
      + security_group_id        = (known after apply)
      + self                     = false
      + source_security_group_id = (known after apply)
      + to_port                  = 443
      + type                     = "ingress"
    }

  # aws_security_group_rule.tableau-inbound3 will be created
  + resource "aws_security_group_rule" "tableau-inbound3" {
      + cidr_blocks              = [
          + "0.0.0.0/0",
        ]
      + from_port                = 8850
      + id                       = (known after apply)
      + ipv6_cidr_blocks         = [
          + "0::0/0",
        ]
      + protocol                 = "tcp"
      + security_group_id        = (known after apply)
      + self                     = false
      + source_security_group_id = (known after apply)
      + to_port                  = 8850
      + type                     = "ingress"
    }

Plan: 4 to add, 1 to change, 0 to destroy.
aws_security_group.tableau_sg: Creating...
aws_security_group.tableau_sg: Creation complete after 3s [id=sg-0c04cfe6fbebd39eb]
aws_security_group_rule.tableau-inbound3: Creating...
aws_security_group_rule.tableau-inbound1: Creating...
aws_security_group_rule.outbound: Creating...
aws_security_group_rule.outbound: Still creating... [10s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [10s elapsed]
aws_security_group_rule.tableau-inbound3: Still creating... [10s elapsed]
aws_security_group_rule.outbound: Still creating... [20s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [20s elapsed]
aws_security_group_rule.tableau-inbound3: Still creating... [20s elapsed]
aws_security_group_rule.tableau-inbound3: Still creating... [30s elapsed]
aws_security_group_rule.outbound: Still creating... [30s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [30s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [40s elapsed]
aws_security_group_rule.outbound: Still creating... [40s elapsed]
aws_security_group_rule.tableau-inbound3: Still creating... [40s elapsed]
aws_security_group_rule.outbound: Still creating... [50s elapsed]
aws_security_group_rule.tableau-inbound3: Still creating... [50s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [50s elapsed]
aws_security_group_rule.outbound: Still creating... [1m0s elapsed]
aws_security_group_rule.tableau-inbound3: Still creating... [1m0s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [1m0s elapsed]
aws_security_group_rule.outbound: Still creating... [1m10s elapsed]
aws_security_group_rule.tableau-inbound3: Still creating... [1m10s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [1m10s elapsed]
aws_security_group_rule.tableau-inbound3: Still creating... [1m20s elapsed]
aws_security_group_rule.outbound: Still creating... [1m20s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [1m20s elapsed]
aws_security_group_rule.outbound: Still creating... [1m30s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [1m30s elapsed]
aws_security_group_rule.tableau-inbound3: Still creating... [1m30s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [1m40s elapsed]
aws_security_group_rule.tableau-inbound3: Still creating... [1m40s elapsed]
aws_security_group_rule.outbound: Still creating... [1m40s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [1m50s elapsed]
aws_security_group_rule.tableau-inbound3: Still creating... [1m50s elapsed]
aws_security_group_rule.outbound: Still creating... [1m50s elapsed]
aws_security_group_rule.tableau-inbound3: Still creating... [2m0s elapsed]
aws_security_group_rule.outbound: Still creating... [2m0s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [2m0s elapsed]
aws_security_group_rule.outbound: Still creating... [2m10s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [2m10s elapsed]
aws_security_group_rule.tableau-inbound3: Still creating... [2m10s elapsed]
aws_security_group_rule.tableau-inbound3: Still creating... [2m20s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [2m20s elapsed]
aws_security_group_rule.outbound: Still creating... [2m20s elapsed]
aws_security_group_rule.tableau-inbound3: Still creating... [2m30s elapsed]
aws_security_group_rule.outbound: Still creating... [2m30s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [2m30s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [2m40s elapsed]
aws_security_group_rule.tableau-inbound3: Still creating... [2m40s elapsed]
aws_security_group_rule.outbound: Still creating... [2m40s elapsed]
aws_security_group_rule.tableau-inbound3: Still creating... [2m50s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [2m50s elapsed]
aws_security_group_rule.outbound: Still creating... [2m50s elapsed]
aws_security_group_rule.outbound: Still creating... [3m0s elapsed]
aws_security_group_rule.tableau-inbound3: Still creating... [3m0s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [3m0s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [3m10s elapsed]
aws_security_group_rule.outbound: Still creating... [3m10s elapsed]
aws_security_group_rule.tableau-inbound3: Still creating... [3m10s elapsed]
aws_security_group_rule.outbound: Still creating... [3m20s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [3m20s elapsed]
aws_security_group_rule.tableau-inbound3: Still creating... [3m20s elapsed]
aws_security_group_rule.outbound: Still creating... [3m30s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [3m30s elapsed]
aws_security_group_rule.tableau-inbound3: Still creating... [3m30s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [3m40s elapsed]
aws_security_group_rule.tableau-inbound3: Still creating... [3m40s elapsed]
aws_security_group_rule.outbound: Still creating... [3m40s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [3m50s elapsed]
aws_security_group_rule.outbound: Still creating... [3m50s elapsed]
aws_security_group_rule.tableau-inbound3: Still creating... [3m50s elapsed]
aws_security_group_rule.tableau-inbound3: Still creating... [4m0s elapsed]
aws_security_group_rule.outbound: Still creating... [4m0s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [4m0s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [4m10s elapsed]
aws_security_group_rule.tableau-inbound3: Still creating... [4m10s elapsed]
aws_security_group_rule.outbound: Still creating... [4m10s elapsed]
aws_security_group_rule.tableau-inbound3: Still creating... [4m20s elapsed]
aws_security_group_rule.outbound: Still creating... [4m20s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [4m20s elapsed]
aws_security_group_rule.outbound: Still creating... [4m30s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [4m30s elapsed]
aws_security_group_rule.tableau-inbound3: Still creating... [4m30s elapsed]
aws_security_group_rule.tableau-inbound3: Still creating... [4m40s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [4m40s elapsed]
aws_security_group_rule.outbound: Still creating... [4m40s elapsed]
aws_security_group_rule.outbound: Still creating... [4m50s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [4m50s elapsed]
aws_security_group_rule.tableau-inbound3: Still creating... [4m50s elapsed]
aws_security_group_rule.tableau-inbound3: Still creating... [5m0s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [5m0s elapsed]
aws_security_group_rule.outbound: Still creating... [5m0s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [5m10s elapsed]
aws_security_group_rule.outbound: Still creating... [5m10s elapsed]
aws_security_group_rule.outbound: Still creating... [5m20s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [5m20s elapsed]
aws_security_group_rule.outbound: Still creating... [5m30s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [5m30s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [5m40s elapsed]
aws_security_group_rule.outbound: Still creating... [5m40s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [5m50s elapsed]
aws_security_group_rule.outbound: Still creating... [5m50s elapsed]
aws_security_group_rule.outbound: Still creating... [6m0s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [6m0s elapsed]
aws_security_group_rule.outbound: Still creating... [6m10s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [6m10s elapsed]
aws_security_group_rule.outbound: Still creating... [6m20s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [6m20s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [6m30s elapsed]
aws_security_group_rule.outbound: Still creating... [6m30s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [6m40s elapsed]
aws_security_group_rule.outbound: Still creating... [6m40s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [6m50s elapsed]
aws_security_group_rule.outbound: Still creating... [6m50s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [7m0s elapsed]
aws_security_group_rule.outbound: Still creating... [7m0s elapsed]
aws_security_group_rule.outbound: Still creating... [7m10s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [7m10s elapsed]
aws_security_group_rule.outbound: Still creating... [7m20s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [7m20s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [7m30s elapsed]
aws_security_group_rule.outbound: Still creating... [7m30s elapsed]
aws_security_group_rule.outbound: Still creating... [7m40s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [7m40s elapsed]
aws_security_group_rule.outbound: Still creating... [7m50s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [7m50s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [8m0s elapsed]
aws_security_group_rule.outbound: Still creating... [8m0s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [8m10s elapsed]
aws_security_group_rule.outbound: Still creating... [8m10s elapsed]
aws_security_group_rule.outbound: Still creating... [8m20s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [8m20s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [8m30s elapsed]
aws_security_group_rule.outbound: Still creating... [8m30s elapsed]
aws_security_group_rule.outbound: Still creating... [8m40s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [8m40s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [8m50s elapsed]
aws_security_group_rule.outbound: Still creating... [8m50s elapsed]
aws_security_group_rule.outbound: Still creating... [9m0s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [9m0s elapsed]
aws_security_group_rule.outbound: Still creating... [9m10s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [9m10s elapsed]
aws_security_group_rule.outbound: Still creating... [9m20s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [9m20s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [9m30s elapsed]
aws_security_group_rule.outbound: Still creating... [9m30s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [9m40s elapsed]
aws_security_group_rule.outbound: Still creating... [9m40s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [9m50s elapsed]
aws_security_group_rule.outbound: Still creating... [9m50s elapsed]
aws_security_group_rule.outbound: Still creating... [10m0s elapsed]
aws_security_group_rule.tableau-inbound1: Still creating... [10m0s elapsed]
aws_security_group_rule.outbound: Still creating... [10m10s elapsed]
aws_security_group_rule.outbound: Still creating... [10m20s elapsed]
aws_security_group_rule.outbound: Still creating... [10m30s elapsed]
aws_security_group_rule.outbound: Still creating... [10m40s elapsed]
aws_security_group_rule.outbound: Still creating... [10m50s elapsed]
aws_security_group_rule.outbound: Still creating... [11m0s elapsed]
aws_security_group_rule.outbound: Still creating... [11m10s elapsed]
aws_security_group_rule.outbound: Still creating... [11m20s elapsed]
aws_security_group_rule.outbound: Still creating... [11m30s elapsed]
aws_security_group_rule.outbound: Still creating... [11m40s elapsed]
aws_security_group_rule.outbound: Still creating... [11m50s elapsed]
aws_security_group_rule.outbound: Still creating... [12m0s elapsed]
aws_security_group_rule.outbound: Still creating... [12m10s elapsed]
aws_security_group_rule.outbound: Still creating... [12m20s elapsed]
aws_security_group_rule.outbound: Still creating... [12m30s elapsed]
aws_security_group_rule.outbound: Still creating... [12m40s elapsed]
aws_security_group_rule.outbound: Still creating... [12m50s elapsed]
aws_security_group_rule.outbound: Still creating... [13m0s elapsed]
aws_security_group_rule.outbound: Still creating... [13m10s elapsed]
aws_security_group_rule.outbound: Still creating... [13m20s elapsed]
aws_security_group_rule.outbound: Still creating... [13m30s elapsed]
aws_security_group_rule.outbound: Still creating... [13m40s elapsed]
aws_security_group_rule.outbound: Still creating... [13m50s elapsed]
aws_security_group_rule.outbound: Still creating... [14m0s elapsed]
aws_security_group_rule.outbound: Still creating... [14m10s elapsed]
aws_security_group_rule.outbound: Still creating... [14m20s elapsed]
aws_security_group_rule.outbound: Still creating... [14m30s elapsed]
aws_security_group_rule.outbound: Still creating... [14m40s elapsed]
aws_security_group_rule.outbound: Still creating... [14m50s elapsed]
aws_security_group_rule.outbound: Still creating... [15m0s elapsed]
╷
│ Error: Error finding matching egress Security Group Rule (sgrule-2183716400) for Group sg-0c04cfe6fbebd39eb
│
│   with aws_security_group_rule.outbound,
│   on main.tf line 174, in resource "aws_security_group_rule" "outbound":
│  174: resource "aws_security_group_rule" "outbound" {
│
╵
╷
│ Error: Error finding matching ingress Security Group Rule (sgrule-2738003981) for Group sg-0c04cfe6fbebd39eb
│
│   with aws_security_group_rule.tableau-inbound1,
│   on main.tf line 184, in resource "aws_security_group_rule" "tableau-inbound1":
│  184: resource "aws_security_group_rule" "tableau-inbound1" {
│
╵
╷
│ Error: Error finding matching ingress Security Group Rule (sgrule-3936320274) for Group sg-0c04cfe6fbebd39eb
│
│   with aws_security_group_rule.tableau-inbound3,
│   on main.tf line 194, in resource "aws_security_group_rule" "tableau-inbound3":
│  194: resource "aws_security_group_rule" "tableau-inbound3" {
│
╵
Releasing state lock. This may take a few moments...

Funny thing is that the Security Group turns out to be as designed. So apparently the errors are decoys.

separate-rules.tf.txt

Related issue: #21573

[buchs]

Oh, by the way, when I had separate rules for IPv4 and IPv6, TF also became confused, thinking they were duplicate rules. Again, it seemed to accomplish the right thing, but with errors/loooooong timeouts reported by TF.

[justinretzolk]

Hey @buchs 👋 Thank you for taking the time to file this issue! So that we have all of the necessary information in order to investigate this, can you add the Terraform configuration to the issue description as well?

[buchs]

The terraform configuration is in the attached file - separate-rules.tf.txt. See note about it being edited to remove some non-relevant information that I wanted to keep private.

[justinretzolk]

Hey @buchs - sorry for the confusion there - I'd failed to scroll down all the way 🙃.

[Toilal]

I think this issue is related to ipv6. ipv6 can be compressed (https://iplocation.io/ipv6-compress) in either terraform configuration or aws responses, making hashes or comparison fail in some cases. 0::0/0 and ::/0 are the same from network perspective, but does terraform consider this ?

[Toilal]

I just workaround the issue by applying this snippet to ipv6 retrieved with http provider from https://betteruptime.com/ips.txt.

Note that those adresses was causing the issue because there's a 0 after : separator.

2600:3c00::f03c:92ff:feaa:0335
2600:3c00::f03c:92ff:feaa:0354

should be

2600:3c00::f03c:92ff:feaa:335
2600:3c00::f03c:92ff:feaa:354

use replace function to compress ipv6 adresses

// replace(ip, "/:0+/", ":")

// My use case is to grab ips from this online list
data "http" "betteruptime_ips" {
  url = "https://betteruptime.com/ips.txt"
}

betteruptime_cidr_blocks = compact([for ip in split("\n", data.http.betteruptime_ips.body) : (can(regex("::", ip)) ? "" : "${ip}/32")])
betteruptime_ipv6_cidr_blocks = compact([for ip in split("\n", data.http.betteruptime_ips.body) : (can(regex("::", ip)) ? "${replace(ip, "/:0+/", ":")}/128" : "")])

[github-actions]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.