aws_route -> Error: error reading Route for Route Table (rtb-0823xxxxxxxxxxxxxx) with destination (156.x.x.x/xx): couldn't find resource

[chadp123]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

terraform cli - 0.12.21
aws provider - 3.37.0

Affected Resource(s)

• aws_route

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

Limited to relevant config and obfuscated cidr.. Our VPC is a /24

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key: https://keybase.io/hashicorp

variable "vpc_name" {
  description = "Name of the VPC"
  type        = string
  default     = "vpc_main"
}

variable "vpg_name" {
  description = "Name of the vpg"
  type        = string
}

variable "vpc_cidr_block" {
  description = "CIDR for the whole VPC"
  type        = string
}

variable "incoming_cidr_blocks" {
  description = "Incoming CIDR blocks for default security group rules."
  type        = list(string)
}

data "aws_availability_zones" "available" {
}

data "aws_region" "current" {
}

data "aws_vpn_gateway" "selected" {
  filter {
    name   = "tag:Name"
    values = [var.vpg_name]
  }
}

locals {
  subnet_mask       = element(split("/", var.vpc_cidr_block), 1)
  subnet_exp        = 24 - local.subnet_mask
  subnet_count      = 3
  public_cidr       = cidrsubnet(var.vpc_cidr_block, local.subnet_exp, local.subnet_count)
  public_subnet_exp = local.subnet_exp + 1

  az_count = length(data.aws_availability_zones.available.names)
}

resource "aws_vpc" "main" {
  cidr_block           = var.vpc_cidr_block
  enable_dns_hostnames = true

  tags = merge(
    var.common_tags,
    {
      "Name" = var.vpc_name
    },
    {
      "Terraform" = "true"
    },
  )
}

resource "aws_route_table" "az_subnets" {
  count  = local.subnet_count
  vpc_id = aws_vpc.main.id

  tags = merge(
    var.common_tags,
    {
      "Terraform" = "true"
      "Name"      = "private routes for ${element(aws_subnet.az_subnets.*.availability_zone, count.index)}"
      "Public"    = "false"
    },
  )
}

resource "aws_route_table_association" "az_subnets" {
  count          = local.subnet_count
  subnet_id      = element(aws_subnet.az_subnets.*.id, count.index)
  route_table_id = element(aws_route_table.az_subnets.*.id, count.index)
}

resource "aws_route" "az_subnets_default" {
  count                  = local.subnet_count
  route_table_id         = element(aws_route_table.az_subnets.*.id, count.index)
  destination_cidr_block = "0.0.0.0/0"
  nat_gateway_id         = element(aws_nat_gateway.nat_gw.*.id, count.index)
}

resource "aws_route" "az_dx" {
  count                  = local.subnet_count
  route_table_id         = element(aws_route_table.az_subnets.*.id, count.index)
  destination_cidr_block = "10.0.0.0/8"
  gateway_id             = data.aws_vpn_gateway.selected.id
  depends_on             = [aws_vpn_gateway_attachment.vpn_attachment]
}

resource "aws_route" "az_mf_bigip" {
  count                  = local.subnet_count
  route_table_id         = element(aws_route_table.az_subnets.*.id, count.index)
  destination_cidr_block = "203.x.x.0/24"
  gateway_id             = data.aws_vpn_gateway.selected.id
  depends_on             = [aws_vpn_gateway_attachment.vpn_attachment]
}

# This was the update to our existing terraform to add additional route
resource "aws_route" "az_dx2" {
  count                  = local.subnet_count
  route_table_id         = element(aws_route_table.az_subnets.*.id, count.index)
  destination_cidr_block = "156.x.0.0/16"
  gateway_id             = data.aws_vpn_gateway.selected.id
  depends_on             = [aws_vpn_gateway_attachment.vpn_attachment]
}

Debug Output

Panic Output

Expected Behavior

Added the az_dx2 resource expecting the creation of the additional route.

Actual Behavior

The route is created as expected but the terraform failed with:

Seems to be last subnet in our list of three - timeout?

Error: error reading Route for Route Table (rtb-0823xxxxxxxxxxx) with destination (156.x.0.0/16): couldn't find resource

Steps to Reproduce

1. Apply terraform without the additional route
2. terraform apply
3. Add the additional route
4. terraform apply

Important Factoids

References

Looks similar although not exact the the following issues:

• terraform apply times out but route is still created and terraform does not track this #10666
• Blocks of type "route" are not expected here #8542
• Error finding route after creating it: Unable to find matching route for Route Table (<rtb_id>) and destination CIDR block (<destionation_cidr>) #12073

[ewbankkit]

@chadp123 Thanks for raising this issue.
Does the error occur while the the az_dx2 route is being created or on subsequent refresh/plan?

[chadp123]

We create plan and then apply that plan as part of a CI process. So it's the apply where route az_dx2 is first being created. The resource is created on the aws side but not reflected in the state. I should note that we applied the terraform to two accounts our nonprod/prod. There are 3 private subnets these routes are applied to based on our count variable.

In nonprod one of the route tables faced this issue -> module.vpc.aws_route.az_csg_dx2[1]
In prod two of the route tables faced the issue -> module.vpc.aws_route.az_csg_dx2[0], module.vpc.aws_route.az_csg_dx2[1]

Generating a new plan would produces a plan that indicates it will create the routes. I imported the resources in order to resolve.

[github-actions]

This functionality has been released in v3.47.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[ghost]

On my side, I still get the error erratically, since the 3.47.0 provider. I use the official VPC module: https://github.com/terraform-aws-modules/terraform-aws-vpc, with a public and private subnet.

So far, rolling back to 3.46.0 appears to solve the issue. EDIT: no, issue still exist with 3.46.0, though it seems less frequent, maybe?

[alewando]

Unfortunately I'm still seeing this issue also. I opened #19985 to track this again, since closed issues won't get any attention.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.