Support capacity resource groups in aws_resourcegroups_group

[ajlake]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Capacity Reservation groups as described below are not currently supported by aws_resourcegroups_group.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/create-cr-group.html

Support was recently added to Launch Templates for receiving the ARN of a reservation group (via capacity_reservation_resource_group_arn) in #24283, but producing a usable resource group in the first place does not appear to be possible within terraform right now.

New or Affected Resource(s)

• aws_resourcegroups_group

Potential Terraform Configuration

I think we can more or less mirror boto: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/resource-groups.html

resource "aws_resourcegroups_group" "test" {
  name = "test-group"

  configuration {
    type = "AWS::EC2::CapacityReservationPool"
  }

  configuration {
    type = "AWS::ResourceGroups::Generic"

    parameters {
      name   = "allowed-resource-types"
      values = ["AWS::EC2::CapacityReservation"]
    }
  }
}

The second problem to solve is how to handle the resource groupings. In the AWS docs example:

aws resource-groups group-resources --group MyCRGroup --resource-arns arn:aws:ec2:sa-east-1:123456789012:capacity-reservation/cr-1234567890abcdef1 arn:aws:ec2:sa-east-1:123456789012:capacity-reservation/cr-54321abcdef567890

One option is to take in a list of ARNs:

resource "aws_resourcegroups_group" "test" {
  name = "test-group"

  ...
  
  grouped_resources = [
    "arn:aws:ec2:sa-east-1:123456789012:capacity-reservation/cr-1234567890abcdef1",
    "arn:aws:ec2:sa-east-1:123456789012:capacity-reservation/cr-54321abcdef567890",
  ]
}

References

[justinretzolk]

Hey @ajlake 👋 Thank you for taking the time to raise this! I think it's worthwhile to leave this open as a feature request to add the ability to specify a configuration for the aws_resourcegroups_group resource. That said, I was able to figure out a way to create a group with the capacity reservations as is. This involves using the query parameter of the aws_resourcegroups_group resource as described in the AWS document Creating query-based groups in AWS Resource Groups. Another helpful document when I was looking into this was the ResourceQuery API reference. Can you take a look over the below example configuration and let me know if it satisfies your needs?

# Create an example capacity reservation
resource "aws_ec2_capacity_reservation" "test" {
  instance_type     = "t2.micro"
  instance_platform = "Linux/UNIX"
  availability_zone = "us-east-1a"
  instance_count    = 1

  tags = {
    "Environment" = "dev"
  }
}

# Create a second example capacity reservation, just for good measure
resource "aws_ec2_capacity_reservation" "test1" {
  instance_type     = "t2.micro"
  instance_platform = "Linux/UNIX"
  availability_zone = "us-east-1a"
  instance_count    = 1

  tags = {
    "Environment" = "dev"
  }
}

# Create the resource group using a tag-based query
resource "aws_resourcegroups_group" "test" {
  name = "jretzolk-test-group"

  resource_query {
    query = <<JSON
{
	"ResourceTypeFilters": [
		"AWS::EC2::CapacityReservation"
	],
	"TagFilters": [
		{
			"Key": "Environment",
			"Values": ["dev"]
		}
	]
}
JSON
  }
}

After creating this configuration, I ran the following AWS CLI command to verify (output from the command below)

$ aws resource-groups list-group-resources --group jretzolk-test-group

{
    "ResourceIdentifiers": [
        {
            "ResourceArn": "arn:aws:ec2:us-east-1:<redacted>:capacity-reservation/cr-06fd838d2bebfb830",
            "ResourceType": "AWS::EC2::CapacityReservation"
        },
        {
            "ResourceArn": "arn:aws:ec2:us-east-1:<redacted>:capacity-reservation/cr-00dcc3bad65fd6d91",
            "ResourceType": "AWS::EC2::CapacityReservation"
        }
    ],
    "Resources": [
        {
            "Identifier": {
                "ResourceArn": "arn:aws:ec2:us-east-1:<redacted>:capacity-reservation/cr-06fd838d2bebfb830",
                "ResourceType": "AWS::EC2::CapacityReservation"
            }
        },
        {
            "Identifier": {
                "ResourceArn": "arn:aws:ec2:us-east-1:<redacted>:capacity-reservation/cr-00dcc3bad65fd6d91",
                "ResourceType": "AWS::EC2::CapacityReservation"
            }
        }
    ]
}

[martincastrocm]

@justinretzolk I've done that too and although the capacity reservation gets grouped on the Reservation Group (as you are effectively showing) it fails when using that Reservation Group for launching EC2 instances: throwing a malformed reservation group error. (Unfortunately I do not have the error log as I workarounded it a while ago)
That's why the correct way of doing this is following the documentation @ajlake provided: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/create-cr-group.html that difers in the tag mechanism AWS uses for grouping resources. The support for the fr asked is needed :)

[ajlake]

@martincastrocm is correct. I tried the query approach first which did not work. I wrote a simple Go CLI, wrapped it in a bash script, and wrote a module with a null_resource to work around this for now.

For anyone else working around this, this is the essence of the workaround:
https://gist.github.com/ajlake/813d39d7b23352f567ff82166b417718

The result is a module that can be used like this:

module "example_reservation" {
  source = "../capacity_reservation"

  region           = "us-east-1"
  reservation_name = "example-name"

  capacity_allocations = { for z in var.zones : z =>
    {
      instance_type     = var.instance_type
      instance_count    = 2,
      instance_platform = "Linux/UNIX",
    }
  }
  instance_match_criteria = "targeted"
}

# Consumption side
resource "aws_launch_template" "template" {
...
  capacity_reservation_specification {
      capacity_reservation_target {
        capacity_reservation_resource_group_arn = module.example_reservation.resource_group_arn
      }
  }
...
}

[martincastrocm]

Hi @justinretzolk it's been a while, is there any ETA to support this? Thanks!

[justinretzolk]

Hey @martincastrocm 👋 Thanks for checking in on this! Unfortunately, I'm not able to provide an ETA on when this will be looked into due to the potential of shifting priorities (we prioritize work by count of ":+1:" reactions, as well as a few other things). We have more information on how we prioritize over on our prioritization guide, if you're interested.

[airbnb-gps]

#26934 added the configuration block and #27985 fixed the schema to enable it to work with Capacity Reservations using an empty value.

So the remaining missing piece here is the GroupResources API integration to allow associating a list of ARNs with a group. The association must be via ARNs: tag-based queries won't work with resource groups using service configurations (read: the configuration block).

[ewbankkit]

@airbnb-gps The aws_resourcegroups_resource resource added via #31430 and released in Terraform AWS Provider v5.1.0 provides the GroupResources/UngroupResources functionality.

[IAXES]

@justinretzolk I followed the example you provided above, and was also able to query the resource group via the AWS CLI v2 and generate similar results to your example. However, I noticed some weird errors with Terragrunt/Terraform. For starters, when I attempted to use the ARN of the resource group as an input to a launch template, i.e.:

   # Launch EC2 instances into specific, deliberate capacity reservation.
   capacity_reservation_specification {
     capacity_reservation_target {
       capacity_reservation_resource_group_arn = aws_resourcegroups_group.resource_group_for_ec2_capacity.arn
     }
   }
 }

... TF would complain that that:

│ Error: updating Auto Scaling Group (blah_blah):
ValidationError: You must use a valid fully-formed launch template. The given
resource group arn arn:aws:resource-groups:us-west-2:0123456789:group/blah_blah
does not exist.

This seemed weird. The Terragrunt stack I'm using is quite mature, so I'm skeptical that it's a bug in profile and/or provider code. So, I try to validate via the AWS CLI:

$> aws --profile my_profile
--region us-west-2 ec2 run-instances --launch-template
LaunchTemplateName=blah_blah,Version='55' --dry-run

An error occurred (InvalidParameterValue) when calling the RunInstances
operation: The given resource group arn
arn:aws:resource-groups:us-west-2:0123456789:group/blah_blah
does not exist.

So, I destroyed the TF-generated resource group, re-created it manually via the AWS CLI (using a guide from the AWS docs), imported it: no issues at all. So, in the end, I went with a similar approach to @ajlake (thanks for the gist!), using a null_resource to call the AWS CLI to handle the create-group and group-resources operations. Worked fine.

resource "null_resource" "capacity_reservation_group" {
  triggers = {
    cr_arn              = aws_ec2_capacity_reservation.capacity_reservation.arn
    profile             = var.aws_profile
    account_id          = var.aws_account_id
    region              = var.aws_region
    resource_group_name = local.resource_group_name
    resource_group_arn  = local.resource_group_arn
  }

  # Run shell command to create the resource group that "consumes" the EC2
  # capacity reservation.
  provisioner "local-exec" {
    command = <<EOF
aws \
    --profile "${self.triggers.profile}" \
    resource-groups \
    create-group \
    --name "${self.triggers.resource_group_name}" \
    --configuration \
      '{"Type":"AWS::EC2::CapacityReservationPool"}' \
      '{"Type":"AWS::ResourceGroups::Generic", "Parameters": [{"Name": "allowed-resource-types", "Values": ["AWS::EC2::CapacityReservation"]}]}' \
   && \
aws \
  --profile "${self.triggers.profile}" \
  resource-groups \
  group-resources \
  --group "${self.triggers.resource_group_name}" \
  --resource-arns "${self.triggers.cr_arn}" \
  && \
  echo "Done."
EOF
  }

  # Run shell command to destroy the resource group that "consumes" the EC2
  # capacity reservation.
  provisioner "local-exec" {
    when    = destroy
    command = <<EOF
aws \
    --profile "${self.triggers.profile}" \
    resource-groups \
    delete-group \
    --group "${self.triggers.resource_group_name}"
EOF
  }
}

So, it seems that the configuration block also needs to be supported before we can use a 100% TF-based approach to creating and populating resource groups (at least for this use case, likely additional/more cases as well).

In the meantime, I've left some code snippets and error messages for future readers (so search engines index the keywords; took me longer than I'd care to admit to find this issue to help set me on the right path).

Cheers!

[ederst]

@justinretzolk I think this issue is obsolete, at least I was able to do it with the Pulumi AWS provider (which uses the "bridged" TF provider) following the CloudFormation example:

cr = aws.ec2.CapacityReservation(...)

cr_group = aws.resourcegroups.Group(
    resource_name,
    args=aws.resourcegroups.GroupArgs(
        configurations=[
            aws.resourcegroups.GroupConfigurationArgs(
                type='AWS::EC2::CapacityReservationPool',
                parameters=[],
            ),
            aws.resourcegroups.GroupConfigurationArgs(
                type='AWS::ResourceGroups::Generic',
                parameters=[
                    aws.resourcegroups.GroupConfigurationParameterArgs(
                        name='allowed-resource-types',
                        values=['AWS::EC2::CapacityReservation'],
                    )
                ],
            ),
        ],
    ),
)

aws.resourcegroups.Resource(
    resource_name,
    args=aws.resourcegroups.ResourceArgs(
        resource_arn=cr.arn,
        group_arn=cr_group.arn,
    )
)

The following TF resources can be used - as this commenter also pointed out:

• https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/resourcegroups_group
• https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/resourcegroups_resource

What could be better: Documentation on how to use it in this context, i guess.