Error reported with wrong resource name

[merckm-cr]

Description :
When running a test against a resource with multiple occurrences in a terraform file the failing resource will be identified with the name of the first resource of the type, although the failure occurs in a resource with a different name.
In my case I would like to do some checks on Network Load Balancers with several Listeners configured. The test is to only allow TCP or TLS listeners. The example Terraform has 3 listeners. 1. TCP listener (which should not fail), 2. UDP listener which should fail and 3. TCP_UDP listener which should fail. As I understand it will only show the first failure which in this case is the UDP listener. The error identifies the correct reason for failure but shows the name of the first Listener as the offending resource which is not correct.

To Reproduce

1. Here the terraform code with all supporting resources

provider "aws" {
  region  = "eu-west-1"
  profile = "cr-mmerck"
}

# ----------------------------------------------------------------
# Data sources
# ----------------------------------------------------------------

# ----------------------------------------------------------------
# Helper Resources needed to provision some of the test Resources
# ----------------------------------------------------------------
resource "aws_vpc" "test-vpc" {
  cidr_block       = "10.0.0.0/16"

  tags = {
    Name = "TestVPC Compliant checks"
  }
}

resource "aws_internet_gateway" "test-igw" {
  vpc_id = "${aws_vpc.test-vpc.id}"

  tags = {
    Name = "Test VPC IGW"
  }
}

resource "aws_subnet" "test-subnet-pubA" {
  vpc_id     = "${aws_vpc.test-vpc.id}"
  cidr_block = "10.0.0.0/24"

  tags = {
    Name = "Public Subnet A Compliant checks"
  }
}

resource "aws_subnet" "test-subnet-pubB" {
  vpc_id     = "${aws_vpc.test-vpc.id}"
  cidr_block = "10.0.1.0/24"

  tags = {
    Name = "Public Subnet B Subnet Compliant checks"
  }
}

resource "aws_subnet" "test-subnet-privA" {
  vpc_id     = "${aws_vpc.test-vpc.id}"
  cidr_block = "10.0.10.0/24"

  tags = {
    Name = "Private Subnet A Subnet Compliant checks"
  }
}

resource "aws_subnet" "test-subnet-privB" {
  vpc_id     = "${aws_vpc.test-vpc.id}"
  cidr_block = "10.0.11.0/24"

  tags = {
    Name = "Private Subnet B Compliant checks"
  }
}

resource "aws_route_table" "test-pub-rtb" {
  vpc_id = "${aws_vpc.test-vpc.id}"

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = "${aws_internet_gateway.test-igw.id}"
  }

  tags = {
    Name = "Test VPC Public Route Table"
  }
}

resource "aws_route_table_association" "test-pubA-rtb-assoc" {
  subnet_id      = "${aws_subnet.test-subnet-pubA.id}"
  route_table_id = "${aws_route_table.test-pub-rtb.id}"
}

resource "aws_route_table_association" "test-pubB-rtb-assoc" {
  subnet_id      = "${aws_subnet.test-subnet-pubB.id}"
  route_table_id = "${aws_route_table.test-pub-rtb.id}"
}

# ----------------------------------------------------------------
# Compliant
# ----------------------------------------------------------------
# Create Loadbalancer with listener on port 443 and redirect on port 80 to 8080
# [1] Security Group
resource "aws_security_group" "sg-loadBalancer-noncompliant" {
  name        = "sg-loadBalancer-noncompliant"
  description = "Allow HTTP/HTTPS inbound traffic"
  vpc_id      = "${aws_vpc.test-vpc.id}"

  ingress {
    # HTTP (change to whatever ports you need)
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    # HTTPS
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port       = 0
    to_port         = 0
    protocol        = "-1"
    cidr_blocks     = ["0.0.0.0/0"]
  }
}


# Network LoadBalancer
resource "aws_lb" "nlb-noncompliant" {
  name               = "nlb-noncompliant"
  internal           = false
  load_balancer_type = "network"
  security_groups    = ["${aws_security_group.sg-loadBalancer-noncompliant.id}"]
  subnets            = ["${aws_subnet.test-subnet-pubA.id}",
                        "${aws_subnet.test-subnet-pubB.id}"]
}

# [1] Listener on TCP on port other then 443
resource "aws_lb_listener" "tcp-listener-noncompliant" {
  load_balancer_arn = "${aws_lb.nlb-noncompliant.arn}"
  port              = "80"
  protocol          = "TCP"

  default_action {
    type = "fixed-response"

    fixed_response {
      content_type = "text/plain"
      message_body = "HEALTHY"
      status_code  = "200"
    }
  }
}

# [2] Listener on UDP
resource "aws_lb_listener" "udp-listener-noncompliant" {
  load_balancer_arn = "${aws_lb.nlb-noncompliant.arn}"
  port              = "1443"
  protocol          = "UDP"

  default_action {
    type = "fixed-response"

    fixed_response {
      content_type = "text/plain"
      message_body = "HEALTHY"
      status_code  = "200"
    }
  }
}

# [3] Listener on TCP_UDP
resource "aws_lb_listener" "tcpudp-listener-noncompliant" {
  load_balancer_arn = "${aws_lb.nlb-noncompliant.arn}"
  port              = "53"
  protocol          = "TCP_UDP"

  default_action {
    type = "fixed-response"

    fixed_response {
      content_type = "text/plain"
      message_body = "HEALTHY"
      status_code  = "200"
    }
  }
}

2. <Used terraform-compliance parameters>
   terraform-compliance -f features/ -p tf/out.plan

Used python package
4.
Scenario: Only allow TLS and TCP protocols on NLBs
Given I have aws_lb_listener resource configured
When it contain protocol
Then its value must not match the ".(UDP|udp)$" regex
Failure: protocol property in aws_lb_listener.tcp-listener-noncompliant resource matches with .(UDP|udp)$ regex. It is set to TCP_UDP.

5. <Your feature/scenario/steps>

Scenario: Only allow TLS and TCP protocols on NLBs
    Given I have aws_lb_listener resource configured
    When it contain protocol
    Then its value must not match the ".*(UDP|udp)$" regex

Expected behavior :

The error message should be:
Failure: protocol property in aws_lb_listener.tcpudp-listener-noncompliant resource matches with .*(UDP|udp)$ regex. It is set to TCP_UDP.
Tested versions :

• <terraform-compliance version (terraform-compliance -v)> 1.0.51
• <terraform version (terraform -v)> Terraform v0.12.12
• <python runtime version, if running as a python package (python --version)> Python 3.7.4

Additional context
Add any other context about the problem here.

[eerkunt]

Thanks for reporting this issue @merckm-cr 🎉

Yes, this was a bug within one of the internal functions that returns the resource name on the failure of that specific step. The problem is fixed on #175, it will be released soon.

[eerkunt]

Could you please have a try with 1.0.53 version ?

[merckm-cr]

This works now and can be closed.