Feature Request: Show all test failures for tests that cover multiple resources

[GJKrupa]

Feature Request

Feature description :
When running scenarios against multiple resources (e.g. the example provided to test that all tag-able resources have tags), include the ability to show results for all failed resources, not just the first one encountered.

Suggested Solution description ( if you have any ) :
Possibly a command-line flag or an option in the feature/scenario to not stop on first failure

Describe alternatives you've considered :
N/A

Additional context :
We're trying to scan for AWS resources that don't have a standard set of billing tags attached but it's time-consuming to re-run the plan and the features after each individually-fixed resource to find the next failure.

[eerkunt]

Thanks for the feature request @GJKrupa 🎉

terraform-compliance never intended to be a linting tool that will visualise all the problems at once within the plan or state. Also, it will require quite a bit refactoring nearly all the steps that exist.

It looks like this is a common feature request for many people since we already rejected 2 of them that is similar to this.

I will have a deeper look about how we can re-structure the whole code to enable this kind of a feature and will update you in this Issue.

[eerkunt]

Hi @GJKrupa,
Can you please have a try with the new release ? 🎉

[GJKrupa]

Hi. Unfortunately I'm not able to test version 1.1.3 as I'm getting the following error:

ERROR: Failed to convert terraform plan file to JSON format via terraform. Here is the error :
None
Failed to marshal plan to json: error marshaling prior state: unsupported attribute "placement_group_id"

[eerkunt]

Hi @GJKrupa,

This is new, and and it is coming directly from terraform.

Is it possible to run terraform show -json <your plan file> ?

[GJKrupa]

I'm getting the same error from that command. Looks like there's something out of sync with our state file. Will get back to you once I figure out what's wrong (hashicorp/terraform#23297 seems to be related)

[GJKrupa]

I've managed to run the tool against our repo now (there were a few glitches to work out) and it's working like a dream. I'm seeing the full list of resources that are missing specific tags rather than just the first one. Thanks very much.

[eerkunt]

Great news!

I am curious about the glitches.

I found one already that I am working on it. In some systems, we see an additional Failure : empty line below the step.

let me know the others you find please :)

[GJKrupa]

The glitches aren't all that interesting - some missing data items in our repo that prevented the plan from succeeding without targeting and the issue mentioned earlier prevents terraform-compliance from running on a targeted plan.

[eerkunt]

Ah I see, I thought it was glitches specific to terraform-compliance :)

[eerkunt]

Well, then I am closing this issue, if everything is ok for you ?

[GJKrupa]

Sorry, should have been clearer with my language. Yes, please feel free to close the ticket.

[ghost]

This issue's conversation is now locked. If you want to continue this discussion please open a new issue.