SARIF output support #573

arothian · 2021-10-11T21:57:56Z

Adds support for output results in SARIF format #568

Pass rule definitions to result formatters
Add names to rule definitions/violations
Utility method to generate violations from rule classes

arothian · 2021-10-11T22:05:07Z

lib/cfn-nag/cfn_nag.rb

@@ -152,17 +153,12 @@ def audit_result(violations)
    }
  end

-  def fatal_violation(message)


This was moved to a class method in Violation

arothian · 2021-10-11T22:06:04Z

lib/cfn-nag/base_rule.rb

+      violation(logical_resource_ids)
+    end
+
+    def violation(logical_resource_ids, line_numbers = nil)


Allow rule implementation classes to generate violation objects without needing to run audit. This is useful for unit testing and other locations

arothian · 2021-10-11T22:18:20Z

Open Questions

Should the rules list contain all rules in the scan or only those with results?
Handling of file paths in results

NickLiffen · 2021-10-12T10:43:01Z

Open Questions

Should the rules list contain all rules in the scan or only those with results?

Take a look here. There actually isn't a right or wrong answer here. The main thing is people can go and see more information about the rules ran. At a minimum, it should have the rules where is a result. In a perfect world, all rules would be great, but it can just be with results.

Handling of file paths in results

do you mean how to structure to location format so it shows the line path within the GUI?

NickLiffen · 2021-10-12T16:02:27Z

Nice work @arothian 💯

This is actually super close to being done 👍 I just tested it and the only problem I am seeing is:

"region": {
    "startLine": -1
 }

This data should be structured like this:

"region": {
    "startLine": 54,
 }

I see it is working in some situations, for example:

              "physicalLocation": {
                "artifactLocation": {
                  "uri": "../GCSMTTR/template.yml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 526
                }
              },

However, for quite a few others I am seeing:

                "region": {
                  "startLine": -1
                }

The problem here is the -1. Are you able to replicate this? If not I am happy to send you the file that I am working with. I tried to upload it to GitHub Code Scanning but it failed as the file wasn't valid.

NickLiffen · 2021-10-12T16:17:52Z

It looks like it may be rule specific 🤔

arothian · 2021-10-12T16:30:25Z

@NickLiffen Are you using AWS::Serverless::Function in your template? I think the line number is missing in that case as the Lambda function is technically created as part of the transform.

I'll see if we can get a line number that matches back to the pre-transformed resource; otherwise, I can likely drop off the line number entirely for those results.

NickLiffen · 2021-10-12T16:38:05Z

@arothian you are indeed correct, this is using SAM, so they are AWS::Serverless::Function set.

It would be amazing if you could grab the line number before pre-transformed resources. Worst comes to worst (not saying we should do this) we see a lot of companies putting line 1 if they don't know where it is coming from. Most customers find it useful (and they prefer the tool if the actual line numbering works (as that's the value of SARIF and the code scanning GUI). However, if it's too complex architectural (which I understand sometimes these solutions can be), then maybe just default to the line 1. but only for lines we can't really find.

arothian · 2021-10-12T22:06:07Z

Alright, I've added some protection to this so that it will default to line 1 if for some reason the finding doesn't have a valid line number detected. That should help protect against this scenario.

I'll open a separate PR around the issue with -1 line numbers for SAM resources, as it's not directly related to SARIF support here.

NickLiffen · 2021-10-13T13:51:13Z

@arothian this is good to ship 💯 (IMO)

thegonch · 2021-10-15T18:25:03Z

lib/cfn-nag/result_view/sarif_results.rb

+    {
+      name: 'cfn_nag',
+      informationUri: 'https://github.com/stelligent/cfn_nag',
+      semanticVersion: Gem.loaded_specs['cfn-nag'].version.to_s,


Looks like we need to do checks for when this is nil or using .& format because rake:test-all is coming back with failures in these cases:

Failures: 1) SarifResults#driver with four rules should contain name, informationUri, semanticVersion and rules attributes Failure/Error: semanticVersion: Gem.loaded_specs['cfn-nag'].version.to_s, NoMethodError: undefined method `version' for nil:NilClass # ./lib/cfn-nag/result_view/sarif_results.rb:38:in `driver' # ./spec/result_view/sarif_results_spec.rb:36:in `block (4 levels) in <top (required)>' 2) SarifResults#driver with four rules should contain a driver with four rules Failure/Error: semanticVersion: Gem.loaded_specs['cfn-nag'].version.to_s, NoMethodError: undefined method `version' for nil:NilClass # ./lib/cfn-nag/result_view/sarif_results.rb:38:in `driver' # ./spec/result_view/sarif_results_spec.rb:44:in `block (4 levels) in <top (required)>' 3) SarifResults#driver with four rules should contain a rule with id, name, fullDescription Failure/Error: semanticVersion: Gem.loaded_specs['cfn-nag'].version.to_s, NoMethodError: undefined method `version' for nil:NilClass # ./lib/cfn-nag/result_view/sarif_results.rb:38:in `driver' # ./spec/result_view/sarif_results_spec.rb:50:in `block (4 levels) in <top (required)>' Finished in 12.88 seconds (files took 11.49 seconds to load) 871 examples, 3 failures Failed examples: rspec ./spec/result_view/sarif_results_spec.rb:35 # SarifResults#driver with four rules should contain name, informationUri, semanticVersion and rules attributes rspec ./spec/result_view/sarif_results_spec.rb:43 # SarifResults#driver with four rules should contain a driver with four rules rspec ./spec/result_view/sarif_results_spec.rb:49 # SarifResults#driver with four rules should contain a rule with id, name, fullDescription

@thegonch Addressed by setting up a version.rb file and updating several code references to use that instead.

- Pass rule definitions to result formatters - Add names to rule definitions/violations

thegonch

Good fixes, however, during local testing, the end to end test fails on the version_spec test as it expects the hardcoded version 0.0.01 to be in use.

begin end-to-end tests...
Run options: include {:end_to_end=>true}
..F....1
.F......F..

Failures:

  1) cfn_nag --version when ensuring the local gem is installed equals 0.0.01
     Failure/Error:
       expect { system %( cfn_nag --version ) }
         .to output(a_string_matching('0.0.01'))
         .to_stdout_from_any_process

       expected block to output a string matching "0.0.01" to stdout, but output "0.0.0\n"
       Diff:
       @@ -1,2 +1,2 @@
       -(a string matching "0.0.01")
       +0.0.0

     # ./spec/end_to_end/cfn_nag/version_spec.rb:6:in `block (3 levels) in <top (required)>'

  2) cfn_nag_rules --version when ensuring the local gem is installed equals 0.0.01
     Failure/Error:
       expect { system %( cfn_nag_rules --version ) }
         .to output(a_string_matching('0.0.01'))
         .to_stdout_from_any_process

       expected block to output a string matching "0.0.01" to stdout, but output "0.0.0\n"
       Diff:
       @@ -1,2 +1,2 @@
       -(a string matching "0.0.01")
       +0.0.0

     # ./spec/end_to_end/cfn_nag_rules/version_spec.rb:6:in `block (3 levels) in <top (required)>'

  3) cfn_nag_scan --version when ensuring the local gem is installed equals 0.0.01
     Failure/Error:
       expect { system %( cfn_nag_scan --version ) }
         .to output(a_string_matching('0.0.01'))
         .to_stdout_from_any_process

       expected block to output a string matching "0.0.01" to stdout, but output "0.0.0\n"
       Diff:
       @@ -1,2 +1,2 @@
       -(a string matching "0.0.01")
       +0.0.0

     # ./spec/end_to_end/cfn_nag_scan/version_spec.rb:6:in `block (3 levels) in <top (required)>'

Finished in 19.08 seconds (files took 7.03 seconds to load)
18 examples, 3 failures

Failed examples:

rspec ./spec/end_to_end/cfn_nag/version_spec.rb:5 # cfn_nag --version when ensuring the local gem is installed equals 0.0.01
rspec ./spec/end_to_end/cfn_nag_rules/version_spec.rb:5 # cfn_nag_rules --version when ensuring the local gem is installed equals 0.0.01
rspec ./spec/end_to_end/cfn_nag_scan/version_spec.rb:5 # cfn_nag_scan --version when ensuring the local gem is installed equals 0.0.01

NickLiffen · 2021-10-22T18:13:18Z

👋 @thegonch / @arothian just wanted to see where we were with this 👍

I have a few developers looking at this PR interesting in consuming the SARIF output with actions. Interested to see if this is something that could get driven home 🚀

Update version spec to reflect the version.rb values

arothian requested a review from thegonch October 11, 2021 22:04

arothian commented Oct 11, 2021

View reviewed changes

thegonch requested changes Oct 15, 2021

View reviewed changes

arothian added 5 commits October 15, 2021 17:55

SARIF output support

dfe507e

- Pass rule definitions to result formatters - Add names to rule definitions/violations

Resolve todo

6c9d596

Update location details to use relative paths to srcroot

d8cb64f

Provide a default line number

d6193cb

Migrate to a version.rb file for tracking version internally

e1cb0fc

arothian force-pushed the sarif_support branch from 447871a to e1cb0fc Compare October 15, 2021 21:57

thegonch requested changes Oct 18, 2021

View reviewed changes

Remove ENV version override

34462fe

Update version spec to reflect the version.rb values

thegonch approved these changes Oct 25, 2021

View reviewed changes

arothian merged commit 67ae380 into stelligent:master Oct 25, 2021

arothian deleted the sarif_support branch October 25, 2021 21:34

NickLiffen mentioned this pull request Jun 22, 2022

Supporting SARIF output kowainik/stan#467

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SARIF output support #573

SARIF output support #573

arothian commented Oct 11, 2021

arothian Oct 11, 2021

arothian Oct 11, 2021

arothian commented Oct 11, 2021

NickLiffen commented Oct 12, 2021

NickLiffen commented Oct 12, 2021

NickLiffen commented Oct 12, 2021

arothian commented Oct 12, 2021

NickLiffen commented Oct 12, 2021

arothian commented Oct 12, 2021

NickLiffen commented Oct 13, 2021

thegonch Oct 15, 2021

arothian Oct 15, 2021

thegonch left a comment

NickLiffen commented Oct 22, 2021

SARIF output support #573

SARIF output support #573

Conversation

arothian commented Oct 11, 2021

arothian Oct 11, 2021

Choose a reason for hiding this comment

arothian Oct 11, 2021

Choose a reason for hiding this comment

arothian commented Oct 11, 2021

NickLiffen commented Oct 12, 2021

NickLiffen commented Oct 12, 2021

NickLiffen commented Oct 12, 2021

arothian commented Oct 12, 2021

NickLiffen commented Oct 12, 2021

arothian commented Oct 12, 2021

NickLiffen commented Oct 13, 2021

thegonch Oct 15, 2021

Choose a reason for hiding this comment

arothian Oct 15, 2021

Choose a reason for hiding this comment

thegonch left a comment

Choose a reason for hiding this comment

NickLiffen commented Oct 22, 2021