Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for SARIF #311

Closed
mathroule opened this issue Jun 19, 2023 · 26 comments
Closed

Add support for SARIF #311

mathroule opened this issue Jun 19, 2023 · 26 comments
Labels
enhancement New feature or request

Comments

@mathroule
Copy link

mathroule commented Jun 19, 2023

How to output as SARIF using error message formatter?

Is Go template flexible enough to do it or it's easier to output as JSON and then convert it to SARIF?

Thanks

@rhysd
Copy link
Owner

rhysd commented Jun 19, 2023

I don't know the SARIF format. So I can't answer it. For the syntax of Go template, please refer the following document. Basic control flows (e.g. loops, branches, blocks, ...) are supported.

@rhysd rhysd added the question Further information is requested label Jun 19, 2023
@mathroule
Copy link
Author

SARIF (Static Analysis Results Interchange Format) is an OASIS standard supported by GitHub Code Scanning.

@rhysd
Copy link
Owner

rhysd commented Jun 19, 2023

I don't know the details of the specification so I can't answer this question.

@mathroule
Copy link
Author

mathroule commented Jun 19, 2023

Ok. So I guess there is no plan yet to support SARIF in actionlint?

@rhysd
Copy link
Owner

rhysd commented Jun 19, 2023

Is this a feature request? Since the description of this issue are questions, I thought it just asked questions.

So I guess there is no plan yet to support SARIF in actionlint?

As of now, I have no plan since it may be possible by -format option. If the option can cover SARIF, adding SARIF support separately is unnecessary. Once I understand the option cannot cover SARIF, I would consider how to support it.

@mathroule
Copy link
Author

mathroule commented Jun 19, 2023

Basically, a SARIF is a JSON file with a structure that can look like this for actionlint:

{
    "$schema": "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/master/Schemata/sarif-schema-2.1.0.json",
    "version": "2.1.0",
    "runs": [
        {
            "tool": {
                "driver": {
                    "name": "GitHub Actions lint",
                    "version": "v1.6.25",
                    "informationUri": "https://github.com/rhysd/actionlint",
                    "rules": [
                        {
                            "id": "unexpected-keys",
                            "name": "UnexpectedKeys",
                            "defaultConfiguration": {
                                "level": "error"
                            },
                            "properties": {
                                "description": "Unexpected keys",
                                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md#check-unexpected-keys"
                            },
                            "shortDescription": {
                                "text": "Unexpected keys"
                            },
                            "fullDescription": {
                                "text": "Workflow syntax defines what keys can be defined in which mapping object. When other keys are defined, they are simply ignored and don't affect workflow behavior. It means typo in keys is not detected by GitHub."
                            },
                            "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md#check-unexpected-keys"
                        }
                    ]
                }
            },
            "results": [
                {
                    "ruleId": "unexpected-keys",
                    "ruleIndex": 0,
                    "message": {
                        "text": "unexpected key \"cancel-in\" for \"concurrency\" section. expected one of \"cancel-in-progress\", \"group\""
                    },
                    "locations": [
                        {
                            "physicalLocation": {
                                "artifactLocation": {
                                    "uri": ".github/workflows/check-pull-request.yml",
                                    "uriBaseId": "%SRCROOT%"
                                },
                                "region": {
                                    "startLine": 10,
                                    "startColumn": 3,
                                    "endColumn": 12
                                }
                            }
                        }
                    ]
                }
            ]
        }
    ]
}

The SARIF file contains:

  • Metadata about the linter
  • The matching rules (can be used by several results)
  • The results

So I'm not sure if the Go template output would be enough to generate a JSON structure like this.

@rhysd
Copy link
Owner

rhysd commented Jun 19, 2023

Thanks for the explanation. I'm also looking at examples in the spec.

https://docs.oasis-open.org/sarif/sarif/v2.1.0/os/sarif-v2.1.0-os.html#_Toc34317948

I'm not fully understanding, but

@rhysd rhysd added enhancement New feature or request and removed question Further information is requested labels Jun 19, 2023
@rhysd
Copy link
Owner

rhysd commented Jun 19, 2023

@mathroule BTW, what is your use case with this feature?

@mathroule
Copy link
Author

@mathroule BTW, what is your use case with this feature?

Linting GitHub Actions workflows and uploading the results to GitHub Code Scanning.

@rhysd
Copy link
Owner

rhysd commented Jun 19, 2023

It sounds great. It might be off-topic, but how do you plan to add the support for GitHub Code Scanning once actionlint's -format can output errors in SARIF?

@mathroule
Copy link
Author

It sounds great. It might be off-topic, but how do you plan to add the support for GitHub Code Scanning once actionlint's -format can output errors in SARIF?

By using a GitHub Actions: https://docs.github.com/en/code-security/code-scanning/integrating-with-code-scanning/uploading-a-sarif-file-to-github#example-workflow-for-sarif-files-generated-outside-of-a-repository

It assumes the workflow running the lint process, does not contains syntax errors, and can be run on GitHub Actions. Anyway, it's still useful for other workflows.

@rhysd
Copy link
Owner

rhysd commented Jun 21, 2023

I tried some templates for -format action, however it is difficult to enumerate all rules since currently there is no list of all rules in implementation of actionlint. I'll try to add some implementation to extend the error formatter used in -format so that errors can be output in SARIF format.

@mathroule
Copy link
Author

Indeed the list of rules must be outputted to be able to generate a SARIF. That's would be great to have it implemented.
Many thanks!

@rhysd
Copy link
Owner

rhysd commented Jun 22, 2023

@mathroule

I'm working on this in issue-311 branch.

I tried to output errors in SARIF format as follows. I verified the output with sarif-tools and it was correct. Does the output look good to you?

There are some limitations:

  • queryURI is fixed because sections in docs.md are not corresponding to each rules
  • ID and short description are same because actionlint's rules don't have short descriptions

Content of template.txt

{
    "$schema": "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/master/Schemata/sarif-schema-2.1.0.json",
    "version": "2.1.0",
    "runs": [
        {
            "tool": {
                "driver": {
                    "name": "GitHub Actions lint",
                    "version": "v1.6.25",
                    "informationUri": "https://github.com/rhysd/actionlint",
                    "rules": [
                        {{$first := true}}
                        {{range $ = allKinds }}
                            {{if $first}}{{$first = false}}{{else}},{{end}}
                            {
                                "id": {{json $.Name}},
                                "name": {{json $.Name}},
                                "defaultConfiguration": {
                                    "level": "error"
                                },
                                "properties": {
                                    "description": {{json $.Description}},
                                    "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
                                },
                                "shortDescription": {
                                    "text": {{json $.Name}}
                                },
                                "fullDescription": {
                                    "text": {{json $.Description}}
                                },
                                "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
                            }
                        {{end}}
                    ]
                }
            },
            "results": [
                {{$first := true}}
                {{range $ = .}}
                    {{if $first}}{{$first = false}}{{else}},{{end}}
                    {
                        "ruleId": {{json $.Kind}},
                        "ruleIndex": {{kindIndex $.Kind}},
                        "message": {
                            "text": {{json $.Message}}
                        },
                        "locations": [
                            {
                                "physicalLocation": {
                                    "artifactLocation": {
                                        "uri": {{json $.Filepath}},
                                        "uriBaseId": "%SRCROOT%"
                                    },
                                    "region": {
                                        "startLine": {{$.Line}},
                                        "startColumn": {{$.Column}},
                                        "endColumn": {{$.EndColumn}}
                                    }
                                }
                            }
                        ]
                    }
                {{end}}
            ]
        }
    ]
}

Command

actionlint -format "$(cat template.txt)" testdata/examples/main.yaml

Output

{
  "$schema": "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/master/Schemata/sarif-schema-2.1.0.json",
  "version": "2.1.0",
  "runs": [
    {
      "tool": {
        "driver": {
          "name": "GitHub Actions lint",
          "version": "v1.6.25",
          "informationUri": "https://github.com/rhysd/actionlint",
          "rules": [
            {
              "id": "syntax-check",
              "name": "syntax-check",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for GitHub Actions workflow syntax",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "syntax-check"
              },
              "fullDescription": {
                "text": "Checks for GitHub Actions workflow syntax"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "matrix",
              "name": "matrix",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for matrix combinations in \"matrix:\"",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "matrix"
              },
              "fullDescription": {
                "text": "Checks for matrix combinations in \"matrix:\""
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "credentials",
              "name": "credentials",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for credentials in \"services:\" configuration",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "credentials"
              },
              "fullDescription": {
                "text": "Checks for credentials in \"services:\" configuration"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "shell-name",
              "name": "shell-name",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for shell names used for scripts in \"run:\"",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "shell-name"
              },
              "fullDescription": {
                "text": "Checks for shell names used for scripts in \"run:\""
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "runner-label",
              "name": "runner-label",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for GitHub-hosted and preset self-hosted runner labels in \"runs-on:\"",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "runner-label"
              },
              "fullDescription": {
                "text": "Checks for GitHub-hosted and preset self-hosted runner labels in \"runs-on:\""
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "events",
              "name": "events",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for workflow trigger events at \"on:\"",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "events"
              },
              "fullDescription": {
                "text": "Checks for workflow trigger events at \"on:\""
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "job-needs",
              "name": "job-needs",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for job IDs in \"needs:\". Undefined IDs and cyclic dependencies are checked",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "job-needs"
              },
              "fullDescription": {
                "text": "Checks for job IDs in \"needs:\". Undefined IDs and cyclic dependencies are checked"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "action",
              "name": "action",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for popular actions released on GitHub, local actions, and action calls at \"uses:\"",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "action"
              },
              "fullDescription": {
                "text": "Checks for popular actions released on GitHub, local actions, and action calls at \"uses:\""
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "env-var",
              "name": "env-var",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for environment variables configuration at \"env:\"",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "env-var"
              },
              "fullDescription": {
                "text": "Checks for environment variables configuration at \"env:\""
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "id",
              "name": "id",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for duplication and naming convention of job/step IDs",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "id"
              },
              "fullDescription": {
                "text": "Checks for duplication and naming convention of job/step IDs"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "glob",
              "name": "glob",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for glob syntax used in branch names, tags, and paths",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "glob"
              },
              "fullDescription": {
                "text": "Checks for glob syntax used in branch names, tags, and paths"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "permissions",
              "name": "permissions",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for permissions configuration in \"permissions:\". Permission names and permission scopes are checked",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "permissions"
              },
              "fullDescription": {
                "text": "Checks for permissions configuration in \"permissions:\". Permission names and permission scopes are checked"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "workflow-call",
              "name": "workflow-call",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for reusable workflow calls. Inputs and outputs of called reusable workflow are checked",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "workflow-call"
              },
              "fullDescription": {
                "text": "Checks for reusable workflow calls. Inputs and outputs of called reusable workflow are checked"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "expression",
              "name": "expression",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Syntax and semantics checks for expressions embedded with ${{ }} syntax",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "expression"
              },
              "fullDescription": {
                "text": "Syntax and semantics checks for expressions embedded with ${{ }} syntax"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "deprecated-commands",
              "name": "deprecated-commands",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for deprecated \"set-output\", \"save-state\", \"set-env\", and \"add-path\" commands at \"run:\"",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "deprecated-commands"
              },
              "fullDescription": {
                "text": "Checks for deprecated \"set-output\", \"save-state\", \"set-env\", and \"add-path\" commands at \"run:\""
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "shellcheck",
              "name": "shellcheck",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for shell script sources in \"run:\" using shellcheck",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "shellcheck"
              },
              "fullDescription": {
                "text": "Checks for shell script sources in \"run:\" using shellcheck"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "pyflakes",
              "name": "pyflakes",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for Python script when \"shell: python\" is configured using Pyflakes",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "pyflakes"
              },
              "fullDescription": {
                "text": "Checks for Python script when \"shell: python\" is configured using Pyflakes"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            }
          ]
        }
      },
      "results": [
        {
          "ruleId": "syntax-check",
          "ruleIndex": 0,
          "message": {
            "text": "unexpected key \"branch\" for \"push\" section. expected one of \"branches\", \"branches-ignore\", \"paths\", \"paths-ignore\", \"tags\", \"tags-ignore\", \"types\", \"workflows\""
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "testdata/examples/main.yaml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 3,
                  "startColumn": 5,
                  "endColumn": 11
                }
              }
            }
          ]
        },
        {
          "ruleId": "glob",
          "ruleIndex": 10,
          "message": {
            "text": "character '\\' is invalid for branch and tag names. only special characters [, ?, +, *, \\ ! can be escaped with \\. see `man git-check-ref-format` for more details. note that regular expression is unavailable. note: filter pattern syntax is explained at https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#filter-pattern-cheat-sheet"
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "testdata/examples/main.yaml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 5,
                  "startColumn": 11,
                  "endColumn": 14
                }
              }
            }
          ]
        },
        {
          "ruleId": "runner-label",
          "ruleIndex": 4,
          "message": {
            "text": "label \"linux-latest\" is unknown. available labels are \"windows-latest\", \"windows-2022\", \"windows-2019\", \"windows-2016\", \"ubuntu-latest\", \"ubuntu-22.04\", \"ubuntu-20.04\", \"ubuntu-18.04\", \"macos-latest\", \"macos-latest-xl\", \"macos-13-xl\", \"macos-13\", \"macos-13.0\", \"macos-12-xl\", \"macos-12\", \"macos-12.0\", \"macos-11\", \"macos-11.0\", \"macos-10.15\", \"self-hosted\", \"x64\", \"arm\", \"arm64\", \"linux\", \"macos\", \"windows\". if it is a custom label for self-hosted runner, set list of labels in actionlint.yaml config file"
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "testdata/examples/main.yaml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 10,
                  "startColumn": 28,
                  "endColumn": 40
                }
              }
            }
          ]
        },
        {
          "ruleId": "expression",
          "ruleIndex": 13,
          "message": {
            "text": "\"github.event.head_commit.message\" is potentially untrusted. avoid using it directly in inline scripts. instead, pass it through an environment variable. see https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions for more details"
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "testdata/examples/main.yaml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 13,
                  "startColumn": 41,
                  "endColumn": 72
                }
              }
            }
          ]
        },
        {
          "ruleId": "action",
          "ruleIndex": 7,
          "message": {
            "text": "input \"node_version\" is not defined in action \"actions/setup-node@v3\". available inputs are \"always-auth\", \"architecture\", \"cache\", \"cache-dependency-path\", \"check-latest\", \"node-version\", \"node-version-file\", \"registry-url\", \"scope\", \"token\""
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "testdata/examples/main.yaml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 17,
                  "startColumn": 11,
                  "endColumn": 23
                }
              }
            }
          ]
        },
        {
          "ruleId": "expression",
          "ruleIndex": 13,
          "message": {
            "text": "property \"platform\" is not defined in object type {os: string}"
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "testdata/examples/main.yaml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 21,
                  "startColumn": 20,
                  "endColumn": 34
                }
              }
            }
          ]
        },
        {
          "ruleId": "expression",
          "ruleIndex": 13,
          "message": {
            "text": "receiver of object dereference \"permissions\" must be type of object but got \"string\""
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "testdata/examples/main.yaml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 22,
                  "startColumn": 17,
                  "endColumn": 51
                }
              }
            }
          ]
        }
      ]
    }
  ]
}

@rhysd
Copy link
Owner

rhysd commented Jun 22, 2023

Oh, I've just understood that ruleIndex is not mandatory.

https://docs.oasis-open.org/sarif/sarif/v2.1.0/os/sarif-v2.1.0-os.html#_Toc34317644

If ruleIndex can be omitted, the implementation can get much simpler. I'll update the branch.

@mathroule
Copy link
Author

Thanks @rhysd that's awesome!

I used this online validator: https://sarifweb.azurewebsites.net/Validation with "GitHub ingestion rules" enabled, and it reported some warnings with the sample output you provided:

  • Version should follow semantic versioning and become: "version": "1.6.25"
  • Rules ID and name must be different
  • Rule ID must be a "stable, opaque identifier" (the SARIF specification (3.49.3) explains the reasons for this)
  • Rule name is a PascalCase identifier that is understandable to an end user (3.49.7)

About ruleIndex, indeed it can be omitted.

Do you think the template can be directly integrated into actionlint to avoid the need for duplication? And call actionlint with something like that: actionlint -format sarif

@rhysd
Copy link
Owner

rhysd commented Jun 25, 2023

Thank you for the feedback.

Rule ID must be a "stable, opaque identifier" (the SARIF specification (3.49.3) explains the reasons for this)

This is not possible because actionlint doesn't have a stable and opaque identifier for each rule. Each rule has its name only.

Rule name is a PascalCase identifier that is understandable to an end user (3.49.7)

This is possible by adding a new template function. I'll add it.

Do you think the template can be directly integrated into actionlint to avoid the need for duplication?

It's possible, but I don't want to add support for dedicated format because it increases maintenance cost. For example, when version of SARIF is bumped, we need to follow it. So actionlint's policy is to support several formats (e.g. JSON, jSONL, Markdown, SARIF, ...) through -format option and Go template.

@rhysd
Copy link
Owner

rhysd commented Jun 25, 2023

I updated the branch.

  • Remove rule index as it is optional
  • Add toPascalCase template function so that rule names are in pascal case

Now it works as follows:

Template file

{
    "$schema": "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/master/Schemata/sarif-schema-2.1.0.json",
    "version": "2.1.0",
    "runs": [
        {
            "tool": {
                "driver": {
                    "name": "GitHub Actions lint",
                    "version": "1.6.25",
                    "informationUri": "https://github.com/rhysd/actionlint",
                    "rules": [
                        {{$first := true}}
                        {{range $ = allKinds }}
                            {{if $first}}{{$first = false}}{{else}},{{end}}
                            {
                                "id": {{json $.Name}},
                                "name": {{$.Name | toPascalCase | json}},
                                "defaultConfiguration": {
                                    "level": "error"
                                },
                                "properties": {
                                    "description": {{json $.Description}},
                                    "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
                                },
                                "shortDescription": {
                                    "text": {{json $.Name}}
                                },
                                "fullDescription": {
                                    "text": {{json $.Description}}
                                },
                                "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
                            }
                        {{end}}
                    ]
                }
            },
            "results": [
                {{$first := true}}
                {{range $ = .}}
                    {{if $first}}{{$first = false}}{{else}},{{end}}
                    {
                        "ruleId": {{json $.Kind}},
                        "message": {
                            "text": {{json $.Message}}
                        },
                        "locations": [
                            {
                                "physicalLocation": {
                                    "artifactLocation": {
                                        "uri": {{json $.Filepath}},
                                        "uriBaseId": "%SRCROOT%"
                                    },
                                    "region": {
                                        "startLine": {{$.Line}},
                                        "startColumn": {{$.Column}},
                                        "endColumn": {{$.EndColumn}}
                                    }
                                }
                            }
                        ]
                    }
                {{end}}
            ]
        }
    ]
}

Command

actionlint -format "$(cat template.txt)" testdata/examples/main.yaml

Output (formatted)

{
  "$schema": "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/master/Schemata/sarif-schema-2.1.0.json",
  "version": "2.1.0",
  "runs": [
    {
      "tool": {
        "driver": {
          "name": "GitHub Actions lint",
          "version": "1.6.25",
          "informationUri": "https://github.com/rhysd/actionlint",
          "rules": [
            {
              "id": "action",
              "name": "Action",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for popular actions released on GitHub, local actions, and action calls at \"uses:\"",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "action"
              },
              "fullDescription": {
                "text": "Checks for popular actions released on GitHub, local actions, and action calls at \"uses:\""
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "expression",
              "name": "Expression",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Syntax and semantics checks for expressions embedded with ${{ }} syntax",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "expression"
              },
              "fullDescription": {
                "text": "Syntax and semantics checks for expressions embedded with ${{ }} syntax"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "shellcheck",
              "name": "Shellcheck",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for shell script sources in \"run:\" using shellcheck",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "shellcheck"
              },
              "fullDescription": {
                "text": "Checks for shell script sources in \"run:\" using shellcheck"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "events",
              "name": "Events",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for workflow trigger events at \"on:\"",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "events"
              },
              "fullDescription": {
                "text": "Checks for workflow trigger events at \"on:\""
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "shell-name",
              "name": "ShellName",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for shell names used for scripts in \"run:\"",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "shell-name"
              },
              "fullDescription": {
                "text": "Checks for shell names used for scripts in \"run:\""
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "job-needs",
              "name": "JobNeeds",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for job IDs in \"needs:\". Undefined IDs and cyclic dependencies are checked",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "job-needs"
              },
              "fullDescription": {
                "text": "Checks for job IDs in \"needs:\". Undefined IDs and cyclic dependencies are checked"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "permissions",
              "name": "Permissions",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for permissions configuration in \"permissions:\". Permission names and permission scopes are checked",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "permissions"
              },
              "fullDescription": {
                "text": "Checks for permissions configuration in \"permissions:\". Permission names and permission scopes are checked"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "workflow-call",
              "name": "WorkflowCall",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for reusable workflow calls. Inputs and outputs of called reusable workflow are checked",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "workflow-call"
              },
              "fullDescription": {
                "text": "Checks for reusable workflow calls. Inputs and outputs of called reusable workflow are checked"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "syntax-check",
              "name": "SyntaxCheck",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for GitHub Actions workflow syntax",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "syntax-check"
              },
              "fullDescription": {
                "text": "Checks for GitHub Actions workflow syntax"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "runner-label",
              "name": "RunnerLabel",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for GitHub-hosted and preset self-hosted runner labels in \"runs-on:\"",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "runner-label"
              },
              "fullDescription": {
                "text": "Checks for GitHub-hosted and preset self-hosted runner labels in \"runs-on:\""
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "env-var",
              "name": "EnvVar",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for environment variables configuration at \"env:\"",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "env-var"
              },
              "fullDescription": {
                "text": "Checks for environment variables configuration at \"env:\""
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "id",
              "name": "Id",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for duplication and naming convention of job/step IDs",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "id"
              },
              "fullDescription": {
                "text": "Checks for duplication and naming convention of job/step IDs"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "glob",
              "name": "Glob",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for glob syntax used in branch names, tags, and paths",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "glob"
              },
              "fullDescription": {
                "text": "Checks for glob syntax used in branch names, tags, and paths"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "pyflakes",
              "name": "Pyflakes",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for Python script when \"shell: python\" is configured using Pyflakes",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "pyflakes"
              },
              "fullDescription": {
                "text": "Checks for Python script when \"shell: python\" is configured using Pyflakes"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "credentials",
              "name": "Credentials",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for credentials in \"services:\" configuration",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "credentials"
              },
              "fullDescription": {
                "text": "Checks for credentials in \"services:\" configuration"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "deprecated-commands",
              "name": "DeprecatedCommands",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for deprecated \"set-output\", \"save-state\", \"set-env\", and \"add-path\" commands at \"run:\"",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "deprecated-commands"
              },
              "fullDescription": {
                "text": "Checks for deprecated \"set-output\", \"save-state\", \"set-env\", and \"add-path\" commands at \"run:\""
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            {
              "id": "matrix",
              "name": "Matrix",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for matrix combinations in \"matrix:\"",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "shortDescription": {
                "text": "matrix"
              },
              "fullDescription": {
                "text": "Checks for matrix combinations in \"matrix:\""
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            }
          ]
        }
      },
      "results": [
        {
          "ruleId": "syntax-check",
          "message": {
            "text": "unexpected key \"branch\" for \"push\" section. expected one of \"branches\", \"branches-ignore\", \"paths\", \"paths-ignore\", \"tags\", \"tags-ignore\", \"types\", \"workflows\""
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "testdata/examples/main.yaml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 3,
                  "startColumn": 5,
                  "endColumn": 11
                }
              }
            }
          ]
        },
        {
          "ruleId": "glob",
          "message": {
            "text": "character '\\' is invalid for branch and tag names. only special characters [, ?, +, *, \\ ! can be escaped with \\. see `man git-check-ref-format` for more details. note that regular expression is unavailable. note: filter pattern syntax is explained at https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#filter-pattern-cheat-sheet"
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "testdata/examples/main.yaml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 5,
                  "startColumn": 11,
                  "endColumn": 14
                }
              }
            }
          ]
        },
        {
          "ruleId": "runner-label",
          "message": {
            "text": "label \"linux-latest\" is unknown. available labels are \"windows-latest\", \"windows-2022\", \"windows-2019\", \"windows-2016\", \"ubuntu-latest\", \"ubuntu-22.04\", \"ubuntu-20.04\", \"ubuntu-18.04\", \"macos-latest\", \"macos-latest-xl\", \"macos-13-xl\", \"macos-13\", \"macos-13.0\", \"macos-12-xl\", \"macos-12\", \"macos-12.0\", \"macos-11\", \"macos-11.0\", \"macos-10.15\", \"self-hosted\", \"x64\", \"arm\", \"arm64\", \"linux\", \"macos\", \"windows\". if it is a custom label for self-hosted runner, set list of labels in actionlint.yaml config file"
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "testdata/examples/main.yaml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 10,
                  "startColumn": 28,
                  "endColumn": 40
                }
              }
            }
          ]
        },
        {
          "ruleId": "expression",
          "message": {
            "text": "\"github.event.head_commit.message\" is potentially untrusted. avoid using it directly in inline scripts. instead, pass it through an environment variable. see https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions for more details"
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "testdata/examples/main.yaml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 13,
                  "startColumn": 41,
                  "endColumn": 72
                }
              }
            }
          ]
        },
        {
          "ruleId": "action",
          "message": {
            "text": "input \"node_version\" is not defined in action \"actions/setup-node@v3\". available inputs are \"always-auth\", \"architecture\", \"cache\", \"cache-dependency-path\", \"check-latest\", \"node-version\", \"node-version-file\", \"registry-url\", \"scope\", \"token\""
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "testdata/examples/main.yaml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 17,
                  "startColumn": 11,
                  "endColumn": 23
                }
              }
            }
          ]
        },
        {
          "ruleId": "expression",
          "message": {
            "text": "property \"platform\" is not defined in object type {os: string}"
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "testdata/examples/main.yaml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 21,
                  "startColumn": 20,
                  "endColumn": 34
                }
              }
            }
          ]
        },
        {
          "ruleId": "expression",
          "message": {
            "text": "receiver of object dereference \"permissions\" must be type of object but got \"string\""
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "testdata/examples/main.yaml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 22,
                  "startColumn": 17,
                  "endColumn": 51
                }
              }
            }
          ]
        }
      ]
    }
  ]
}

@mathroule
Copy link
Author

mathroule commented Jun 26, 2023

Thanks @rhysd!

One last thing: it would be great to dynamically output the GitHub Actions lint version. For instance, having something like this: "version": "1.6.25", =>"version": {{ getVersion }}.

Also, the code snippet can be outputted like this in the region block:

"region": {
    "startLine": {{$.Line}},
    "startColumn": {{$.Column}},
    "endColumn": {{$.EndColumn}},
    "snippet": {
        "text": {{json $.Snippet}}
    }
}

@rhysd
Copy link
Owner

rhysd commented Jun 26, 2023

Thanks. That's good point. I'll implement it tomorrow.

@rhysd
Copy link
Owner

rhysd commented Jun 27, 2023

I added getVersion function at 0f67af8. Now

{
    "$schema": "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/master/Schemata/sarif-schema-2.1.0.json",
    "version": "2.1.0",
    "runs": [
        {
            "tool": {
                "driver": {
                    "name": "GitHub Actions lint",
                    "version": {{ getVersion | json }},
                    "informationUri": "https://github.com/rhysd/actionlint",
                    "rules": [
                        {{$first := true}}
                        {{range $ = allKinds }}
                            {{if $first}}{{$first = false}}{{else}},{{end}}
                            {
                                "id": {{json $.Name}},
                                "name": {{$.Name | toPascalCase | json}},
                                "defaultConfiguration": {
                                    "level": "error"
                                },
                                "properties": {
                                    "description": {{json $.Description}},
                                    "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
                                },
                                "fullDescription": {
                                    "text": {{json $.Description}}
                                },
                                "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
                            }
                        {{end}}
                    ]
                }
            },
            "results": [
                {{$first := true}}
                {{range $ = .}}
                    {{if $first}}{{$first = false}}{{else}},{{end}}
                    {
                        "ruleId": {{json $.Kind}},
                        "message": {
                            "text": {{json $.Message}}
                        },
                        "locations": [
                            {
                                "physicalLocation": {
                                    "artifactLocation": {
                                        "uri": {{json $.Filepath}},
                                        "uriBaseId": "%SRCROOT%"
                                    },
                                    "region": {
                                        "startLine": {{$.Line}},
                                        "startColumn": {{$.Column}},
                                        "endColumn": {{$.EndColumn}},
                                        "snippet": {{json $.Snippet}}
                                    }
                                }
                            }
                        ]
                    }
                {{end}}
            ]
        }
    ]
}

formats errors as follows:

{
  "$schema": "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/master/Schemata/sarif-schema-2.1.0.json",
  "version": "2.1.0",
  "runs": [
    {
      "tool": {
        "driver": {
          "name": "GitHub Actions lint",
          "version": "1.6.25",
          "informationUri": "https://github.com/rhysd/actionlint",
          "rules": [
            {
              "id": "workflow-call",
              "name": "WorkflowCall",
              "defaultConfiguration": {
                "level": "error"
              },
              "properties": {
                "description": "Checks for reusable workflow calls. Inputs and outputs of called reusable workflow are checked",
                "queryURI": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
              },
              "fullDescription": {
                "text": "Checks for reusable workflow calls. Inputs and outputs of called reusable workflow are checked"
              },
              "helpUri": "https://github.com/rhysd/actionlint/blob/main/docs/checks.md"
            },
            // ...
          ]
        }
      },
      "results": [
        {
          "ruleId": "syntax-check",
          "message": {
            "text": "unexpected key \"branch\" for \"push\" section. expected one of \"branches\", \"branches-ignore\", \"paths\", \"paths-ignore\", \"tags\", \"tags-ignore\", \"types\", \"workflows\""
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "testdata/examples/main.yaml",
                  "uriBaseId": "%SRCROOT%"
                },
                "region": {
                  "startLine": 3,
                  "startColumn": 5,
                  "endColumn": 11,
                  "snippet": "    branch: main\n    ^~~~~~~"
                }
              }
            }
          ]
        },
        // ...
      ]
    }
  ]
}

@mathroule
Copy link
Author

Thanks @rhysd for taking the time to work on SARIF support. I really appreciate your help and dedication!

Any plan to merge issue-311 branch and release a new version of actionlint?

@rhysd
Copy link
Owner

rhysd commented Jun 29, 2023

@mathroule Nothing blocks merging the branch. I just confirmed the output looks good to you before doing it. I'll merge it soon.

I need some other issues to address before releasing next version. So it would take a while for shipping the next release.

@rhysd rhysd closed this as completed in cd0cf19 Jun 30, 2023
@rhysd
Copy link
Owner

rhysd commented Jun 30, 2023

I merged the branch. The SARIF template is put here: https://github.com/rhysd/actionlint/blob/main/testdata/format/sarif_template.txt

@mathroule
Copy link
Author

@rhysd the template https://github.com/rhysd/actionlint/blob/main/testdata/format/sarif_template.txt contains an error with snippet, it should be:

                                        "snippet": {
                                            "text": {{json $.Snippet}}
                                        }

Instead of:

                                        "snippet": {{json $.Snippet}}

@rhysd
Copy link
Owner

rhysd commented Jul 1, 2023

Ah, thanks for catching it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants