Custom policy checks with non-JSON output produce errors

[bgalkows]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

Since Atlantis v0.24.0, using policy scanning tools which yield non-JSON output (anything but Conftest) in a policy check step now produce errors. With past Atlantis versions, our workflow included a custom workflow with a policy check step to run Cnspec for Terraform plan scanning. This is no longer possible as now it appears policy checks only support JSON output.

Here is the error output that results from running policy check tools other than Conftest:
unable to unmarshal conftest output

Throwing this error for any non-JSON output was added around line 500 of this file:
https://github.com/runatlantis/atlantis/blob/main/server/events/project_command_runner.go

Reproduction Steps

To reproduce this error, simply run any policy scanning tool that does not produce JSON output within a policy check step. See below for a basic example:

policy_check:
    steps:
        - run: cnspec scan terraform plan example.plan --policy-bundle example_policy.mql.yaml

Logs

There are no relevant logs since this bug occurs outside of atlantis plan and atlantis apply statements in the policy check steps of custom workflows.

Environment details

• Atlantis version: v0.24.0 and above

Our deployment is Kubernetes on GCP, but this is not relevant to the bug. This can be reproduced with any deployment setup.

Additional Context

We have been using other policy scanning tools with Atlantis for a long time, and this change has blocked us from updating to any versions newer than v0.23.5.

If Conftest is intended to be the only supported policy checking tool, that would fully explain this error. However, a breaking change like this should be thoroughly highlighted in the release notes.

[raffimohammed]

Running into the same issue on v0.25.0. I run a python program instead of conftest which just prints non-json text.

workflows:
custom_workflow1:
policy_check:
steps:
- run: python ~/bin/approve_status_checks.py

[kumaresh0]

Seems this not fixed Non-json outputs still produces errors

#4243