Conftest has significant memory footprint when evaluating policies against big configs

[msarvar]

Hi! We recently started seeing OOMs in our CI boxed when running the conftest to run our policy enforcement. It was only happening when using conftest in conjunction with regula and somewhat bit terraform plan json file. I created an example project https://github.com/msarvar/contest-tracing to demonstrate the issue. The README has instructions on how to reproduce the issue.

Here is a memory usage example when running it on my machine:

cmd: conftest, mem: 106.14 Mb, cpu: 139.00
cmd: conftest, mem: 449.69 Mb, cpu: 284.40
cmd: conftest, mem: 895.57 Mb, cpu: 145.20
cmd: conftest, mem: 1251.38 Mb, cpu: 209.10
cmd: conftest, mem: 1711.14 Mb, cpu: 101.90
cmd: conftest, mem: 1967.42 Mb, cpu: 147.10
cmd: conftest, mem: 2414.09 Mb, cpu: 99.50
cmd: conftest, mem: 2889.59 Mb, cpu: 319.90
cmd: conftest, mem: 3131.41 Mb, cpu: 136.50
cmd: conftest, mem: 3440.85 Mb, cpu: 100.10
cmd: conftest, mem: 3924.21 Mb, cpu: 100.00
cmd: conftest, mem: 4331.94 Mb, cpu: 99.40
cmd: conftest, mem: 4655.29 Mb, cpu: 947.60
cmd: conftest, mem: 4684.96 Mb, cpu: 155.90
cmd: conftest, mem: 4977.67 Mb, cpu: 99.10
cmd: conftest, mem: 5368.38 Mb, cpu: 100.10
cmd: conftest, mem: 5766.21 Mb, cpu: 100.00
cmd: conftest, mem: 6199.54 Mb, cpu: 99.70

p.s. The example is an output of a while loop that parses conftest out of ps aux output.

I did some digging into the code, and turn out the reason is due to always on tracing. I have a PR that will make tracing optional and only trace when --trace flag is provided. The same example repo without tracing runs under a second with less than 50mb footprint.

[msarvar]

Looks like this PR introduced the regression. Specifically this logic was lost.

[jpreese]

Resolved via #556. Thanks @msarvar