The benchmark results in an error but the check shows success

[MahdiBM]

swift package --disable-sandbox \
      benchmark thresholds check \
      --path $PWD/Benchmarks/Thresholds/ \
      --format markdown

succeeds when the actual benchmark has resulted in an error.

[hassila]

You mean if the benchmark crashed, or?

[MahdiBM]

No the benchmark run doesn't crash. It simply throws an error.

I noticed the benchmark says Checking ["benchmark name 1", "benchmark name 2"] around the end, but if the benchmark run has resulted in an error, it doesn't mention that benchmark's name in there at all.

Probably all benchmark runs for a specific benchmark must fail for this to happen.
Then the Benchmarking package doesn't have any results for that certain benchmark to compare to, so it simply skips the comparison instead of complaining that where are the results that I'm supposed to compare to or something. It should just fail the benchmark comparison if any of the benchmarks resulted in an error.