Return a non 0 exit code when script execution fails

[luizbafilho]

k6 is always returning 0 exit code, even if we explicitly call fail or there is a script code error.

For example:

import { fail } from "k6";

export default function() {
  fail("failing")
};

$ k6 run script.js
$ echo $? 
0 # this should be a non 0 exit code

[na--]

Hmm thinking about this again in the context of your PR, I'm not sure exiting with an error code here is the correct approach. Actual parser errors and other things that prevent the script from running should cause k6 to exit with an error and they currently do, at least from my limited testing. But things are more complicated when runtime errors (i.e. fail() or exceptions) are concerned.

To demonstrate why, take the following example:

import { fail } from "k6";

export let options = {
    iterations: 3,
};

export default function () {
    if (__ITER % 3 == 0) {
        fail("test");
    }
};

Keep in mind that fail() is basically just a wrapper around throw new Error(sth), even the example we give in that link shows how to use it for non-critical errors. So should we return an error when only one out of three iterations fail? The answer probably is "it depends" - some users may consider it OK, others may want to fail the tests. Both use cases can be handled with custom thresholds and abortOnFail. So something like this:

import { fail } from "k6";
import { Rate } from "k6/metrics";

let error_counter = new Rate("rt_errors");

export let options = {
    iterations: 3,
    thresholds: {
        "rt_errors": [{ threshold: "rate<0.4", abortOnFail: true }]
    }
};

function actualDefaultFunc() {
    if (__ITER % 3 == 0) {
        fail("test");
    }
}

export default function () {
    try {
        actualDefaultFunc();
        error_counter.add(false);
    } catch (e) {
        error_counter.add(true);
        fail(e);
    }
};

Works and k6 exits without and error, even though one out of 3 iterations has failed. But if we decrease the errors rate threshold, or even if we increase the number of iterations to 4 (so 50% - 2 out of 4 iterations fail), k6 correctly exits with the error code 99 and a message ERRO[0001] some thresholds have failed. So the current k6 without any fixes correctly and flexibly can handle both use cases.

[na--]

Closing this in favor of #877