Exit code is zero if thresholds are not met

[u6f6o]

Environment

Locally:

• k6 version: k6 v0.25.1
• OS and version: mac Os Catalina 10.15.2

Gitlab:

• Docker image: loadimpact/k6:latest on gitlab-runner 12.9.1

Expected Behavior

According to https://k6.io/docs/using-k6/thresholds the exit code should be != 0 in case of thresholds that are not met:

If any of the thresholds had failed, the little green checkmark
✓ next to the threshold name (failed requests, http_req_duration) would have been a red cross ✗ instead, and k6 would have generated a non-zero exit code.

Actual Behavior

Exit code is 0

Steps to Reproduce the Problem

1. Define a new loadtest with thresholds that are not met
2. Check exit code

[mstoykov]

works for me ... I think you are mistaking checks and thresholds?
Can you provide an example script that fails, you can use https://test-api.k6.io or https://httpbin.test.k6.io/ :)

[u6f6o]

Thx for the fast response. Yes, I can. So there is the base.js file which looks like the following:

import {check} from "k6";
import http from "k6/http";
import {Rate, Trend} from 'k6/metrics';

export let trendRTT = new Trend('RTT');
export let contentOKRate = new Rate('Content OK');
export let crododileIds = ["1", "2", "FOOBAR"]

export function execute() {
    const crocodileId = crododileIds[Math.floor(Math.random() * crododileIds.length)]
    const url = 'https://test-api.k6.io/public/crocodiles/' + crocodileId + '/'
    const res = http.get(url);
    const contentOK = check(res, {
        "is status 200": (r) => r.status === 200
    });
    contentOKRate.add(contentOK);
    trendRTT.add(res.timings.duration);
}

And then there is the smoke.js file that imports the other js file (in my project there is also a load.js, stress.js file):

import {execute} from "./base.js";

export let options = {
    vus: 1,  // 1 user looping for 1 minute
    duration: '1m',
    thresholds: {
        'Content OK  ': ['rate>0.99']
    },
    throw: true
};

export default function() {
    execute();
}

When I execute this test, the thresholds are not met, yet the exit code is 0:

[u6f6o]

I think I found it. Idk why, but for some reason there are some extra spaces in 'Content OK ', when I remove them, everything works as expected. Sorry for the false alarm!

[u6f6o]

The only thing I am wondering about. Would it make sense maybe to add a warning/error if a threshold id refers to something that actually does not exist?

[na--]

👍 We sort of know about this issue and some related ones: #1346, #961

The threshold handling code needs an overhaul and we intend to make one soon: #1443

but it's worth pointing out the current issue specifically, so I did it in #1443 (comment)