Aggregated submetrics for nested groups

[efdknittlfrank]

Feature Description

It would sometimes be really beneficial to be able to aggregate submetrics for nested groups. Here is an example:

export const options = {
  thresholds: {
    'http_reqs{group:::main}': ['count==3'], // not possible with k6 0.35
    'http_reqs{group:::main::sub}': ['count==1'],
  },
};
export default function() {
  group('main', function() {
    http.get('http://example.com');
    group('sub', function() {
      http.get('http://example.com');
    });
    http.get('http://example.com');
  });
}

Currently, the above test script fails because only 2 HTTP requests are made from the main group directly. The third request is happening inside a nested group. Unfortunately, there is no way to get the overall http_reqs count for a group, including all its sub groups. This is probably due to the fact that a group creates a "group path" which will then be used to tag each request and every request can only have a single group (path).

Maybe there is already a workaround available, for instance through the k6/execution module and setting scenario tags for the current VU? (Can they be removed at the end of a group?). Even if this works, it would add a lot of boilerplate to the test scripts. Of course, each request could be tagged separately, but it is too easy to miss one request and end up with incomplete/inaccurate metrics.

Suggested Solution (optional)

I'm not sure what the best way is to handle this. A few things come to mind:

• Tag HTTP requests with multiple groups, similar to how tags work. This could even be backwards compatible by introducing a new tag groups (note the plural s). The tag value would contain a list of all groups in the hierarchy, e.g. groups=['main', 'sub'] and allows filtering by any of the group names in the thresholds map.
• Allow some sort of wildcards or regular expressions in the sub-metric filter in the thresholds map. This is probably less discoverable by users and could lead to a breaking change, as group names currently are simply strings without any forbidden characters (not sure about the double-colon though). Implementing regular expressions is probably the most flexible, but also the most complicated at the same time (and could lead to an explosion of sub-metrics created, if I understand correctly how k6 currently handles its metrics).
• Always match nested groups when filtering in the thresholds map. This is a breaking change, but does not require tagging each request with an additional tag (e.g. "groups")

Already existing or connected issues / PRs (optional)

No response

[na--]

Thanks for opening this issue! We have some long-term plans to completely overhaul how the thresholds are implemented (#1441, #1136, #1313 and others), so we appreciate every use case that is not well served by the current implementation! And this is a very valid use case! I think it can be considered a subset of #1313, though it's so well described that I think I'd prefer to leave both issues open, for now.

We are in the midst of heavy refactoring in how thresholds and metrics work (e.g. #2071, #2251, #1831, #1321) and we still don't know how the next version of thresholds will look like. If it's remotely similar to the current one, I'd probably prefer your second suggestion ("some sort of wildcards or regular expressions in the sub-metric filter") as the solution to this issue, but that might end up affecting the performance negatively, so we might go with something else - it's too early to make such a decision 🤷‍♂️

All of that said,

Maybe there is already a workaround available

Yes, there is! 🎉 Since just the last k6 version, in fact 😅 In k6 v0.35.0 we added the ability to set (and unset) VU-wide metric tags from inside of the script. So you can have something like this now:

import http from 'k6/http';
import exec from 'k6/execution';
import { group } from 'k6';

export const options = {
    thresholds: {
        'http_reqs{mygroup:main}': ['count==3'],
        'http_req_duration{mygroup:main}': ['max<1000'],
    },
};
export default function () {
    exec.vu.tags['mygroup'] = 'main';
    group('main', function () {
        http.get('https://test.k6.io');
        group('sub', function () {
            http.get('https://httpbin.test.k6.io/anything');
        });
        http.get('https://test-api.k6.io');
    });

    delete exec.vu.tags['mygroup'];

    http.get('https://httpbin.test.k6.io/delay/3');
}

[efdknittlfrank]

Thanks for your extensive answer and referencing all existing issues of this code.

All of that said,

Maybe there is already a workaround available

Yes, there is! 🎉 Since just the last k6 version, in fact 😅 In k6 v0.35.0 we added the ability to set (and unset) VU-wide metric tags from inside of the script. So you can have something like this now:

Thanks for the example code! In fact, the quote reply is incomplete, as the text continues to read 😉 →

Maybe there is already a workaround available, for instance through the k6/execution module and setting scenario tags for the current VU?

So yes, I already expected something like this 🙂 It's definitely a viable workaround and it could be easily wrapped in a helper function for the time being:

function taggedGroup(name, fn, tags) {
  for (const [k, v] of Object.entries(tags)) exec.vu.tags[k] = v;
  const result = group(name, fn);
  for (const k of Object.keys(tags)) delete exec.vu.tags[k];
  return result;
}

A small problem is that one would actually need to keep a stack of tag values if inner groups overwrite a tag of an outer group. Otherwise the tag would be completely reset and missing from the second part of the outer group. Something to keep in mind when using this method: don't re-use tags.

One more questions: can exec.vu.tags be used in setup/teardown functions? I will try it out and extend this comment with the outcome, it could be useful for future visitors.

exec.vu.tags can be used in setup/teardown functions and as such solves my use case!

[efdknittlfrank]

A function which supports stacked tags could be easily implemented as follows:

function taggedGroup(name, fn, tags) {
	// store current tags
	const previousTags = Object.assign({}, exec.vu.tags);

	try {
		// apply tags
		Object.assign(exec.vu.tags, tags);
		return group(name, fn);
	} finally {
		// restore tags
		for (const k of Object.keys(tags)) {
			if (k in previousTags) {
				exec.vu.tags[k] = previousTags[k];
			} else {
				delete exec.vu.tags[k];
			}
		}
	}
}

Improvements welcome

[efdknittlfrank]

Actually, once you start applying the tags, you don't need to create a group at all. The taggedGroup function can be written more generically, simply accepting tags and a callback:

function tagged(tags, fn) {
	// create copy of current tags
	const previousTags = Object.assign({}, exec.vu.tags);
	try {
		// apply tags
		Object.assign(exec.vu.tags, tags);
		return fn();
	} finally {
		// restore tags
		for (const k of Object.keys(tags)) {
			if (k in previousTags) {
				exec.vu.tags[k] = previousTags[k];
			} else {
				delete exec.vu.tags[k];
			}
		}
	}
}

Usage:

tagged({ mytag: 42 }, () => {
  tagged({ yourtag: 21 }, () => {
    http.get('https://httpbin.test.k6.io/get');
  });
});

[mstoykov]

Given that the workaround seems to work for this case, the lack of other people reporting it and the unlikeliness of:

1. us having tags with multiple values - which seems like a thing that doesn't exist anywhere - it is always key-value *pair.
2. having wildcards in thresholds
3. having wildcards in thresholds that aren't wildcards.

I am going to close this.