Better url aggregating functionality

[marklagendijk]

In the documentation about Aggregating results for HTTP requests it says:

By default, requests report the name tag with the value of the request URL. For URLs that contain dynamic parts this might not be desirable since it can introduce a large number of unique URLs in the metrics stream.

However, this means that one needs to do this for every url that is affected by this (in our case most of the urls).
Also, this does not play nice in combination with response.submitForm(), because with response.submitForm() the url is read from the HTML form in the response, and may be unknown to the developer.

I have 2 proposals to solve this issue in a nice way.

1. Remove query parameter values

I would propose to remove the query param values from the default name tag. This feature should be optional. You could argue that it should be on by default, but for backwards compatibility this might not be desirable. When enabled the default name tag for https://example.com?t=1 or https://example.com?t=2 would become something like https://example.com?t=[]

2. Allow specifying a function that calculates the name tag

Proposal 1 only solves the issue for query parameters, but path parameters can have the same issue.
A solution which would work for all cases is to allow specifying a function as value for a tag.
Example:

response.submitForm({
  params: {
    tags: {
      name: request => request.url.replace(/\/path\/.*/, '/path')
    }
  }
});

[na--]

Thanks for the issue, I think both proposals have merit and would be useful!

The new option, which you're correct to say should be disabled by default for backwards compatibility, probably needs to only be accessible as an environment variable and a script option. I don't think it would merit a CLI flag at this time, though we can always add it later.

Regarding implementation specifics for the second one, I'd prefer if the callback wasn't part of the tags map, rather I think it should be specified directly in the params root. And if we're going to have a callback anyway, we might as well get more use out of it. It can be something that accepts the whole tags map (and likely the request and response objects as well) before k6 emits it and could modify whatever the user needs, including the name tag.

There will be some difficulties with implementing this callback in both cases though, just because of how the k6 HTTP request making code is structured, but I don't think they are insurmountable... And the whole feature could also be made more useful when we implement those two issues: #761 and #884

[marklagendijk]

The new option, which you're correct to say should be disabled by default for backwards compatibility, probably needs to only be accessible as an environment variable and a script option. I don't think it would merit a CLI flag at this time, though we can always add it later.

I agree.

Regarding implementation specifics for the second one, I'd prefer if the callback wasn't part of the tags map, rather I think it should be specified directly in the params root. And if we're going to have a callback anyway, we might as well get more use out of it. It can be something that accepts the whole tags map (and likely the request and response objects as well) before k6 emits it and could modify whatever the user needs, including the name tag.

Nice idea! That would make the feature a lot more powerful. When we have this we can just add an example in the aggregating http request results documentation using this callback.

And the whole feature could also be made more useful when we implement those two issues: #761 and #884

Nice issues! I already thought about creating one for default options, but nice that it is already there 😃.

[marklagendijk]

I implemented the first proposal in #1171

[olegbespalov]

Closing in favor of #2461, which is inspired by a design doc of a new HTTP