feature/adding event hubs output plugin

[polatengin]

Allow using Azure Event Hubs as an output target, so you can use --output eventhubs to fork k6 outputs to Azure Event Hubs.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[imiric]

Thanks for proposing this @polatengin! I left a few minor comments, and the testing situation should be improved before we consider merging this.

Also make sure to run go mod vendor / go mod tidy and commit the changes, otherwise you won't get a passing build in CI.

The k6 team is not familiar with Azure Event Hubs, but from what I could gather it exposes a Kafka-compliant endpoint. Is there a reason why the current Kafka output or the new Kafka extension couldn't be used instead of adding a custom collector for it?

It also appears to be based on AMQP, which is a much more generic and widely used protocol. From that perspective it would be preferable if we added AMQP support instead (say, as a new k6 extension), which would close #1073, and supported Event Hubs that way.

I'm interested to hear your thoughts on this, and also from @na-- @mstoykov.

[imiric]

Please update all documentation comments for this collector.

[imiric]

I'm not clear what you're testing here... SampleToRow() is a CSV collector function, and here you're comparing two structs for equality, so I don't see how it could fail. 😕

In this file I would expect to see tests of the Collector methods. If you can't start a mock server to test against, consider abstracting away the c.client calls, e.g. by passing some interface you can mock out in the tests.

[na--]

I don't have the time to look at this PR closely right now, sorry... I am also not very familiar with Azure Event Hubs or Application Insights (#1757). But, in general, if we can use a generic open protocol to output metrics to some service, there needs to be a good reason not to use it and instead to use a custom SDK for it (https://github.com/Azure/azure-event-hubs-go in this case, from what I can see). Moreover, that dependency itself seems to depend on a dead library... 😕

In any case, I also wanted to say that we plan to majorly refactor how the k6 outputs are handled, currently targeting k6 v0.31.0 (Feb 2021). We aim to fix a bunch of issues (#1606, #1075, #1423 among others) and to also enable the addition of output extensions with xk6. So it's very unlikely that we'll merge this PR or #1757 before k6 v0.31.0. And even then, I can't promise that these outputs would be merged straight into the k6 core, they might be better off as extensions for a while, to mature. We'd need to gauge the user interest against the maintenance burden of such changes.

[simskij]

I can assist here if it's any help, @na--. I'm quite familiar with Azure 👍

[polatengin]

I can assist here if it's any help, @na--. I'm quite familiar with Azure 👍

Hi @simskij thanks for your support offer. It's greatly appreciated 👍

[polatengin]

In any case, I also wanted to say that we plan to majorly refactor how the k6 outputs are handled, currently targeting k6 v0.31.0 (Feb 2021). We aim to fix a bunch of issues (#1606, #1075, #1423 among others) and to also enable the addition of output extensions with xk6. So it's very unlikely that we'll merge this PR or #1757 before k6 v0.31.0

Hey @na-- , is there an Early Access program for the new output system with xk6? I'd like to be in such program if you have one, also please count me in if you're going to do some design sessions and looking for community contributions.

[imiric]

@polatengin There's no "Early Access" program for xk6, but you're welcome to try creating an extension and publishing it on GitHub. Proper documentation is still pending, but you can take a look at this article, and use existing extensions as examples. It's really quite straightforward :)

If you need any help or have any suggestions feel free to post on the forum or Slack.

[mstoykov]

Hi @polatengin , thanks for this PR 🙇.

k6 has had support for output extensions for some time now. As of today it also has a template repo for output extensions. This plus all the other extension outputs should be enough for people to extract this code in an extension and use it without it being in k6 core.

Given that I am closing this PR.