Enable kafka logs integration (gelf and json format)

[ivanape]

Enable to send logs to a Kafka topic supporting GELF and JSON formats.

To activate this feature

k6 run --log-output=kafka=[localhost:29092,localhost:29093,localhost:29094],topics=logs,level=debug,format=gelf,label.project_key=my_project_key,label.project_name=my_project_name demo.js

Example used during testing:

import { check, sleep } from "k6";
import http from "k6/http";
export let options = {
  duration: "1m",
  vus: 1
};
export default function() {

  console.log("Init test");

  let res;
  res = http.get("https://httpbinn.org/get");
  check(res, { "status is 200": r => r.status === 200 });
  res = http.get("https://httpbin.org/bearer", {
    headers: { Authorization: "Bearer da39a3ee5e6b4b0d3255bfef95601890afd80709-xyz" }
  });

  if (res.status != 200) {
    console.error(res.body);
  }
  check(res, {
    "status is 200": r => r.status === 200,
    "is authenticated": r => r.json()["authenticated"] === true
  });
  res = http.get("https://httpbin.org/base64/azYgaXMgYXdlc29tZSE=");
  check(res, {
    "status is 200": r => r.status === 200,
    "k6 is awesome!": r => r.body === "k6 is awesome!"
  });
  sleep(1);

  console.log("End test");

}

Screenshots:

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[ivanape]

@na--, @mstoykov, could you provide me some help in order to run the checks?

For me it's not clear what is happening :)

Thank you!

[na--]

@ivanape, thanks for making this PR! I re-ran the tests, the xk6 ones were failing because of grafana/xk6#17 (comment)

That said, I'm not sure we'll merge this PR in the k6 core any time soon... 😞 I don't want to discourage you, but I'm not sure there is a ton of demand for shipping logs to kafka. And I haven't seen an open issue about this, so I assume you went straight for the PR, despite the advice here?

We've actually decided to gradually going to deprecate the existing kafka output and move it to an xk6 extension: #1934. So we might just allow extensions for the logging subsystem of k6 as well, we'll see. We'll discuss this internally and get back to you, but for sure this is currently not a top priority for us, sorry.

[na--]

I'm closing this, since, as I mentioned above, it's not likely it will be merged in the k6 core.

At this point, when it comes to custom log outputs, we'll probably only merge support for output extensions (#1939), if anyone contributes a good PR for that.

For built-in log outputs, we'll probably only merge additional support for local files (#2249), or contribute it ourselves soon. These local files can have uses in CI, and can be used by various log forwarding software to ship logs to other places (maybe even kafka).

Thank you again for trying to contribute this and sorry we won't merge it! 😞