diff --git a/src/data/markdown/translated-guides/en/07 Misc/05 k6 Extensions.md b/src/data/markdown/translated-guides/en/07 Misc/05 k6 Extensions.md index 1d49416863..83b8ba8910 100644 --- a/src/data/markdown/translated-guides/en/07 Misc/05 k6 Extensions.md +++ b/src/data/markdown/translated-guides/en/07 Misc/05 k6 Extensions.md @@ -1,4 +1,607 @@ --- title: 'k6 Extensions' -redirect: 'https://k6.io/blog/extending-k6-with-xk6' ---- \ No newline at end of file +--- + +> *Since v0.29.0 (JavaScript extensions) and v0.31.0 (Output extensions)* + +Traditionally, extending k6 with custom functionality that isn't available in the +open source tool has been possible in one of two ways: + +- By importing a JavaScript library. This is a simple way to extend the functionality + of a test script, though it has some drawbacks. Since JavaScript in k6 runs in a + virtual machine [that is unlike the one used in web browsers and + NodeJS](/#what-k6-does-not), it's not possible to use libraries that write to + disk, use JS APIs other than the ones built into k6, or need some new binary + protocol support. Furthermore, while k6 performs well in most use cases, all + execution is interpreted by the [Go JavaScript runtime](https://github.com/dop251/goja), + which can impact the performance of certain resource intensive operations + (cryptography, compression, etc.). + +- By forking the [k6 repository](https://github.com/k6io/k6), doing the changes in Go + and submitting a pull request for review by the core team (preferably following + the [contribution guidelines](https://github.com/k6io/k6/blob/master/CONTRIBUTING.md)). + This is a great way to contribute new core features, but it can be a lengthy + process, and submissions might be rejected if they don't align with the long-term + vision of the project. + +To address these issues and allow the community to more easily adapt k6 to fit their +needs, we released the [xk6 framework](https://github.com/k6io/xk6) and the concept +of k6 extensions. + + +## What are k6 extensions? + +k6 extensions are standalone Go projects that call k6 APIs but are otherwise +unrestricted in their functionality. This provides the freedom for extension authors +to experiment with novel integrations with k6 that could eventually easily become +part of core. In this sense extensions can be thought of as a "testing ground" for +eventual promotion upstream, once their features and API is stable, and given that +the extension is generally useful for all k6 users. + + +## What is xk6? + +[xk6](https://github.com/k6io/xk6) is a command-line tool and framework inspired by +[xcaddy](https://github.com/caddyserver/xcaddy), designed for building custom k6 +binaries that bundle one or more extensions written in Go. + +Its main features are: + +- Ease of use: with a few commands even less technical users should be able to build + their own k6 binaries, given that they have the Go toolchain installed and any + dependencies required by a specific extension. +- Simple API for Go programmers that handles the Go<->JS translation, with the + ability to call any public k6 Go API. Extensions are first-class components along + with other built-in modules. +- Cross-platform like Go and runs great on macOS, Windows and Linux. + + +### Extension types + +The initial version of xk6 supported only JavaScript extensions, but since then we've +added support for Output extensions, and are considering expanding this to other +areas of k6 as well. + +The currently supported extension types are: + +#### JavaScript extension + +These extensions enhance the k6 JavaScript API to add support for new network +protocols, achieve better performance than equivalent JS libraries, or implement +features that are unlikely to be made part of the k6 core. + +Some examples include: [xk6-sql](https://github.com/imiric/xk6-sql), +[xk6-crypto](https://github.com/szkiba/xk6-crypto) and [xk6-file](https://github.com/avitalique/xk6-file). + + +#### Output extension + +While k6 has built-in support for many popular [output +backends](/docs/getting-started/results-output/), this list will undoubtedly not be +exhaustive. Support for new systems and novel ways of handling the metric data +generated by k6 can be easily added with Output extensions. These receive metric +samples from k6, and are able to do any processing or further dispatching. + +Some examples include: [xk6-output-kafka](https://github.com/grafana/xk6-output-kafka) +and [xk6-prometheus](https://github.com/szkiba/xk6-prometheus). + + +## Getting started + +There are two ways in which xk6 can be used: + +- By k6 users that wish to enhance their tests with existing extensions. A + familiarity with the command line and Go is preferred, but not required. + +- By Go developers interested in creating their own k6 extension. They'll need to be + familiar with both Go and JavaScript, understand how the k6 Go<->JS bridge works, + and maintain a public repository for the extension that keeps up to date with any + breaking API changes while xk6 is being stabilized. + + + + +## Using xk6 to build a k6 binary + +You might have found a neat k6 extension on the [Ecosystem page](/ecosystem) or on +[GitHub](https://github.com/topics/xk6) and wish to use it in your tests. Great! The +process is relatively simple and we're working on simplifying it further for end +users. + +You'll first need to setup [Go](https://golang.org/doc/install) and +[Git](https://git-scm.com/). Make sure that your `$PATH` environment variable is +updated and that `go version` returns the correct version. + +Then install xk6 with: + + + +```bash +$ go install github.com/k6io/xk6/cmd/xk6@latest +``` + + + +And confirm that `which xk6` on Linux/macOS or `where xk6` on Windows returns a +valid path. Otherwise ensure that `$GOPATH` is correctly defined and that +`$GOPATH/bin` is added to your `$PATH` environment variable. See the +[Go documentation](https://golang.org/cmd/go/#hdr-GOPATH_environment_variable) for details. + +Once xk6 is installed, building a k6 binary with one or more extensions can be done +with the following command: + + + +```bash +$ xk6 build latest \ + --with github.com/imiric/xk6-sql \ + --with github.com/szkiba/xk6-prometheus +``` + + + +This will build a `k6` binary in the current directory based on the most recently +released k6 version, bundling a JavaScript and an Output extension. Now you can run a +script with this binary that uses the [SQL JS API](https://github.com/imiric/xk6-sql) +and the [Prometheus output](https://github.com/szkiba/xk6-prometheus). + +Note that when running the script we have to specify the binary just built in the +current directory (`./k6`), as otherwise some other `k6` binary found on the system +could be executed which might not have the extensions built-in. This is only the case +on Linux and macOS, as Windows shells will execute the binary in the current +directory first. + +Also note that because of the way xk6 works, vendored dependencies (the `vendor` +directory created by `go mod vendor`) will **not** be taken into account when +building a binary, and you don't need to commit them to the extension repository. + + +## Writing a new extension + +The first thing you should do before starting work on a new extension is to confirm +that a similar extension doesn't already exist for your use case. Take a look at +the [Ecosystem page](/ecosystem) and the [`xk6` topic on GitHub](https://github.com/topics/xk6). +For example, if a system you need support for can be tested with a generic protocol +like MQTT, prefer using [xk6-mqtt](https://github.com/pmalhaire/xk6-mqtt) +instead of creating an extension that uses some custom protocol. +Also, prefer to write a pure JavaScript library that can be used in k6 if you can +avoid writing an extension in Go, since a JS library will be better supported and +likely easier to write and reuse than an extension. + +Next, you should decide the type of extension you need. A JavaScript extension is a +good fit if you want to extend the JS functionality of your script, or add support +for a new network protocol to test with. An Output extension would be more suitable +if you need to process the metrics emitted by k6 in some way, submit them to a +specific storage backend that was previously unsupported, etc. The k6 APIs you'll +need to use and things to consider while developing will be different in each case. + + +### Writing a new JavaScript extension + +A simple JavaScript extension consists of a main module struct that exposes +methods that can be called from a k6 test script. For example: + + + + + +```go +package compare + +type Compare struct{} + +func (*Compare) IsGreater(a, b int) bool { + return a > b +} +``` + + + +In order to use this from k6 test scripts we need to register the module +by adding the following: + + + +```go +import "go.k6.io/k6/js/modules" + +func init() { + modules.Register("k6/x/compare", new(Compare)) +} +``` + + + +Note that all k6 extensions should have the `k6/x/` prefix and the short name +must be unique among all extensions built in the same k6 binary. + +The final extension code will look like so: + + + +```go +package compare + +import "go.k6.io/k6/js/modules" + +func init() { + modules.Register("k6/x/compare", new(Compare)) +} + +type Compare struct{} + +func (*Compare) IsGreater(a, b int) bool { + return a > b +} +``` + + + +We can then build a k6 binary with this extension by running +`xk6 build --with xk6-compare=.`. In this case `xk6-compare` is the +Go module name passed to `go mod init`, but in a real-world scenario +this would be a URL. + +Finally we can use the extension in a test script: + + + +```javascript +import compare from 'k6/x/compare'; + +export default function () { + console.log(compare.isGreater(2, 1)); +} +``` + + + +And run the test with `./k6 run test.js`, which should output `INFO[0000] true`. + + +#### Notable features + +The k6 Go-JS bridge has a few features we should highlight: + +- Go method names will be converted from Pascal case to Camel case when + accessed in JS, as in the example above: `IsGreater` becomes `isGreater`. + +- Similarly, Go field names will be converted from Pascal case to Snake case. + For example, the struct field `SomeField string` will be accessible in JS as + the `some_field` object property. This behavior is configurable with the `js` + struct tag, so this can be changed + with SomeField string `js:"someField"` + or the field can be hidden with `js:"-"`. + +- Methods with a name prefixed with `X` will be transformed to JS + constructors, and will support the `new` operator. + For example, defining the following method on the above struct: + + + +```go +type Comparator struct{} + +func (*Compare) XComparator() *Comparator { + return &Comparator{} +} +``` + + + + Would allow creating a `Comparator` instance in JS with `new compare.Comparator()`, + which is a bit more idiomatic to JS. + + +#### Advanced JavaScript extension + +> ℹ️ **Note** +> +> The internal JavaScript module API is currently (October 2021) in a state of +> flux. The traditional approach of initializing JS modules involves calling +> [`common.Bind()`](https://pkg.go.dev/go.k6.io/k6/js/common#Bind) +> on any objects that need to be exposed to JS. This method has a few technical +> issues we want to improve, and also isn't flexible enough to implement +> new features like giving extensions access to internal k6 objects. +> Starting from v0.32.0 we've introduced a new approach for writing +> JS modules and is the method we'll be describing below. While this new API +> is recommended for new modules and extensions, note that it's still in +> development and might change while it's being stabilized. + +If your extension requires access to internal k6 objects to, for example, +inspect the state of the test during execution, we will need to make some +slightly more complicated changes to the above example. + +Our main `Compare` struct should implement the +[`modules.Instance` interface](https://pkg.go.dev/go.k6.io/k6/js/modules#Instance) +and embed +[`modules.InstanceCore`](https://pkg.go.dev/go.k6.io/k6/js/modules#InstanceCore) +in order to access internal k6 objects such as: +- [`lib.State`](https://pkg.go.dev/go.k6.io/k6/lib#State): the VU state with + values like the VU ID and iteration number. +- [`goja.Runtime`](https://pkg.go.dev/github.com/dop251/goja#Runtime): the + JavaScript runtime used by the VU. +- a global `context.Context` which contains other interesting objects like + [`lib.ExecutionState`](https://pkg.go.dev/go.k6.io/k6/lib#ExecutionState). + +Additionally there should be a root module implementation of the +[`modules.IsModuleV2` interface](https://pkg.go.dev/go.k6.io/k6/js/modules#IsModuleV2) +that will serve as a factory of `Compare` instances for each VU. Note that this +can have memory implications depending on the size of your module. + +Here's how that would look like: + + + +```go +package compare + +import "go.k6.io/k6/js/modules" + +func init() { + modules.Register("k6/x/compare", New()) +} + +type ( + // RootModule is the global module instance that will create Compare + // instances for each VU. + RootModule struct{} + + // Compare represents an instance of the JS module. + Compare struct { + // InstanceCore provides some useful methods for accessing internal k6 + // objects like the global context, VU state and goja runtime. + modules.InstanceCore + // Comparator is the exported module instance. + *Comparator + } +) + +// Ensure the interfaces are implemented correctly. +var ( + _ modules.Instance = &Compare{} + _ modules.IsModuleV2 = &RootModule{} +) + +// New returns a pointer to a new RootModule instance. +func New() *RootModule { + return &RootModule{} +} + +// NewModuleInstance implements the modules.IsModuleV2 interface and returns +// a new instance for each VU. +func (*RootModule) NewModuleInstance(m modules.InstanceCore) modules.Instance { + return &Compare{InstanceCore: m, Comparator: &Comparator{}} +} + +// Comparator is the exported module instance. +type Comparator struct{} + +// IsGreater returns true if a is greater than b, or false otherwise. +func (*Comparator) IsGreater(a, b int) bool { + return a > b +} + +// GetExports implements the modules.Instance interface and returns the exports +// of the JS module. +func (c *Compare) GetExports() modules.Exports { + return modules.Exports{Default: c.Comparator} +} +``` + + + +Currently this module isn't taking advantage of the methods provided by +[`modules.InstanceCore`](https://pkg.go.dev/go.k6.io/k6/js/modules#InstanceCore) +because our simple example extension doesn't require it, but here is +a contrived example of how that could be done: + + + +```go +type InternalState struct { + ActiveVUs int64 `js:"activeVUs"` + Iteration int64 + VUID uint64 `js:"vuID"` + VUIDFromRuntime goja.Value `js:"vuIDFromRuntime"` +} + +func (c *Compare) GetInternalState() *InternalState { + state := c.GetState() + ctx := c.GetContext() + es := lib.GetExecutionState(ctx) + rt := c.GetRuntime() + + return &InternalState{ + VUID: state.VUID, + VUIDFromRuntime: rt.Get("__VU"), + Iteration: state.Iteration, + ActiveVUs: es.GetCurrentlyActiveVUsCount(), + } +} +``` + + + +Running a script like: + + + +```javascript +import compare from 'k6/x/compare'; + +export default function () { + const state = compare.getInternalState(); + console.log(`Active VUs: ${state.activeVUs} +Iteration: ${state.iteration} +VU ID: ${state.vuID} +VU ID from runtime: ${state.vuIDFromRuntime}`); +} +``` + + + +Should output: + + + +```bash +INFO[0000] Active VUs: 1 +Iteration: 0 +VU ID: 1 +VU ID from runtime: 1 source=console +``` + + + +> ℹ️ **Note** +> +> For a more extensive usage example of this API, take a look at the +> [`k6/execution`](https://github.com/grafana/k6/blob/v0.34.1/js/modules/k6/execution/execution.go) +> module. + +Notice that the JavaScript runtime will transparently convert Go types like +`int64` to their JS equivalent. For complex types where this is not +possible your script might fail with a `TypeError` and you will need to convert +your object to a [`goja.Object`](https://pkg.go.dev/github.com/dop251/goja#Object) or [`goja.Value`](https://pkg.go.dev/github.com/dop251/goja#Value). + +For example: + + + +```go +type Comparator struct{} + +func (*Compare) XComparator(call goja.ConstructorCall, rt *goja.Runtime) *goja.Object { + return rt.ToValue(&Comparator{}).ToObject(rt) +} +``` + + + +This also demonstrates the native constructors feature from goja, where methods +with this signature will be transformed to JS constructors, and also have +the benefit of receiving the `goja.Runtime`, which is an alternative way +to access it in addition to the `GetRuntime()` method shown above. + + +#### Things to keep in mind + +- The code in the `default` function (or another function specified by + [`exec`](/using-k6/scenarios/#common-options)) will be executed many + times during a test run and possibly in parallel by thousands of VUs. + As such any operation of your extension meant to run in that context + needs to be performant and [thread-safe](https://en.wikipedia.org/wiki/Thread_safety). +- Any heavy initialization should be done in the [init + context](/javascript-api/init-context/) if possible, and not as part of the + `default` function execution. +- Custom metric emission can be done by creating new metrics using + [`stats.New()`](https://pkg.go.dev/go.k6.io/k6/stats#New) + and emitting them using [`stats.PushIfNotDone()`](https://pkg.go.dev/go.k6.io/k6/stats#PushIfNotDone). + For an example of this see the [`xk6-remote-write` extension](https://github.com/dgzlopes/xk6-remote-write). + + +### Writing a new Output extension + +Output extensions are written similarly to JavaScript extensions, but have a +different API and performance considerations. + +The core of an Output extension is a struct that implements the [`output.Output` +interface](https://pkg.go.dev/go.k6.io/k6/output#Output). For example: + + + +```go +package log + +import ( + "fmt" + "io" + + "go.k6.io/k6/output" + "go.k6.io/k6/stats" +) + +// Register the extension on module initialization. +func init() { + output.RegisterExtension("logger", New) +} + +// Logger writes k6 metric samples to stdout. +type Logger struct { + out io.Writer +} + +// New returns a new instance of Logger. +func New(params output.Params) (output.Output, error) { + return &Logger{params.StdOut}, nil +} + +// Description returns a short human-readable description of the output. +func (*Logger) Description() string { + return "logger" +} + +// Start initializes any state needed for the output, establishes network +// connections, etc. +func (o *Logger) Start() error { + return nil +} + +// AddMetricSamples receives metric samples from the k6 Engine as they're +// emitted and prints them to stdout. +func (l *Logger) AddMetricSamples(samples []stats.SampleContainer) { + for i := range samples { + all := samples[i].GetSamples() + for j := range all { + fmt.Fprintf(l.out, "%d %s: %f\n", all[j].Time.UnixNano(), all[j].Metric.Name, all[j].Value) + } + } +} + +// Stop finalizes any tasks in progress, closes network connections, etc. +func (*Logger) Stop() error { + return nil +} +``` + + + +Notice a couple of things: + +- The module initializer `New()` receives an instance of + [`output.Params`](https://pkg.go.dev/go.k6.io/k6/output#Params). + With this object the extension can access the output-specific configuration, + interfaces to the filesystem, synchronized stdout and stderr, and more. +- `AddMetricSamples` in this example simply writes to stdout. In a real-world + scenario this output might have to be buffered and flushed periodically to avoid + memory leaks. Below we'll discuss some helpers you can use for that. + + +#### Additional features + +- Output structs can optionally implement additional interfaces that allows them to + [receive thresholds](https://pkg.go.dev/go.k6.io/k6/output#WithThresholds), + [test run status updates](https://pkg.go.dev/go.k6.io/k6/output#WithRunStatusUpdates) + or [interrupt a test run](https://pkg.go.dev/go.k6.io/k6/output#WithTestRunStop). +- Because output implementations typically need to process large amounts of data that + k6 produces and dispatch it to another system, we've provided a couple of helper + structs you can use in your extensions: + [`output.SampleBuffer`](https://pkg.go.dev/go.k6.io/k6/output#SampleBuffer) + is a thread-safe buffer for metric samples to help with memory management and + [`output.PeriodicFlusher`](https://pkg.go.dev/go.k6.io/k6/output#PeriodicFlusher) + will periodically run a function which is useful for flushing or dispatching the + buffered samples. + For usage examples see the [`statsd` output](https://pkg.go.dev/go.k6.io/k6/output/statsd). + + +## Supported modes of execution + +xk6 and the examples above work great for local execution, i.e. running a +script with a binary built by xk6 on your own infrastructure. Custom k6 binaries are +also supported by the [k6 Kubernetes operator](https://github.com/grafana/k6-operator#using-extensions). + +They're currently not supported in [k6 Cloud](https://k6.io/docs/cloud/), but adding support +is on our roadmap. However, it is possible to run a binary built with xk6 and send +metrics produced by extension code to the Cloud using the +[`cloud` output](https://k6.io/docs/results-visualization/cloud/).