Release v0.31.0

lib/consts/consts.go

@@ -28,7 +28,7 @@ import (
 )
 
 // Version contains the current semantic version of k6.
-const Version = "0.30.0"
+const Version = "0.31.0"
 
 // VersionDetails can be set externally as part of the build process
 var VersionDetails = "" // nolint:gochecknoglobals

release notes/v0.31.0.md

@@ -0,0 +1,89 @@
+k6 v0.31.0 is here! :tada: It's a smaller release with some significant performance improvements, a new `http_req_failed` metric and changes to the output subsystem that enable output extensions with [xk6](https://github.com/k6io/xk6)!
+
+## New features
+
+### Output cleanup and extensions ([#1874](https://github.com/loadimpact/k6/pull/1874))
+
+The state of k6's output packages has been a development pain point for a long time, which made it difficult to add new outputs in a consistent way. In the refactor done in v0.31.0, this has been mostly addressed and outputs now implement a simpler and cleaner `Output` interface.
+
+In addition to this, it is now possible to implement custom k6 output extensions in Go with [xk6](https://github.com/k6io/xk6)! This is very useful if you use a system that's currently not supported by [the built-in outputs](https://k6.io/docs/getting-started/results-output), or need some custom handling of the metrics k6 produces.
+
+Writing output extensions is done very similarly to [how JS module extensions are currently written](https://k6.io/blog/extending-k6-with-xk6), though instead of calling `js/modules.Register()`, you should implement the new [`Output` interface](https://github.com/loadimpact/k6/blob/2036baeaa83d76ce17aae95fef7e78e3ef24c735/output/types.go#L51-L74) and call [`output.RegisterExtension()`](https://github.com/loadimpact/k6/blob/2036baeaa83d76ce17aae95fef7e78e3ef24c735/output/extensions.go#L47) with your constructor.
+
+We are working on the proper documentation for this, as well as the overdue xk6 documentation about JS extensions, so keep a lookout for those on [k6.io/docs](https://k6.io/docs/).
+
+
+### Marking requests as failed ([#1856](https://github.com/loadimpact/k6/pull/1856))
+
+It's now possible to declare expected HTTP response statuses for either the entire test or for individual HTTP requests, and k6 will emit a new `http_req_failed` metric as well as tag HTTP metrics with `expected_response: <bool>`. By default, k6 will now fail requests that return HTTP 4xx/5xx response codes.
+
+For example:
+
+```javascript
+import http from 'k6/http';
+
+// Set expected statuses globally for all requests.
+http.setResponseCallback(http.expectedStatuses({min: 200, max: 399}, 418));
+
+export default function () {
+  // This request will be marked as failed.
+  http.get('https://httpbin.test.k6.io/status/400');
+  // This request will be considered as "passed" because of the responseCallback override.
+  http.get('https://httpbin.test.k6.io/status/400', { responseCallback: http.expectedStatuses(400) });
+}
+```
+
+Running this script will produce a summary like:
+
+```
+http_req_duration..............: avg=204.57ms min=203.31ms med=204.57ms max=205.82ms p(90)=205.57ms p(95)=205.7ms
+  { expected_response:true }...: avg=203.31ms min=203.31ms med=203.31ms max=203.31ms p(90)=203.31ms p(95)=203.31ms
+http_req_failed................: 50.00% ✓ 1   ✗ 1
+```
+
+Note the new `http_req_duration` sub-metric for expected responses only, and the new `http_req_failed` `Rate` metric. This new metric and metric tag have many potential use cases, and one of the most important ones is the ability to set better [thresholds](https://k6.io/docs/using-k6/thresholds). For example:
+- `'http_req_failed': ['rate<0.1']`, i.e. fail the test if more than 10% of requests fail.
+- `'http_req_duration{expected_response:true}': ['p(95)<300', 'p(99.9)<500']` - fail the test if the the 95th percentile HTTP request duration is above 300ms or the 99.9th percentile is above 500ms; specifying `expected_response:true` here may be important, because a lot of times failed requests may return more quickly than normal ones, thus skewing the results and wrongly satisfying the threshold.
+
+If the response callback is not specified, the default expected statuses will be `{min: 200, max: 399}`. The previous behavior of not emitting anything can be achieved by setting the callback to `null`, i.e. `http.setResponseCallback(null)`. Additionally, the `expected_response` tag can be disabled by removing it from the default list of [system tags](https://k6.io/docs/using-k6/options#system-tags), e.g. `k6 run --system-tags 'proto,subproto,status,method,url,name,group,check,error,error_code,tls_version,scenario,service'`.
+
+The `http.setResponseCallback()` is planned to allow arbitrary JS functions to process responses in the future, but for now only the `http.expectedStatuses()` callback is supported.
+
+
+## Other enhancements and UX improvements
+
+- JS: Because of the awesome improvements to [goja](https://github.com/dop251/goja), the JS runtime k6 uses, it's no longer necessary for k6 to load [core.js](https://github.com/zloirock/core-js) to polyfill missing JS features when using the default [`--compatibility-mode=extended`](https://k6.io/docs/using-k6/javascript-compatibility-mode). So in v0.31.0 core.js has been dropped entirely, yielding some significant CPU and memory usage improvements. The actual numbers will depend on the use case, but for simple tests users can expect a memory drop of about 2MB per VU (from ~2.7MB to ~600KB), and a slight CPU decrease of about 5-10%. For more complex tests with a lot of JS code this benefit won't be as pronounced. Another benefit of this change is that initializing VUs and starting a test is substantially faster than before! ([#1824](https://github.com/loadimpact/k6/pull/1824))
+- JS: Also because of goja improvements, some unused Babel plugins were disabled which should have minor performance benefits as well. ([#1822](https://github.com/loadimpact/k6/pull/1822))
+- JS: Expanded `ArrayBuffer` support in most internal modules, so now you can pass `ArrayBuffer` to `http.file()`, in `k6/encoding` and `k6/crypto` functions. This makes working with binary files more efficient as it doesn't require string translations. In upcoming versions we plan to expand this to the WebSocket module, as well as make some potentially breaking changes for APIs that currently return an array of integers or string (see the details in the Breaking Changes announcement below). ([#1800](https://github.com/loadimpact/k6/pull/1800))
+- The Docker image base was updated to Alpine 3.13. Thanks @andriisoldatenko! ([#1821](https://github.com/loadimpact/k6/pull/1821))
+- The Debian package now includes `ca-certificates` as a dependency. Thanks @Bablzz! ([#1854](https://github.com/loadimpact/k6/pull/1854))
+
+
+## Bugs fixed!
+
+- Execution: Aborting a test during VU initialization (e.g. with `^C`) will now properly propagate to any used outputs. ([#1869](https://github.com/loadimpact/k6/pull/1869))
+- Execution: A race condition between the Engine and the outputs' finalization code was fixed, ensuring that all metrics are properly emitted before exiting. ([#1869](https://github.com/loadimpact/k6/pull/1869))
+- Execution: Another race condition in the Engine was fixed, which may have resulted in the end-of-test summary missing some of the last test metric data. ([#1888](https://github.com/loadimpact/k6/pull/1888))
+- Cloud: the test name is now properly validated and will raise an error if not set via the `ext.loadimpact.name` JS option or config, or the `K6_CLOUD_NAME` environment variable. ([#1870](https://github.com/loadimpact/k6/pull/1870))
+- JS: Babel is now also run on compilation errors, which improves support of some obscure language features. ([#1861](https://github.com/loadimpact/k6/pull/1861))
+- JS: `SharedArray` introduced in v0.30.0 can now be iterated with [`forEach`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/forEach). ([#1848](https://github.com/loadimpact/k6/pull/1848))
+
+
+## Internals
+
+- JS: `SharedArray` was rewritten using `goja.DynamicArray` making it more performant and easier to reason about. ([#1848](https://github.com/loadimpact/k6/pull/1848))
+- JS: Some TC39 tests for unsupported features were disabled, improving the runtime of the test suite. ([#1816](https://github.com/loadimpact/k6/pull/1816))
+- CI: Some more tests were enabled on Windows. ([#1855](https://github.com/loadimpact/k6/pull/1855))
+
+
+## Breaking changes
+
+- JS: While we don't expect the core.js removal and Babel changes to impact the vast majority of users, those were substantial changes in how k6 interprets JS and a minority of users might experience issues with their tests. Please report any unexpected JavaScript errors by creating a GitHub issue. In particular `Promise` is now `undefined`, and some unused Babel plugins like `transform-es2015-for-of` and `transform-regenerator` were also removed. This means that some workarounds like the ones mentioned [here](https://github.com/loadimpact/k6/issues/779#issuecomment-674311032) and [here](https://stackoverflow.com/a/65849645/96213) also won't work as is and will need additional polyfills and plugins to work properly.
+
+## Planned future breaking changes
+
+The following are not breaking changes in _this_ release, but we'd like to announce them so users can prepare for them in upcoming releases (likely k6 v0.32.0).
+
+- JS: The `ArrayBuffer` changes in this release are backwards compatible and shouldn't cause any issues, but in v0.32.0 some JS APIs that currently return an array of integers or string for binary data will return `ArrayBuffer` instead. This is the case for `open()` when used with the `'b'` argument, response bodies for requests that specify `responseType: 'binary'`, `crypto.randomBytes()`, `hasher.digest('binary')`, and `encoding.b64decode()`. `Response.json()` and `Response.html()` will also probably stop working when used with requests that specify `responseType: 'binary'`. These changes shouldn't be a problem for most users that were simply using these values to pass them to other internal modules (e.g. opening a binary file and passing it to `http.post()`), but if the scripts modified the binary data or depended on the current array of integers or string values they will need to be adapted to use [typed arrays](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Typed_arrays) instead. You can follow the discussion in [PR #1841](https://github.com/loadimpact/k6/pull/1841) and [issue #1020](https://github.com/loadimpact/k6/issues/1020).
+- As part of the rebranding of Load Impact to k6, the k6 GitHub repository will be moved from https://github.com/loadimpact/k6 to https://github.com/k6io/k6 . Additionally because of Go's usage of URLs in package imports, the URL will be changed throughout the codebase. Since GitHub will maintain a redirect from the old location we don't expect this to impact a lot of users, or even external k6 contributors and xk6 developers, but be aware that the new URL should be used moving forward.
+- Because of the [sunsetting of Bintray](https://jfrog.com/blog/into-the-sunset-bintray-jcenter-gocenter-and-chartcenter/), the DEB, RPM, MSI and Chocolatey package repositories currently hosted on Bintray will be moved to a self-hosted solution sometime in the upcoming weeks. We'll communicate these changes via our [blog](https://k6.io/blog/) as well as the [official documentation](https://k6.io/docs/getting-started/installation).