Skip to content

Commit

Permalink
Add v0.27.0 release notes (#1507)
Browse files Browse the repository at this point in the history 
* WIP Add v0.27.0 release notes

* Address some PR comments

* Address more PR comments

* Reference bugs fixed by #1007

* Actually, remove mention of bugs squashed by #1007

1170 is not a bug, and the other two are already mentioned.

* Apply suggestions from code review

Co-authored-by: na-- <[email protected]>

* Minor cleanup, address some PR comments

* Add scenario example to v0.27.0 release notes

* Add links to issues and PRs, remove 'closes'

Resolves #1507 (comment)

* Mention WS hanging fix in release notes

Resolves #1507 (comment)

* Update scenarios docs link

* Link to 'Breaking changes' section

* Move gracefulStop breaking change higher up

See #1507 (comment)

* Mention new dropped_iterations metric in release notes

Resolves https://github.com/loadimpact/k6/pull/1507/files#r452789367

* Mention more resolved issues in the v0.27.0 release notes

* Apply suggestions from code review

Co-authored-by: na-- <[email protected]>

* Update release notes/v0.27.0.md

Co-authored-by: na-- <[email protected]>

* Apply suggestions from code review

Co-authored-by: na-- <[email protected]>

Co-authored-by: na-- <[email protected]>
  • Loading branch information
@na--
Ivan Mirić and na-- authored Jul 14, 2020
1 parent dc31776 commit b26d97a
Showing 1 changed file with 168 additions and 0 deletions.
168 changes: 168 additions & 0 deletions release notes/v0.27.0.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,168 @@
k6 v0.27.0 is here! :tada:

This is a milestone release containing a major overhaul to the execution subsystem of k6, along with many improvements and bug fixes.

## New features and enhancements!

### New execution engine ([#1007](https://github.com/loadimpact/k6/pull/1007))

After 1.5 years in the making, the k6 team is proud to release the first public version of the new execution engine, offering users new ways of modeling advanced load testing scenarios that can more closely represent real-world traffic patterns.

These new scenarios are entirely optional, and the vast majority of existing k6 scripts and options should continue to work the same as before. There are several minor [breaking changes](#breaking-changes) and fixes of previously undefined behavior, but please [create a new issue](https://github.com/loadimpact/k6/issues/new?labels=bug&template=bug_report.md) if you find some issue we haven't explicitly noted as a breaking change.

See the [documentation](https://k6.io/docs/using-k6/scenarios) for details and examples, or keep reading for the summary.

#### New executors

Some of the currently possible script execution patterns were formalized into standalone executors:

- `shared-iterations`: a fixed number of iterations are "shared" by all VUs, and the test ends once all iterations are executed. This executor is equivalent to the global `vus` and `iterations` (plus optional `duration`) options.
- `constant-vus`: a fixed number of VUs execute as many iterations as possible for a specified amount of time. This executor is equivalent to the global `vus` and `duration` options.
- `ramping-vus`: a variable number of VUs execute as many iterations as possible for a specified amount of time. This executor is equivalent to the global `stages` option.
- `externally-controlled`: control and scale execution at runtime via k6's REST API or the CLI.

You'd still be able to use the global `vus`, `iterations`, `duration`, and `stages` options, they are not deprecated! They are just transparently converted to one of the above executors underneath the hood. And if your test run needs just a single, simple scenario, you may never need to use more than these shortcut options. For more complicated use cases however, you can now fine-tune any of these executors with additional options, and use multiple different executors in the same test run, via the new `scenarios` option, described below.

Additionally, besides the 4 "old" executor types, there are 3 new executors, added to support some of the most frequently requested load testing scenarios that were previously difficult or impossible to model in k6:

- `per-vu-iterations`: each VU executes a fixed number of iterations ([#381](https://github.com/loadimpact/k6/issues/381)).
- `constant-arrival-rate`: iterations are started at a specified fixed rate, for a specified duration. This allows k6 to dynamically change the amount of active VUs during a test run, to achieve the specified amount of iterations per period. This can be very useful for a more accurate representation of RPS (requests per second), for example. See [#550](https://github.com/loadimpact/k6/issues/550) for details.
- `ramping-arrival-rate`: a variable number of iterations are executed in a specified period of time. This is similar to the ramping VUs executor, but instead of specifying how many VUs should loop through the script at any given point in time, the iterations per second k6 should execute at that point in time can be specified.

It's important to also note that all of these executors, except the `externally-controlled` one, can be used both in local k6 execution with `k6 run`, and in the distributed cloud execution with `k6 cloud`. This even includes "old" executors that were previously unavailable in the cloud, like the `shared-iterations` one. Now, you can execute something like `k6 cloud --iterations 10000 --vus 100 script.js` without any issues.

#### Execution scenarios and executor options

Multiple execution scenarios can now be configured in a single test run via the new `scenarios` option. These scenarios can run both sequentially and in parallel, and can independently execute different script functions, have different executor types and execution options, and have custom environment variables and metrics tags.

An example using 3 scenarios:

```js
import http from 'k6/http';
import { sleep } from 'k6';

export let options = {
scenarios: {
my_web_test: { // some arbitrary scenario name
executor: 'constant-vus',
vus: 50,
duration: '5m',
gracefulStop: '0s', // do not wait for iterations to finish in the end
tags: { test_type: 'website' }, // extra tags for the metrics generated by this scenario
exec: 'webtest', // the function this scenario will execute
},
my_api_test_1: {
executor: 'constant-arrival-rate',
rate: 90, timeUnit: '1m', // 90 iterations per minute, i.e. 1.5 RPS
duration: '5m',
preAllocatedVUs: 10, // the size of the VU (i.e. worker) pool for this scenario
maxVUs: 10, // we don't want to allocate more VUs mid-test in this scenario

tags: { test_type: 'api' }, // different extra metric tags for this scenario
env: { MY_CROC_ID: '1' }, // and we can specify extra environment variables as well!
exec: 'apitest', // this scenario is executing different code than the one above!
},
my_api_test_2: {
executor: 'ramping-arrival-rate',
startTime: '30s', // the ramping API test starts a little later
startRate: 50, timeUnit: '1s', // we start at 50 iterations per second
stages: [
{ target: 200, duration: '30s' }, // go from 50 to 200 iters/s in the first 30 seconds
{ target: 200, duration: '3m30s' }, // hold at 200 iters/s for 3.5 minutes
{ target: 0, duration: '30s' }, // ramp down back to 0 iters/s over the last 30 second
],
preAllocatedVUs: 50, // how large the initial pool of VUs would be
maxVUs: 100, // if the preAllocatedVUs are not enough, we can initialize more

tags: { test_type: 'api' }, // different extra metric tags for this scenario
env: { MY_CROC_ID: '2' }, // same function, different environment variables
exec: 'apitest', // same function as the scenario above, but with different env vars
},
},
discardResponseBodies: true,
thresholds: {
// we can set different thresholds for the different scenarios because
// of the extra metric tags we set!
'http_req_duration{test_type:api}': ['p(95)<250', 'p(99)<350'],
'http_req_duration{test_type:website}': ['p(99)<500'],
// we can reference the scenario names as well
'http_req_duration{scenario:my_api_test_2}': ['p(99)<300'],
}
};

export function webtest() {
http.get('https://test.k6.io/contacts.php');
sleep(Math.random() * 2);
}

export function apitest() {
http.get(`https://test-api.k6.io/public/crocodiles/${__ENV.MY_CROC_ID}/`);
// no need for sleep() here, the iteration pacing will be controlled by the
// arrival-rate executors above!
}
```

As shown in the example above and the [documentation](https://k6.io/docs/using-k6/scenarios), all executors have some additional options that improve their flexibility and facilitate code reuse, especially in multi-scenario test runs:
- Each executor has a `startTime` property, which defines at what time, relative to the beginning of the whole test run, the scenario will start being executed.
- Executors have a new `gracefulStop` property that allows for iterations to complete gracefully for some amount of time after the normal executor duration is over ([#879](https://github.com/loadimpact/k6/issues/879), [#1033](https://github.com/loadimpact/k6/issues/1033)). The `ramping-vus` executor additionally also has `gracefulRampDown`, to give iterations time to finish when VUs are ramped down. The default value for both options is `30s`, so it's a slight breaking change, but the old behavior of immediately interrupting iterations can easily be restored by setting these options to `0s`.
- Different executors can execute different functions other than the `default` exported one. This can be specified by the `exec` option in each `scenarios` config, and allows for more flexibility in organizing your tests, easier code reuse, building test suites, etc.
- To allow for even greater script flexibility and code reuse, you can specify different environment variables and tags in each scenario, via the new `env` and `tags` executor options respectively.
- k6 may now emit a new `dropped_iterations` metric in the `shared-iterations`, `per-vu-iterations`, `constant-arrival-rate` and `ramping-arrival-rate` executors; this is done if it can't run an iteration on time, depending on the configured rates (for the arrival-rate executors) or scenario `maxDuration` (for the iteration-based executors), so it's generally a sign of a poor config or an overloaded system under test ([#1529](https://github.com/loadimpact/k6/pull/1529)).


We've also introduced new `--execution-segment` and `--execution-segment-sequence` options, which allow for relatively easy partitioning of test runs across multiple k6 instances. Initially this applies to the test execution (all new executor types are supported!), but opens the door to test data partitioning, an often requested feature. See [#997](https://github.com/loadimpact/k6/issues/997) for more details.

## UX

- CLI: There are separate descriptions and real-time thread-safe progress bars for each individual executor.
- CLI: Improve module import error message ([#1439](https://github.com/loadimpact/k6/issues/1439)).
- JS: The `__VU` variable is now available in the script init context, allowing easier splitting of test input data per VU and reducing RAM usage ([#889](https://github.com/loadimpact/k6/issues/889)).
- A new method to stop engine execution via the REST API ([#1352](https://github.com/loadimpact/k6/issues/1352)). Thanks @hynd!

## Bugs fixed!
- CLI: Stop `--http-debug` from exiting k6 on request dump error ([#1402](https://github.com/loadimpact/k6/issues/1402)). Thanks @berndhartzer!
- CLI: JSON output is now less noisy ([#1469](https://github.com/loadimpact/k6/issues/1469)). Thanks @michiboo!
- CLI: k6 doesn't exit when using `iterations` with `stages` ([#812](https://github.com/loadimpact/k6/issues/812)).
- CLI: Mismatch in check counts in the end-of-test summary ([#1033](https://github.com/loadimpact/k6/issues/1033)).
- Config: Better validation of `stages` ([#875](https://github.com/loadimpact/k6/issues/875)).
- JS: Rare panics in goja ([#867](https://github.com/loadimpact/k6/issues/867)).
- HTTP: Fix request timeout and wrong context used for pushing metrics ([#1260](https://github.com/loadimpact/k6/issues/1260)).
- Execution: Fix sometimes skipping iterations with a `context cancelled` error when rapidly ramping up and down ([#1283](https://github.com/loadimpact/k6/issues/1283)).
- WebSockets: Fix possible connection hanging when the main context is cancelled ([#1260](https://github.com/loadimpact/k6/issues/1542)).
- WebSockets: Avoid leaking goroutines ([#1513](https://github.com/loadimpact/k6/issues/1513)).
- WebSockets: Emit WS metrics as soon as possible instead of when the connection is closed ([#885](https://github.com/loadimpact/k6/issues/885)).


## Internals

- As mentioned above, [#1007](https://github.com/loadimpact/k6/issues/1007) was almost a complete re-write of the execution scheduling parts of k6. This involved deep architectural changes in how test runs are executed and improved overall code coverage by around 2%.
- Switched to Go 1.14 for building and testing k6, bringing some fixes and performance improvements.
- WebSockets: Updated gorilla/websocket library bringing minor performance improvements ([#1515](https://github.com/loadimpact/k6/pull/1515)).
- Code cleanup and formatting. Thanks @thinkerou!


## Breaking changes

- Execution config options (`scenarios`, `stages`, `iterations`, `duration`) from "upper" config layers overwrite execution options from "lower" (i.e. CLI flags > environment variables > JS options > JSON options) config layers. For example, the `--iterations` CLI flag will overwrite any execution options set as environment variables (e.g. `K6_DURATION`, `K6_STAGES`, etc.) or script options (`stages: [ /* ... */]`, `scenarios: { /* ... */ }`, etc. ).

- Previously, if the `iterations` and `vus` script options were specified, but `duration` was not, the script could have ran practically indefinitely, given a sufficiently large number or length of the used iterations. There was no implicit or explicit time limit, one of the reasons this execution pattern was not allowed in `k6 cloud` test runs before. From k6 v0.27.0, by default, if the specified iterations haven't finished, these scripts will abort after 10 minutes, the default `maxDuration` value of `shared-iterations` executors. This default value can easily be changed by either setting the `maxDuration` option in the corresponding `scenarios` entry, or, if just the execution shortcuts were used to run the script, by setting the `duration` / `--duration` / `K6_DURATION` script option.

- Previously, all iterations were interruptible - as soon as the specified `duration` expired, or when VUs were ramped down in a stage, any running iterations were interrupted. Now all executors besides the `externally-controlled` one have a `gracefulStop` period of `30s` by default ([#898](https://github.com/loadimpact/k6/issues/898)). Additionally, the `ramping-vus` executor has a `gracefulRampDown` parameter that configures the ramp-down grace period. For those periods, no new iterations will be started by the executors, but any currently running iterations will be allowed up to the specified periods to finish their execution.

- Using different execution config options on the same level is now a configuration conflict error and will abort the script. For example, executing `k6 run --duration 10s --stages 5s:20 script.js` won't work ([#812](https://github.com/loadimpact/k6/issues/812)). The only exception is combining `duration` and `iterations`, which will result in a `shared-iterations` executor with the specified non-default `maxDuration` ([#1058](https://github.com/loadimpact/k6/issues/1058)).

- The k6 REST API for controlling script execution (i.e. the `k6 pause`, `k6 scale` commands) now only works when a `externally-controlled` executor is configured in the `scenarios` config. The initial pausing of a test (i.e. `k6 run --paused script.js`) still works with all executor types, but once the test is started with `k6 resume` (or the corresponding REST API call), it can't be paused again unless only the `externally-controlled` executor is used.

- Previously, running a script with `k6 run --paused script.js` would have still executed the script's `setup()` function (if it was present and wasn't explicitly disabled with `--no-setup`) and paused immediately after. Now, k6 will pause before it executes `setup()`.

- Previously, if you ramped down and ramped up VUs via `stages`, the `__VU` global variables would have been incremented on the ramp-ups. This will no longer happen, the max value of `__VU` across a test run will never exceed the number of initialized VUs.

- The `vusMax` / `K6_VUS_MAX` / ` -m` / `--max` option is deprecated - it was previously used for the control of the initialized VUs by the REST API. Since that has now been restricted to the `externally-controlled` executor, the equivalent option there is called `maxVUs`.

- Tests with infinite duration are now only possible via the `externally-controlled` executor.

- k6 will now exit with an error if `--address` is specified but the API server is unable to start. Previously this would've resulted in a warning message.

- The format returned by `Date.prototype.toUTCString()` was changed from `Thu Jan 01 1970 00:00:00 GMT+0000 (UTC)` to `Thu, 01 Jan 1970 00:00:00 GMT`. This is a fix that aligns it with the ECMAScript spec. See https://github.com/dop251/goja/issues/119 .

- The default `setupTimeout` and `teardownTimeout` values were changed from 10s to 60s ([#1356](https://github.com/loadimpact/k6/pull/1356)).

0 comments on commit b26d97a

Please sign in to comment.