New executors

[na--]

This pull request totally rewrites the old local executor and a bunch of other things. It builds on #913 and introduces quite a lot of new functionality, but it also has some minor breaking changes.

New features and changes

New executors

The biggest change in this PR is the introduction of 7 distinct new VU executors:

• shared-iterations, a.k.a. iterations + vus.
• constant-vus, a.k.a. duration + vus.
• ramping-vus, a.k.a. stages + vus.
• externally-controlled is similar to ramping-vus, but the number of initialized and active VUs at any given time can be controlled by the k6 REST API - it's the only executor that could be paused or controlled while it's running (closes Rare panic(s) in goja #867, closes K6 scale not works with stage #1028).
• per-vu-iterations is а new feature that allows each configured VU to execute an exact number of iterations; the shared iterations have caused confusion before and there are legitimate reasons people want an equal number of iterations for each VU (closes Make iterations a per-VU parameter? #381, Websocket loop executing only once. #663, Websocket requests are not being executed concurrently #627, iterations does not work when vus is set #378, and probably others).
• constant-arrival-rate executes a constant number of iterations per second (closes Arrival-rate based VU executor #550).
• ramping-arrival-rate executes a variable number of iterations per second, specified as ramp-ups or ramp-downs stages, similar to how the ramping-vus is configured.

These executors can be configured via the new execution JS/JSON option. For now, they aren't directly configurable via CLI flags or environment variables.

However, most of the old simple k6 uses where stages, or iterations or duration are specified in any way, including via CLI flags and env vars, are still supported. They are just transparently converted to their respective new execution executor configurations. There are some breaking changes here, described in detail in the Breaking Changes section below, but generally only when there is a conflict or ambiguity in the specified configuration.

Other new features

• Multiple different executors can be configured in a single test run, both sequentially and in parallel (closes Helper for conditional percent statements  #239).
• Each executor has a startTime property, which defines at what time relative to the test run start it will start executing.
• VUs are reused between the different configured executors when that's possible.
• Executors have new gracefulStop property that allows for iterations to complete gracefully for some amount of time after the normal executor duration is over (closes Uninterruptible iterations #879); even with gracefulStop=0, i.e. the old behavior, metrics are emitted and processed in a much saner way (closes Mismatch in check counts in the end-of-test summary #1033)
• Validation of the supplied execution configuration is very strict (closes Better validation of stages #875 and slightly helps with Configuration issues #883)
• There is a new feature called execution segments that allows tests to be easily partitioned between multiple k6 instances (implements the execution parts of Execution segments for partitioning work between k6 instances #997 and helps towards Clustering and distributed execution #140 and Differences between tests run in cloud vs run environments #864)
• There are separate descriptions and real-time thread-safe progress bars for each individual executor.
• Different executors can execute different functions than the default exported one.
• Different executors can have different environment variables.

Example of the new config

export let options = {
  scenarios: {
    stages_up_down: {
      executor: "ramping-vus",
      startVUs: 0,
      stages: [
        { duration: "10s", target: 20 },
        { duration: "10s", target: 0 },
      ],
      gracefulRampDown: "0s",
      gracefulStop: "0s",
    },
    stages_down_up: {
      executor: "ramping-vus",
      startVUs: 20,
      stages: [
        { duration: "10s", target: 0 },
        { duration: "10s", target: 20 },
      ],
      gracefulRampDown: "0s",
      gracefulStop: "0s",
    },
    const_loop_vus: {
      executor: "constant-vus",
      vus: 20,
      duration: "20s",
      gracefulStop: "0s",
    },
    constant_arr_rate: {
      executor: "constant-arrival-rate",
      rate: 20,
      timeUnit: "1s",
      duration: "10s",
      preAllocatedVUs: 10,
      maxVUs: 20,
      startTime: "20s",
      gracefulStop: "0s",
    },
    variable_arr_rate: {
      executor: "ramping-arrival-rate",
      startRate: 20,
      timeUnit: "1s",
      preAllocatedVUs: 10,
      maxVUs: 20,
      startTime: "20s",
      stages: [
        { target: 0, duration: "5s" },
        { target: 20, duration: "5s" },
      ],
      gracefulStop: "0s",
    },
    per_vu_iters: {
      executor: "per-vu-iterations",
      vus: 20,
      iterations: 10,
      startTime: "30s",
      maxDuration: "10s",
      gracefulStop: "10s",
    },
    shared_iters: {
      executor: "shared-iterations",
      vus: 20,
      iterations: 200,
      startTime: "30s",
      maxDuration: "10s",
      gracefulStop: "10s",
    },
    adjustable_at_will: {
      executor: "externally-controlled",
      vus: 5,
      maxVUs: 10,
      duration: "40s",
    },
  },
};

export default function () {
  console.log(`Start VU ${__VU} ITER ${__ITER}`);
  sleep(0.5 + Math.random() * 0.5);
  console.log(`Done VU ${__VU} ITER ${__ITER}`);
}

Breaking changes

There are a few breaking changes caused by this PR:

• Execution config options (execution, stages, iterations, duration) from "upper" config layers overwrite execution options from "lower" (i.e. CLI flags > environment variables > JS options > JSON options) config layers. For example, the --iterations CLI flag will overwrite any execution options set as environment variables (e.g. K6_DURATION, K6_STAGES, etc.) or script options (stages: [ /* ... */], execution: { /* ... */ }, etc. ).
• Using different execution config options on the same level is now a configuration conflict error and will abort the script. For example, executing k6 run --duration 10s --stages 5s:20 script.js won't work (closes Stuck in infinite loop when specifying iterations and VUs #812). The only exception is combining duration and iterations, which will result in a shared-iterations executor with the specified non-default maxDuration ( Specifying both duration and iterations is deprecated and won't be supported in the future k6 versions #1058).
• The k6 REST API for controlling script execution (i.e. the k6 pause, k6 scale commands) now only works when a externally-controlled executor is configured in the execution config. The initial pausing of a test (i.e. k6 run --paused script.js) still works with all executor types, but once the test is started with k6 resume (or the corresponding REST API call), it can't be paused again unless only the externally-controlled executor is used.
• Previously, running a script with k6 run --paused script.js would have still executed the script's setup() function (if it was present and wasn't explicitly disabled with --no-setup) and paused immediately after. Now, k6 will pause before it executes setup().
• The vusMax / K6_VUS_MAX / -m / --max option is deprecated - it was previously used for the control of the initialized VUs by the REST API. Since that has now been restricted to the externally-controlled executor, the equivalent option there is called maxVUs.
• Tests with infinite duration are now only possible via the externally-controlled executor.
• Previously, all iterations were interruptible - as soon as the specified duration expired, or when VUs were ramped down in a stage, any running iterations were interrupted. Now all executors besides the externally-controlled one have a gracefulStop period of 30s by default (closes Use duration run,The http_reqs value of K6 is different from the service real request count value. #898). Additionally, ramping-vus executor has a gracefulRampDown parameter that configures the ramp-down grace period. For those periods, no new iterations will be started by the executors, but any currently running iterations will be allowed up to the specified periods to finish their execution.

Broken things and TODOs that remain be fixed in this PR:

• updating the externally-controlled executor's parameters via the REST API
• update the description of this pull request 😄
• forward-port the fixes in Fix the execution derivation bug and work around the k6 0.24.0 wrong tar metadata #1057 and Disable the execution mismatch check, but keep the other warnings #1076
• rename things according to New executors #1007 (comment)
• support for cloud execution
• the usage report
• writing a lot more tests... this PR should hopefully significantly increase the code coverage of k6
• improving the multi-progressbar alignments and sizing
• fix the arrival-rate distribution bugs, especially when there are execution segments: New executors #1007 (comment)
• fix a bug where Ctrl+C doesn't work while the k6 scale command to the externally-controlled executor is running
• ramping-arrival-rate bug that @mstoykov found
• remove the ERRO context cancelled message error message that's sometimes shown when an iteration is interrupted
• (evaluation needed) fix that k6 inspect doesn't fail when there are validation errors in options
• update the release notes and the README
• move the execution packages out of lib, refactor core (needs further discussion, previous discussion was in New executors #1007 (comment))
• update the k6 architecture wiki page

TODOs in the code that will be handled in future pull requests, i.e. for now only issues will be created:

• handle Ctrl+C graceful shutdown + teardown execution
• have a special error code for Ctrl+C interrupts
• tag metrics from different executors differently (already described in Tag metrics from different stages #796)
• JavaScript helper functions that could be used to build this new complicated execution configuration, but with a better user experience than manually writing it as a JSON
• changes and additions to the metrics and tags k6 emits, given the new executors - this includes tagging metrics with the executor ID by default, maybe split the vus metric by executor, a new metric for insufficient VUs by the arrival-rate executors

Things that need to be done shortly before or after this PR is merged in master:

• a lot of manual testing, especially on different OSes
• update the documentation (Document how tests are executed (closed vs open system model, VUs etc.) #551)

[imiric]

😅 🎉 👏

Some flaky failures still, but we'll get them in #1357.

[mstoykov]

I still think this is way too big ... but it ain't gonna get smaller so ... merge it