Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Race between the outputs' and the engine's end-of-test finalization #1606

Closed
na-- opened this issue Aug 25, 2020 · 1 comment · Fixed by #1869
Closed

Race between the outputs' and the engine's end-of-test finalization #1606

na-- opened this issue Aug 25, 2020 · 1 comment · Fixed by #1869
Assignees
Milestone

Comments

@na--
Copy link
Member

na-- commented Aug 25, 2020

If you have a script like that:

import { Counter } from 'k6/metrics';
import http from 'k6/http';
import { sleep } from 'k6';

var teardownCounter = new Counter('teardown_counter');

export const options = {
    vus: 1,
    iterations: 1,
};

export function teardown() {
    http.get('https://test.k6.io/?stage=teardown-start');
    teardownCounter.add(10);
    sleep(8);
    http.get('https://test.k6.io/?stage=teardown-middle');
    teardownCounter.add(10);
    sleep(8);
    http.get('https://test.k6.io/?stage=teardown-end');
    teardownCounter.add(10);
}

export default function () {
    http.get('https://test.k6.io/?stage=default');
}

some times, k6 won't emit the metrics for https://test.k6.io/?stage=teardown-end or the last teardown_counter=10 count. This seems to happen because the Engine's finalization code (pushing its last buffered metrics) and the outputs' finalization code (pushing their last metrics) are triggered by cancelling the same context...

They both output as:
https://github.com/loadimpact/k6/blob/c68a4eb450aa0a806b16bef3206ba2163f05e2a7/cmd/run.go#L154-L169
https://github.com/loadimpact/k6/blob/c68a4eb450aa0a806b16bef3206ba2163f05e2a7/cmd/run.go#L268
https://github.com/loadimpact/k6/blob/c68a4eb450aa0a806b16bef3206ba2163f05e2a7/core/engine.go#L156

Here's the output path:
https://github.com/loadimpact/k6/blob/c68a4eb450aa0a806b16bef3206ba2163f05e2a7/core/engine.go#L170-L176
https://github.com/loadimpact/k6/blob/c68a4eb450aa0a806b16bef3206ba2163f05e2a7/stats/cloud/collector.go#L218-L220
https://github.com/loadimpact/k6/blob/c68a4eb450aa0a806b16bef3206ba2163f05e2a7/stats/cloud/collector.go#L261-L263

Here's the Engine's:
https://github.com/loadimpact/k6/blob/c68a4eb450aa0a806b16bef3206ba2163f05e2a7/core/engine.go#L178-L183
https://github.com/loadimpact/k6/blob/c68a4eb450aa0a806b16bef3206ba2163f05e2a7/core/engine.go#L295-L297
https://github.com/loadimpact/k6/blob/c68a4eb450aa0a806b16bef3206ba2163f05e2a7/core/engine.go#L252-L266

Basically, this Engine.processSamples() is racing with the output's finalization - the engine may push some metrics to the outputs after they have stopped listening for them...

@na-- na-- added the bug label Aug 25, 2020
@na-- na-- added this to the v0.28.0 milestone Aug 25, 2020
@na-- na-- changed the title Race between outputs and the engine's Race between outputs and the engine's finalization Aug 25, 2020
@na-- na-- changed the title Race between outputs and the engine's finalization Race between outputs' and the engine's end-of-test finalization Aug 25, 2020
@na-- na-- changed the title Race between outputs' and the engine's end-of-test finalization Race between the outputs' and the engine's end-of-test finalization Aug 25, 2020
@na-- na-- modified the milestones: v0.28.0, v0.29.0 Sep 9, 2020
@na-- na-- modified the milestones: v0.29.0, v0.30.0 Nov 3, 2020
@na-- na-- added the high prio label Nov 26, 2020
@na--
Copy link
Member Author

na-- commented Nov 26, 2020

This is actually a big problem, since k6 run --out cloud might exit before it properly calls the TestFinished cloud API, making the test time out, since for the cloud it seems like k6 just disappears 😱 https://github.com/loadimpact/k6/blob/62665cfa87899ae210578276d7e16b4f4dd8fd7c/stats/cloud/collector.go#L636-L646

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant