Performance runner in js #146129

lizozom · 2022-11-23T11:06:33Z

Summary

Rewrite the performance journey runner using TypeScript, to avoid dangling ES\node processes during test execution.

Results are stable with this runner, as verified on CI

.buildkite/scripts/steps/functional/performance_playwright.sh

src/dev/performance/run_performance_cli.ts

spalger · 2022-11-23T14:54:07Z

You might want to try pushing this branch to the elastic/kibana repo and trying a couple executions of the performance job before merging

Co-authored-by: Spencer <[email protected]>

lizozom · 2022-11-23T15:19:33Z

@spalger, already running on the debug pipeline.
I will wait for 24 hours of data before merging.

…into performance-runner-in-js

…-fix'

…into performance-runner-in-js

…ance-runner-in-js

lizozom · 2022-11-29T10:33:17Z

@spalger figures that the worker was to blame for the instability.
This PR is perfectly stable on our ubuntu agent.
We'll investigate the issues with the other worker separately.
I think we should be good to go to merge.

src/dev/performance/run_performance_cli.ts

kibana-ci · 2022-11-29T12:21:12Z

💚 Build Succeeded

Buildkite Build
Commit: e4b9a96

Metrics [docs]

Unknown metric groups

ESLint disabled in files

id	before	after	diff
`osquery`	1	2	+1

ESLint disabled line counts

id	before	after	diff
`enterpriseSearch`	19	21	+2
`fleet`	59	65	+6
`osquery`	109	115	+6
`securitySolution`	442	448	+6
total			+20

Total ESLint disabled count

id	before	after	diff
`enterpriseSearch`	20	22	+2
`fleet`	68	74	+6
`osquery`	110	117	+7
`securitySolution`	519	525	+6
total			+21

History

💛 Build #90726 was flaky 8c0604b
💚 Build #90640 succeeded 5b533f4
💛 Build #90576 was flaky 04d3038
💚 Build #90365 succeeded 163701a
💚 Build #90292 succeeded b950a0d
💚 Build #90260 succeeded 45efac4

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

dmlemeshko · 2022-11-29T14:51:23Z

src/dev/performance/run_performance_cli.ts

+    for (const journey of journeys) {
+      await startEs();
+      await runWarmup(journey);
+      await runTest(journey);
+      await procRunner.stop('es');
+    }


Do I understand the flow correctly: if runWarmup throws exception for journey 2 (assuming we run 5 total) execution will be stopped with journey 3-5 not run and ES not stopped?

I think currently we do run all the journeys, even if 1 fails (we collect run statuses)

You are right!
Could we merge this one and update in a follow up?
I have multiple additional PRs depending on this one :-)

lizozom · 2022-11-29T17:43:26Z

@dmlemeshko why don't you try re-writing the scalability runner in a similar way and then we can check if the behavior of the worker was related to zombie processes?

## Summary Follow up to #146129 See [comment](#146129 (comment)) for more details. ### Checklist Delete any items that are not applicable to this PR. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/)) - [ ] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US)) - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server)) - [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers) ### Risk Matrix Delete this section if it is not applicable to this PR. Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release. When forming the risk matrix, consider some of the following examples and how they may potentially impact the change: | Risk | Probability | Severity | Mitigation/Notes | |---------------------------|-------------|----------|-------------------------| | Multiple Spaces—unexpected behavior in non-default Kibana Space. | Low | High | Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces. | | Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks. | High | Low | Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure. | | Code should gracefully handle cases when feature X or plugin Y are disabled. | Medium | High | Unit tests will verify that any feature flag or plugin combination still results in our service operational. | | [See more potential risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) | ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

## Summary This PR is the follow-up to #147002 and #146129 and makes few changes to make both performance scripts expose very similar cli and allow run `run_performance` locally for debug purpose. - to run a single test locally against source: - single user journey: `node scripts/run_performance.js --journey-path x-pack/performance/journeys/login.ts` - scalability journey (auto-generated): `node scripts/run_scalability.js --journey-path target/scalability_traces/kibana/login-0184f19e-0903-450d-884d-436d737a3abe.json` `skip-warmup` flag to avoid journey warmup runs for performance data set extraction (we don't need to run journey twice while interested in APM traces only) PR also updates pipeline scripts with new changes

lizozom added 5 commits November 23, 2022 13:13

JS RUNNER

40ca148

change log line

9298153

process write

292941f

re-enable stuff

69c595c

run test twice warmup

cae3d71

lizozom force-pushed the performance-runner-in-js branch from 3fc5174 to cae3d71 Compare November 23, 2022 11:15

remove file

45efac4

lizozom commented Nov 23, 2022

View reviewed changes

.buildkite/scripts/steps/functional/performance_playwright.sh Outdated Show resolved Hide resolved

lizozom marked this pull request as ready for review November 23, 2022 11:18

lizozom requested a review from a team as a code owner November 23, 2022 11:18

lizozom added wg:performance Work tracked by the performance workgroup Team:Performance v8.7.0 release_note:skip Skip the PR/issue when compiling release notes labels Nov 23, 2022

Update run_performance_cli.ts

b950a0d

spalger reviewed Nov 23, 2022

View reviewed changes

src/dev/performance/run_performance_cli.ts Outdated Show resolved Hide resolved

spalger reviewed Nov 23, 2022

View reviewed changes

src/dev/performance/run_performance_cli.ts Outdated Show resolved Hide resolved

Update src/dev/performance/run_performance_cli.ts

ecfb7ad

Co-authored-by: Spencer <[email protected]>

lizozom and others added 10 commits November 23, 2022 17:22

review #1

163701a

env changes

9af57d0

Update journey_ftr_config.ts

04d3038

Merge branch 'performance-runner-in-js' of github.com:lizozom/kibana …

026fe7d

…into performance-runner-in-js

comment

d0e7193

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

5b533f4

…-fix'

APM was enabled anyway and it's useful to have it on always

885d0d8

Merge branch 'performance-runner-in-js' of github.com:lizozom/kibana …

8c0604b

…into performance-runner-in-js

Merge branch 'main' of https://github.com/elastic/kibana into perform…

bf70ad9

…ance-runner-in-js

remove unnecessary env var change

d2c7f91

lizozom requested a review from spalger November 29, 2022 10:33

dmlemeshko reviewed Nov 29, 2022

View reviewed changes

src/dev/performance/run_performance_cli.ts Outdated Show resolved Hide resolved

Update run_performance_cli.ts

50740c5

dmlemeshko reviewed Nov 29, 2022

View reviewed changes

src/dev/performance/run_performance_cli.ts Outdated Show resolved Hide resolved

Update run_performance_cli.ts

e4b9a96

lizozom requested a review from dmlemeshko November 29, 2022 11:04

lizozom linked an issue Nov 29, 2022 that may be closed by this pull request

Create a TypeScript runner for performance tests #146545

Closed

dmlemeshko reviewed Nov 29, 2022

View reviewed changes

spalger approved these changes Nov 29, 2022

View reviewed changes

lizozom merged commit 783ea14 into elastic:main Nov 29, 2022

kibanamachine added the backport:skip This commit does not require backporting label Nov 29, 2022

lizozom mentioned this pull request Nov 29, 2022

Run all journeys before failing a performance run #146605

Merged

9 tasks

dmlemeshko mentioned this pull request Dec 8, 2022

[performance] unify scripts #147202

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Performance runner in js #146129

Performance runner in js #146129

lizozom commented Nov 23, 2022 •

edited

Loading

spalger commented Nov 23, 2022

lizozom commented Nov 23, 2022

lizozom commented Nov 29, 2022

kibana-ci commented Nov 29, 2022

ESLint disabled in files

ESLint disabled line counts

Total ESLint disabled count

dmlemeshko Nov 29, 2022

lizozom Nov 29, 2022

lizozom commented Nov 29, 2022

Performance runner in js #146129

Performance runner in js #146129

Conversation

lizozom commented Nov 23, 2022 • edited Loading

Summary

spalger commented Nov 23, 2022

lizozom commented Nov 23, 2022

lizozom commented Nov 29, 2022

kibana-ci commented Nov 29, 2022

💚 Build Succeeded

Metrics [docs]

ESLint disabled in files

ESLint disabled line counts

Total ESLint disabled count

History

dmlemeshko Nov 29, 2022

Choose a reason for hiding this comment

lizozom Nov 29, 2022

Choose a reason for hiding this comment

lizozom commented Nov 29, 2022

lizozom commented Nov 23, 2022 •

edited

Loading