Migrate Custom threshold > AVG - PCT - FIRED test to the deployment agnostic framework #195902
Conversation
maryam-saeidi
added
release_note:skip
|
We should consider this PR to be blocked by #195890, because right now, the serverless part of deployment agnostic tests is not enabled, so we wouldn't see any problems with the newly added tests. |
maryam-saeidi
added
the
Team:obs-ux-management
|
Pinging @elastic/obs-ux-management-team (Team:obs-ux-management) |
|
@maryam-saeidi if you add the label |
|
@pheyos Thanks for the heads-up; I added the PR that you shared in the TODO section of the PR description. Regarding |
You can add the label now, it will be picked up by the next CI run. And yes, you should wait for the other PR to be merged and then merge main into your PR (which will trigger another CI run and produce the serverless container if the label has been added before). |
|
Funnily enough, before creating this PR, I checked @mgiota's PR and I saw both configs were executed in CI, but it seems it was before skipping this config.
|
| .post(`/api/content_management/rpc/create`) | ||
| .post(`/api/content_management/rpc/delete`) |
|
@pheyos How can I ensure your PR changes are included in the main merge? I might have been a bit too eager when I saw your PR was merged :D |
|
@maryam-saeidi yes, it was a bit too early. When checking the details of your merge, you can see that you merged commit level 5131215 from main, which is earlier than my commit. |
|
@maryam-saeidi yes, when you click |
|
The MKI job failed: Note that the actual value doesnt have the |
|
@pheyos Thanks for the update; how can I find the related MKI job in the future? To recap, adding |
|
@maryam-saeidi there's no automation attached and the |
|
Regarding this MKI failure, I've run the test on MKI locally, and it worked as expected:
We are getting the port from this function, and I am not sure why kibana configuration has "443" as port in this MKI job. @pheyos how can I access debug logs for this MKI job, locally I got a debug log for requesting URL. |
|
@maryam-saeidi the problem with that sort of assertion is that the browser is hiding the default port. At the same time, it's not an error if the port is part of the Kibana configuration. In order to make that sort of assertions more robust, I'm introducing a new method to get the URL parts but with stripped port if it's the default one. Once #197418 is merged you should be able to use that new method in your test and get rid of the "port flakiness". |
There was a problem hiding this comment.
@maryam-saeidi Great work migrating the custom threshold rules to the deployment agnostic solution!
|
Thanks @pheyos for adding getUrlPartsWithStrippedDefaultPort. I used this function in ba30240 and testing on MKI locally works now 🎉 |
💚 Build Succeeded
Metrics [docs]Unknown metric groupsESLint disabled line counts
Total ESLint disabled count
History
|
|
Starting backport for target branches: 8.x https://github.com/elastic/kibana/actions/runs/11497372332 |
💔 All backports failed
Manual backportTo create the backport manually run: Questions ?Please refer to the Backport tool documentation |
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
…gnostic framework (elastic#195902) Part of elastic#183378 ## Summary This PR moves the first Custom threshold rule test to the deployment agnostic test. The rest will follow in a follow-up PR. ## How to run To run serverless ``` node scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts node scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts --grep="Custom Threshold rule" ``` To run stateful ``` node scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts node scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts --grep="Custom Threshold rule" ``` ### TODO - [x] elastic#195890 - [x] Test in MKI before merging #### How to run tests on MKI According to this [discussion](elastic/observability-dev#3519 (comment)), we should test in MKI environment before merging. For details on how to run in MKI, see [this section of the document](https://docs.google.com/document/d/1tiax7xoDYwFXYZjRTgVKkVMjN-SQzBWk4yn1JY6Z5UY/edit#heading=h.ece2z8p74izh) and [this readme](https://github.com/elastic/kibana/blob/main/x-pack/test_serverless/README.md#run-tests-on-mki). (cherry picked from commit 30f81ce) # Conflicts: # x-pack/test/api_integration/deployment_agnostic/apis/observability/alerting/index.ts
kibanamachine
added
the
backport missing
|
Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync. |
…ment agnostic framework (#195902) (#197632) # Backport This will backport the following commits from `main` to `8.x`: - [Migrate Custom threshold > AVG - PCT - FIRED test to the deployment agnostic framework (#195902)](#195902) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Maryam Saeidi","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-24T10:22:59Z","message":"Migrate Custom threshold > AVG - PCT - FIRED test to the deployment agnostic framework (#195902)\n\nPart of #183378\r\n\r\n## Summary\r\nThis PR moves the first Custom threshold rule test to the deployment\r\nagnostic test. The rest will follow in a follow-up PR.\r\n\r\n## How to run\r\n\r\nTo run serverless\r\n```\r\nnode scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts\r\nnode scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts --grep=\"Custom Threshold rule\"\r\n```\r\n\r\nTo run stateful\r\n```\r\nnode scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts\r\nnode scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts --grep=\"Custom Threshold rule\"\r\n```\r\n\r\n### TODO\r\n\r\n- [x] https://github.com/elastic/kibana/pull/195890\r\n- [x] Test in MKI before merging\r\n\r\n\r\n#### How to run tests on MKI\r\n\r\nAccording to this\r\n[discussion](https://github.com/elastic/observability-dev/issues/3519#issuecomment-2379914274),\r\nwe should test in MKI environment before merging. For details on how to\r\nrun in MKI, see [this section of the\r\ndocument](https://docs.google.com/document/d/1tiax7xoDYwFXYZjRTgVKkVMjN-SQzBWk4yn1JY6Z5UY/edit#heading=h.ece2z8p74izh)\r\nand [this\r\nreadme](https://github.com/elastic/kibana/blob/main/x-pack/test_serverless/README.md#run-tests-on-mki).","sha":"30f81ce4e932622e4d284b1e6af8c015c22836f5","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:prev-minor","ci:build-serverless-image","Team:obs-ux-management"],"number":195902,"url":"https://github.com/elastic/kibana/pull/195902","mergeCommit":{"message":"Migrate Custom threshold > AVG - PCT - FIRED test to the deployment agnostic framework (#195902)\n\nPart of #183378\r\n\r\n## Summary\r\nThis PR moves the first Custom threshold rule test to the deployment\r\nagnostic test. The rest will follow in a follow-up PR.\r\n\r\n## How to run\r\n\r\nTo run serverless\r\n```\r\nnode scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts\r\nnode scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts --grep=\"Custom Threshold rule\"\r\n```\r\n\r\nTo run stateful\r\n```\r\nnode scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts\r\nnode scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts --grep=\"Custom Threshold rule\"\r\n```\r\n\r\n### TODO\r\n\r\n- [x] https://github.com/elastic/kibana/pull/195890\r\n- [x] Test in MKI before merging\r\n\r\n\r\n#### How to run tests on MKI\r\n\r\nAccording to this\r\n[discussion](https://github.com/elastic/observability-dev/issues/3519#issuecomment-2379914274),\r\nwe should test in MKI environment before merging. For details on how to\r\nrun in MKI, see [this section of the\r\ndocument](https://docs.google.com/document/d/1tiax7xoDYwFXYZjRTgVKkVMjN-SQzBWk4yn1JY6Z5UY/edit#heading=h.ece2z8p74izh)\r\nand [this\r\nreadme](https://github.com/elastic/kibana/blob/main/x-pack/test_serverless/README.md#run-tests-on-mki).","sha":"30f81ce4e932622e4d284b1e6af8c015c22836f5"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/195902","number":195902,"mergeCommit":{"message":"Migrate Custom threshold > AVG - PCT - FIRED test to the deployment agnostic framework (#195902)\n\nPart of #183378\r\n\r\n## Summary\r\nThis PR moves the first Custom threshold rule test to the deployment\r\nagnostic test. The rest will follow in a follow-up PR.\r\n\r\n## How to run\r\n\r\nTo run serverless\r\n```\r\nnode scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts\r\nnode scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts --grep=\"Custom Threshold rule\"\r\n```\r\n\r\nTo run stateful\r\n```\r\nnode scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts\r\nnode scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts --grep=\"Custom Threshold rule\"\r\n```\r\n\r\n### TODO\r\n\r\n- [x] https://github.com/elastic/kibana/pull/195890\r\n- [x] Test in MKI before merging\r\n\r\n\r\n#### How to run tests on MKI\r\n\r\nAccording to this\r\n[discussion](https://github.com/elastic/observability-dev/issues/3519#issuecomment-2379914274),\r\nwe should test in MKI environment before merging. For details on how to\r\nrun in MKI, see [this section of the\r\ndocument](https://docs.google.com/document/d/1tiax7xoDYwFXYZjRTgVKkVMjN-SQzBWk4yn1JY6Z5UY/edit#heading=h.ece2z8p74izh)\r\nand [this\r\nreadme](https://github.com/elastic/kibana/blob/main/x-pack/test_serverless/README.md#run-tests-on-mki).","sha":"30f81ce4e932622e4d284b1e6af8c015c22836f5"}}]}] BACKPORT-->
kibanamachine
added
v8.17.0
and removed
backport missing
## Summary Use getUrlPartsWithStrippedDefaultPort to avoid this [issue](#195902 (comment)) on MKI: ``` Serverless Observability - Deployment-agnostic api integration - Custom Threshold rule AVG - PCT - FIRED Rule creation should set correct action variables Error: expected 'https://bk-serverless-ftr-3067-e697d43e3ad9-e0ac80.kb.eu-west-1.aws.qa.elastic.cloud/app/observability/alerts/1e0c2d3e-e5c2-4bfe-9df0-d46681253b9f' to sort of equal 'https://bk-serverless-ftr-3067-e697d43e3ad9-e0ac80.kb.eu-west-1.aws.qa.elastic.cloud:443/app/observability/alerts/1e0c2d3e-e5c2-4bfe-9df0-d46681253b9f' ```
…c#199264) ## Summary Use getUrlPartsWithStrippedDefaultPort to avoid this [issue](elastic#195902 (comment)) on MKI: ``` Serverless Observability - Deployment-agnostic api integration - Custom Threshold rule AVG - PCT - FIRED Rule creation should set correct action variables Error: expected 'https://bk-serverless-ftr-3067-e697d43e3ad9-e0ac80.kb.eu-west-1.aws.qa.elastic.cloud/app/observability/alerts/1e0c2d3e-e5c2-4bfe-9df0-d46681253b9f' to sort of equal 'https://bk-serverless-ftr-3067-e697d43e3ad9-e0ac80.kb.eu-west-1.aws.qa.elastic.cloud:443/app/observability/alerts/1e0c2d3e-e5c2-4bfe-9df0-d46681253b9f' ``` (cherry picked from commit 996104f)
…c#199264) ## Summary Use getUrlPartsWithStrippedDefaultPort to avoid this [issue](elastic#195902 (comment)) on MKI: ``` Serverless Observability - Deployment-agnostic api integration - Custom Threshold rule AVG - PCT - FIRED Rule creation should set correct action variables Error: expected 'https://bk-serverless-ftr-3067-e697d43e3ad9-e0ac80.kb.eu-west-1.aws.qa.elastic.cloud/app/observability/alerts/1e0c2d3e-e5c2-4bfe-9df0-d46681253b9f' to sort of equal 'https://bk-serverless-ftr-3067-e697d43e3ad9-e0ac80.kb.eu-west-1.aws.qa.elastic.cloud:443/app/observability/alerts/1e0c2d3e-e5c2-4bfe-9df0-d46681253b9f' ```
Part of #183378
Summary
This PR moves the first Custom threshold rule test to the deployment agnostic test. The rest will follow in a follow-up PR.
How to run
To run serverless
To run stateful
TODO
How to run tests on MKI
According to this discussion, we should test in MKI environment before merging. For details on how to run in MKI, see this section of the document and this readme.