chore(NA): rebalance x-pack cigroups

[mistic]

After #85191 our CI was again over 2h20m and it seems very unstable reaching the overall timeout a couple of times which is generating a lot of failures. That change intends to rebalance the ciGroups by introducing a new ciGroup12 and getting the CI back again into around 1h40m. I'm also expecting to get the CI more stable again once that change is merged in.

[mistic]

@elasticmachine merge upstream

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[tylersmalley]

Current times on master:

name	time
ciGroup1	6:24
ciGroup2	0:20
ciGroup3	26
ciGroup4	0:23
ciGroup5	0:15
ciGroup6	26
ciGroup7	0:14
ciGroup8	0:16
ciGroup9	0:23
ciGroup10	0:16
ciGroup11	0:25
ciGroup12	26
xpack-ciGroup1	0:57
xpack-ciGroup2	1:07
xpack-ciGroup3	1:00
xpack-ciGroup4	0:58
xpack-ciGroup5	0:57
xpack-ciGroup6	0:55
xpack-ciGroup7	0:53
xpack-ciGroup8	0:51
xpack-ciGroup9	0:58
xpack-ciGroup10	1:04
xpack-ciGroup11	1:36

@brianseeders raised a concern last time around limited resources with adding another CI group. If this is the case, it seems like we could drastically decrease the number of OSS ciGroups to compensate.

[spong]

LGTM -- thank you for all your efforts here in keeping the build times down @mistic! 🙂

[dmlemeshko]

LGTM

[FrankHassanabad]

LGTM

[mistic]

@elasticmachine merge upstream

[legrego]

Group change in x-pack/test/encrypted_saved_objects_api_integration/tests/index.ts LGTM! Thanks for rebalancing!

[brianseeders]

@brianseeders raised a concern last time around limited resources with adding another CI group. If this is the case, it seems like we could drastically decrease the number of OSS ciGroups to compensate.

This is primarily a problem on jobs that don't use the tasks framework (es snapshots, code coverage for example). In those jobs, oss and xpack cigroups run on different machines, so reducing the number of oss groups won't do anything for xpack. We could make the machines a little bigger for those jobs, if we need to

[tylersmalley]

@mistic why did you add two new groups? I see ciGroup12 taking 38 minutes and ciGroup13 taking 25. Together that would be under the 1 hour average for the other ciGroups.

[mistic]

@tylersmalley the best I could do with only one added ciGroup was getting the CI into 1h57 on https://kibana-ci.elastic.co/job/elastic+kibana+pipeline-pull-request/94337/ and we will stay with only a little room for new tests to be added. I think the best to do here is to add those two new groups until we found a different fix for those time consuming tests.

[ymao1]

Alerting changes LGTM

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 3b8efd0

Metrics [docs]

Distributable file count

id	before	after	diff
default	47268	48028	+760

History

• 💚 Build #94796 succeeded 3b8efd0
• 💚 Build #94772 succeeded b8495b1
• 💚 Build #94752 succeeded cdcb3e4
• 💚 Build #94712 succeeded a2f14c6
• 💚 Build #94651 succeeded ea89b92

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[tylersmalley]

Reverted. This led to us hitting memory ceiling on the ES verification job, increased those instances in #86192 which resulted in getting "message": "index [.kibana_1] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];: cluster_block_exception" even though the I am only seeing ~3% disk utilization. Will take another swing at this tomorrow.

master: 6671cf3
7.x: af5b7af