Skip to content

Commit

Permalink
[ftr] split alerting_api_integration/spaces_only/config.ts into sma…
Browse files Browse the repository at this point in the history
…ll fast configs (elastic#149854)

## Summary

Trying to address slow config issue:

```
The following "Functional Tests" configs have durations that exceed the maximum amount of time desired for a single CI job. This is not an error, and if you don't own any of these configs then you can ignore this warning.If you own any of these configs please split them up ASAP and ask Operations if you have questions about how to do that.

x-pack/test/alerting_api_integration/spaces_only/config.ts: 41.4 minutes
```

by splitting it into multiple groups.

_1 round (splitting main index file with 3 index suites where each one
has its own setup/tearDown + alerting suite into 4 groups)_

x-pack/test/alerting_api_integration/spaces_only/tests/alerting/group1/config.ts
7m 1s

x-pack/test/alerting_api_integration/spaces_only/tests/alerting/group2/config.ts
**15m 10s**

x-pack/test/alerting_api_integration/spaces_only/tests/alerting/group3/config.ts
**21m 40s**

x-pack/test/alerting_api_integration/spaces_only/tests/alerting/group4/config.ts
5m 30s

x-pack/test/alerting_api_integration/spaces_only/tests/action_task_params/config.ts
2m 31s

x-pack/test/alerting_api_integration/spaces_only/tests/actions/config.ts
4m 22s

_2 round (rebalance groups 1-4 to be more time equal)_

x-pack/test/alerting_api_integration/spaces_only/tests/alerting/group1/config.ts
12m 46s

x-pack/test/alerting_api_integration/spaces_only/tests/alerting/group2/config.ts
8m 46s

x-pack/test/alerting_api_integration/spaces_only/tests/alerting/group3/config.ts
17m 30s

x-pack/test/alerting_api_integration/spaces_only/tests/alerting/group4/config.ts
9m 5s

Here `Alerting eventLog alerts should generate expected alert events for
normal operation` test started to fail, probably there is a dependency
on the previous tests.

_3 round (rebalance groups 1-4, to keep tests order in group 1 up until
`event_log.ts` suite)_

x-pack/test/alerting_api_integration/spaces_only/tests/alerting/group1/config.ts
17m 12s

x-pack/test/alerting_api_integration/spaces_only/tests/alerting/group2/config.ts
8m 28s

x-pack/test/alerting_api_integration/spaces_only/tests/alerting/group3/config.ts
16m 15s

x-pack/test/alerting_api_integration/spaces_only/tests/alerting/group4/config.ts
6m 21s

_4 round (rebalancing groups 3-4 to be more time equal)_

x-pack/test/alerting_api_integration/spaces_only/tests/alerting/group1/config.ts
**17m 14s**

x-pack/test/alerting_api_integration/spaces_only/tests/alerting/group2/config.ts
**8m 37s**

x-pack/test/alerting_api_integration/spaces_only/tests/alerting/group3/config.ts
**12m 40s**

x-pack/test/alerting_api_integration/spaces_only/tests/alerting/group4/config.ts
**9m 49s**

---------

Co-authored-by: kibanamachine <[email protected]>
  • Loading branch information
2 people authored and kqualters-elastic committed Feb 6, 2023
1 parent c757f7f commit 1ec49d2
Show file tree
Hide file tree
Showing 87 changed files with 457 additions and 263 deletions.
7 changes: 6 additions & 1 deletion .buildkite/ftr_configs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,12 @@ enabled:
- x-pack/test/alerting_api_integration/security_and_spaces/group2/config.ts
- x-pack/test/alerting_api_integration/security_and_spaces/group3/config.ts
- x-pack/test/alerting_api_integration/security_and_spaces/group2/config_non_dedicated_task_runner.ts
- x-pack/test/alerting_api_integration/spaces_only/config.ts
- x-pack/test/alerting_api_integration/spaces_only/tests/alerting/group1/config.ts
- x-pack/test/alerting_api_integration/spaces_only/tests/alerting/group2/config.ts
- x-pack/test/alerting_api_integration/spaces_only/tests/alerting/group3/config.ts
- x-pack/test/alerting_api_integration/spaces_only/tests/alerting/group4/config.ts
- x-pack/test/alerting_api_integration/spaces_only/tests/actions/config.ts
- x-pack/test/alerting_api_integration/spaces_only/tests/action_task_params/config.ts
- x-pack/test/api_integration_basic/config.ts
- x-pack/test/api_integration/config_security_basic.ts
- x-pack/test/api_integration/config_security_trial.ts
Expand Down
4 changes: 3 additions & 1 deletion x-pack/test/alerting_api_integration/common/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ interface CreateTestConfigOptions {
rejectUnauthorized?: boolean; // legacy
emailDomainsAllowed?: string[];
testFiles?: string[];
reportName?: string;
useDedicatedTaskRunner: boolean;
}

Expand Down Expand Up @@ -73,6 +74,7 @@ export function createTestConfig(name: string, options: CreateTestConfigOptions)
rejectUnauthorized = true, // legacy
emailDomainsAllowed = undefined,
testFiles = undefined,
reportName = undefined,
useDedicatedTaskRunner,
} = options;

Expand Down Expand Up @@ -154,7 +156,7 @@ export function createTestConfig(name: string, options: CreateTestConfigOptions)
servers,
services,
junit: {
reportName: 'X-Pack Alerting API Integration Tests',
reportName: reportName ? reportName : 'X-Pack Alerting API Integration Tests',
},
esTestCluster: {
...xPackApiIntegrationTestsConfig.get('esTestCluster'),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import expect from '@kbn/expect';
import { Spaces } from '../../../scenarios';
import { getUrlPrefix, getTestRuleData, ObjectRemover, getEventLog } from '../../../../common/lib';
import { FtrProviderContext } from '../../../../common/ftr_provider_context';
import { validateEvent } from '../../../../spaces_only/tests/alerting/event_log';
import { validateEvent } from '../../../../spaces_only/tests/alerting/group1/event_log';

// eslint-disable-next-line import/no-default-export
export default function eventLogTests({ getService }: FtrProviderContext) {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import { createTestConfig } from '../../../common/config';

export const EmailDomainsAllowed = ['example.org', 'test.com'];

// eslint-disable-next-line import/no-default-export
export default createTestConfig('spaces_only', {
disabledPlugins: ['security'],
license: 'trial',
enableActionsProxy: false,
verificationMode: 'none',
customizeLocalHostSsl: true,
preconfiguredAlertHistoryEsIndex: true,
emailDomainsAllowed: EmailDomainsAllowed,
useDedicatedTaskRunner: true,
testFiles: [require.resolve('.')],
reportName: 'X-Pack Alerting API Integration Tests - Action Task Params',
});
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
*/

import { FtrProviderContext } from '../../../common/ftr_provider_context';
import { buildUp, tearDown } from '..';
import { buildUp, tearDown } from '../helpers';

// eslint-disable-next-line import/no-default-export
export default function actionTaskParamsTests({ loadTestFile, getService }: FtrProviderContext) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
* 2.0.
*/

import { createTestConfig } from '../common/config';
import { createTestConfig } from '../../../common/config';

export const EmailDomainsAllowed = ['example.org', 'test.com'];

Expand All @@ -19,4 +19,6 @@ export default createTestConfig('spaces_only', {
preconfiguredAlertHistoryEsIndex: true,
emailDomainsAllowed: EmailDomainsAllowed,
useDedicatedTaskRunner: true,
testFiles: [require.resolve('.')],
reportName: 'X-Pack Alerting API Integration Tests - Actions',
});
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import expect from '@kbn/expect';

import { FtrProviderContext } from '../../../../../common/ftr_provider_context';
import { ObjectRemover } from '../../../../../common/lib';
import { EmailDomainsAllowed } from '../../../../config';
import { EmailDomainsAllowed } from '../../config';

const EmailDomainAllowed = EmailDomainsAllowed[EmailDomainsAllowed.length - 1];

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
*/

import { FtrProviderContext } from '../../../common/ftr_provider_context';
import { buildUp, tearDown } from '..';
import { buildUp, tearDown } from '../helpers';

// eslint-disable-next-line import/no-default-export
export default function actionsTests({ loadTestFile, getService }: FtrProviderContext) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@
*/

import expect from '@kbn/expect';
import { Spaces } from '../../scenarios';
import { getUrlPrefix, getTestRuleData, ObjectRemover } from '../../../common/lib';
import { FtrProviderContext } from '../../../common/ftr_provider_context';
import { Spaces } from '../../../scenarios';
import { getUrlPrefix, getTestRuleData, ObjectRemover } from '../../../../common/lib';
import { FtrProviderContext } from '../../../../common/ftr_provider_context';

// eslint-disable-next-line import/no-default-export
export default function createAggregateTests({ getService }: FtrProviderContext) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@
*/

import expect from '@kbn/expect';
import { Spaces } from '../../scenarios';
import { getUrlPrefix, getTestRuleData, ObjectRemover } from '../../../common/lib';
import { FtrProviderContext } from '../../../common/ftr_provider_context';
import { Spaces } from '../../../scenarios';
import { getUrlPrefix, getTestRuleData, ObjectRemover } from '../../../../common/lib';
import { FtrProviderContext } from '../../../../common/ftr_provider_context';

// eslint-disable-next-line import/no-default-export
export default function createAggregateTests({ getService }: FtrProviderContext) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,16 +13,16 @@ import { RecoveredActionGroup } from '@kbn/alerting-plugin/common';
import { TaskRunning, TaskRunningStage } from '@kbn/task-manager-plugin/server/task_running';
import { ConcreteTaskInstance } from '@kbn/task-manager-plugin/server';
import { ESTestIndexTool, ES_TEST_INDEX_NAME } from '@kbn/alerting-api-integration-helpers';
import { Space } from '../../../common/types';
import { FtrProviderContext } from '../../../common/ftr_provider_context';
import { Space } from '../../../../common/types';
import { FtrProviderContext } from '../../../../common/ftr_provider_context';
import {
getUrlPrefix,
getTestRuleData,
ObjectRemover,
AlertUtils,
ensureDatetimeIsWithinRange,
TaskManagerUtils,
} from '../../../common/lib';
} from '../../../../common/lib';

export function alertTests({ getService }: FtrProviderContext, space: Space) {
const supertestWithoutAuth = getService('supertestWithoutAuth');
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import { createTestConfig } from '../../../../common/config';

export const EmailDomainsAllowed = ['example.org', 'test.com'];

// eslint-disable-next-line import/no-default-export
export default createTestConfig('spaces_only', {
disabledPlugins: ['security'],
license: 'trial',
enableActionsProxy: false,
verificationMode: 'none',
customizeLocalHostSsl: true,
preconfiguredAlertHistoryEsIndex: true,
emailDomainsAllowed: EmailDomainsAllowed,
useDedicatedTaskRunner: true,
testFiles: [require.resolve('.')],
reportName: 'X-Pack Alerting API Integration Tests - Alerting - group1',
});
Original file line number Diff line number Diff line change
Expand Up @@ -8,16 +8,16 @@
import expect from '@kbn/expect';
import { SavedObject } from '@kbn/core/server';
import { RawRule } from '@kbn/alerting-plugin/server/types';
import { Spaces } from '../../scenarios';
import { Spaces } from '../../../scenarios';
import {
checkAAD,
getUrlPrefix,
getTestRuleData,
ObjectRemover,
getConsumerUnauthorizedErrorMessage,
TaskManagerDoc,
} from '../../../common/lib';
import { FtrProviderContext } from '../../../common/ftr_provider_context';
} from '../../../../common/lib';
import { FtrProviderContext } from '../../../../common/ftr_provider_context';

// eslint-disable-next-line import/no-default-export
export default function createAlertTests({ getService }: FtrProviderContext) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@
*/

import expect from '@kbn/expect';
import { Spaces } from '../../scenarios';
import { getUrlPrefix, getTestRuleData, ObjectRemover } from '../../../common/lib';
import { FtrProviderContext } from '../../../common/ftr_provider_context';
import { Spaces } from '../../../scenarios';
import { getUrlPrefix, getTestRuleData, ObjectRemover } from '../../../../common/lib';
import { FtrProviderContext } from '../../../../common/ftr_provider_context';

// eslint-disable-next-line import/no-default-export
export default function createDeleteTests({ getService }: FtrProviderContext) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@
*/

import expect from '@kbn/expect';
import { Spaces } from '../../scenarios';
import { FtrProviderContext } from '../../../common/ftr_provider_context';
import { Spaces } from '../../../scenarios';
import { FtrProviderContext } from '../../../../common/ftr_provider_context';
import {
AlertUtils as RuleUtils,
checkAAD,
Expand All @@ -16,7 +16,7 @@ import {
ObjectRemover,
getEventLog,
TaskManagerDoc,
} from '../../../common/lib';
} from '../../../../common/lib';
import { validateEvent } from './event_log';

// eslint-disable-next-line import/no-default-export
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,16 @@
*/

import expect from '@kbn/expect';
import { Spaces } from '../../scenarios';
import { FtrProviderContext } from '../../../common/ftr_provider_context';
import { Spaces } from '../../../scenarios';
import { FtrProviderContext } from '../../../../common/ftr_provider_context';
import {
AlertUtils,
checkAAD,
getUrlPrefix,
getTestRuleData,
ObjectRemover,
TaskManagerDoc,
} from '../../../common/lib';
} from '../../../../common/lib';

// eslint-disable-next-line import/no-default-export
export default function createEnableAlertTests({ getService }: FtrProviderContext) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@
import expect from '@kbn/expect';
import { IValidatedEvent, nanosToMillis } from '@kbn/event-log-plugin/server';
import { ESTestIndexTool } from '@kbn/alerting-api-integration-helpers';
import { Spaces } from '../../scenarios';
import { getUrlPrefix, getTestRuleData, ObjectRemover, getEventLog } from '../../../common/lib';
import { FtrProviderContext } from '../../../common/ftr_provider_context';
import { Spaces } from '../../../scenarios';
import { getUrlPrefix, getTestRuleData, ObjectRemover, getEventLog } from '../../../../common/lib';
import { FtrProviderContext } from '../../../../common/ftr_provider_context';

// eslint-disable-next-line import/no-default-export
export default function eventLogTests({ getService }: FtrProviderContext) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@
import expect from '@kbn/expect';
import { SuperTest, Test } from 'supertest';
import { fromKueryExpression } from '@kbn/es-query';
import { Spaces } from '../../scenarios';
import { getUrlPrefix, getTestRuleData, ObjectRemover } from '../../../common/lib';
import { FtrProviderContext } from '../../../common/ftr_provider_context';
import { Spaces } from '../../../scenarios';
import { getUrlPrefix, getTestRuleData, ObjectRemover } from '../../../../common/lib';
import { FtrProviderContext } from '../../../../common/ftr_provider_context';

async function createAlert(
objectRemover: ObjectRemover,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@

import expect from '@kbn/expect';
import { SuperTest, Test } from 'supertest';
import { Spaces } from '../../scenarios';
import { getUrlPrefix, getTestRuleData, ObjectRemover } from '../../../common/lib';
import { FtrProviderContext } from '../../../common/ftr_provider_context';
import { Spaces } from '../../../scenarios';
import { getUrlPrefix, getTestRuleData, ObjectRemover } from '../../../../common/lib';
import { FtrProviderContext } from '../../../../common/ftr_provider_context';

const getTestUtils = (
describeType: 'internal' | 'public',
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@
import expect from '@kbn/expect';
import { ESTestIndexTool } from '@kbn/alerting-api-integration-helpers';

import { Spaces } from '../../scenarios';
import { getUrlPrefix, ObjectRemover, getTestRuleData, getEventLog } from '../../../common/lib';
import { FtrProviderContext } from '../../../common/ftr_provider_context';
import { Spaces } from '../../../scenarios';
import { getUrlPrefix, ObjectRemover, getTestRuleData, getEventLog } from '../../../../common/lib';
import { FtrProviderContext } from '../../../../common/ftr_provider_context';

// eslint-disable-next-line import/no-default-export
export default function createGetActionErrorLogTests({ getService }: FtrProviderContext) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@
*/

import expect from '@kbn/expect';
import { Spaces } from '../../scenarios';
import { getUrlPrefix, ObjectRemover, getTestRuleData } from '../../../common/lib';
import { FtrProviderContext } from '../../../common/ftr_provider_context';
import { Spaces } from '../../../scenarios';
import { getUrlPrefix, ObjectRemover, getTestRuleData } from '../../../../common/lib';
import { FtrProviderContext } from '../../../../common/ftr_provider_context';

// eslint-disable-next-line import/no-default-export
export default function createGetAlertStateTests({ getService }: FtrProviderContext) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,15 @@
import expect from '@kbn/expect';
import { omit } from 'lodash';

import { Spaces } from '../../scenarios';
import { Spaces } from '../../../scenarios';
import {
getUrlPrefix,
ObjectRemover,
getTestRuleData,
AlertUtils,
getEventLog,
} from '../../../common/lib';
import { FtrProviderContext } from '../../../common/ftr_provider_context';
} from '../../../../common/lib';
import { FtrProviderContext } from '../../../../common/ftr_provider_context';

// eslint-disable-next-line import/no-default-export
export default function createGetAlertSummaryTests({ getService }: FtrProviderContext) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@
import expect from '@kbn/expect';
import { ESTestIndexTool } from '@kbn/alerting-api-integration-helpers';

import { Spaces } from '../../scenarios';
import { getUrlPrefix, ObjectRemover, getTestRuleData, getEventLog } from '../../../common/lib';
import { FtrProviderContext } from '../../../common/ftr_provider_context';
import { Spaces } from '../../../scenarios';
import { getUrlPrefix, ObjectRemover, getTestRuleData, getEventLog } from '../../../../common/lib';
import { FtrProviderContext } from '../../../../common/ftr_provider_context';

// eslint-disable-next-line import/no-default-export
export default function createGetExecutionLogTests({ getService }: FtrProviderContext) {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import { FtrProviderContext } from '../../../../common/ftr_provider_context';
import { buildUp, tearDown } from '../../helpers';

// eslint-disable-next-line import/no-default-export
export default function alertingTests({ loadTestFile, getService }: FtrProviderContext) {
describe('Alerting', () => {
before(async () => await buildUp(getService));
after(async () => await tearDown(getService));

loadTestFile(require.resolve('./aggregate'));
loadTestFile(require.resolve('./aggregate_post'));
loadTestFile(require.resolve('./create'));
loadTestFile(require.resolve('./delete'));
loadTestFile(require.resolve('./disable'));
loadTestFile(require.resolve('./enable'));
loadTestFile(require.resolve('./find'));
loadTestFile(require.resolve('./get'));
loadTestFile(require.resolve('./get_alert_state'));
loadTestFile(require.resolve('./get_alert_summary'));
loadTestFile(require.resolve('./get_execution_log'));
loadTestFile(require.resolve('./get_action_error_log'));
loadTestFile(require.resolve('./rule_types'));
loadTestFile(require.resolve('./event_log'));
});
}
Loading

0 comments on commit 1ec49d2

Please sign in to comment.