Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Defend Workflows] Add tags for mocked and real Endpoint Cypress tests for Serverless #165094

Merged
Merged
Changes from 1 commit
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
b6642b6
add `cypress:dw:serverless:*` scripts
gergoabraham Aug 24, 2023
8cc56e7
add `Elastic-Api-Version` to all cy requests
gergoabraham Aug 28, 2023
8645b2c
add `http.host` to shared setting, so ES is accessible via local IP
gergoabraham Aug 29, 2023
db6b729
add config and scripts for real endpoint serverless
gergoabraham Aug 29, 2023
b51cdbf
fix test scripts (removing `ts` and `open` from shared scripts)
gergoabraham Aug 29, 2023
774c548
add `@ess` tag for every test
gergoabraham Aug 29, 2023
c85ec25
fix eslint errors
gergoabraham Aug 29, 2023
26f13f1
add `@ess` grep to ess test configs
gergoabraham Aug 29, 2023
96681c4
fine tune grep config
gergoabraham Aug 29, 2023
171d093
move cypress configs into cypress folder
gergoabraham Aug 29, 2023
d012eaf
run tests on CI
gergoabraham Aug 29, 2023
5bd9599
add `@serverless` tag to selected cypress tests
gergoabraham Aug 29, 2023
bdabdfa
fix eslint errors
gergoabraham Aug 30, 2023
df6c6f6
improve earlier eslint fix: remove unnecessary click
gergoabraham Aug 30, 2023
5b3f9e5
add `@brokenInServerless` to all "real endpoint" tests, until we got …
gergoabraham Aug 30, 2023
f01466d
add reporter config to cypress configs
gergoabraham Aug 30, 2023
0715a0c
fix lint errors
gergoabraham Sep 8, 2023
783402a
remove CI job for real endpoints against serverless
gergoabraham Sep 11, 2023
addb780
remove `@brokenInServerless` tags from skipped real endpoint tests
gergoabraham Sep 11, 2023
0019766
fix the eslint fix
gergoabraham Sep 11, 2023
4da5f86
run a placeholder test case for mocked_data
gergoabraham Sep 14, 2023
48ec79d
Revert "remove `@brokenInServerless` tags from skipped real endpoint …
gergoabraham Sep 14, 2023
3aaf36a
Revert "remove CI job for real endpoints against serverless"
gergoabraham Sep 14, 2023
88d79c2
run a placeholder test case for real endpoint serverless tests
gergoabraham Sep 14, 2023
bf44385
remove placeholder tests
gergoabraham Sep 14, 2023
4ac5a03
prevent `junit:merge` from failing CI step
gergoabraham Sep 14, 2023
08fab69
Merge branch 'main' into test/olm-7133-reuse-tests-for-serverless
gergoabraham Sep 19, 2023
4a0cea8
remove `http.host` server arg from shared config
gergoabraham Sep 19, 2023
11132d6
prepare `runEndpointLoaderScript()` for serverless
gergoabraham Sep 19, 2023
63e5342
add `dataLoaders()` back to serverless config
gergoabraham Sep 19, 2023
a2dca01
enable `Policy Details` cy test for serverless
gergoabraham Sep 19, 2023
6ae9f21
re-enable configurations that have problems fixed
gergoabraham Sep 19, 2023
b08267a
update readme
gergoabraham Sep 19, 2023
ad1c3fe
add reusable base config for cypress
gergoabraham Sep 19, 2023
6f90e66
Merge branch 'main' into test/olm-7133-reuse-tests-for-serverless
gergoabraham Sep 19, 2023
7ec96a3
add IS_SERVERLESS to cy config
gergoabraham Sep 19, 2023
75cde75
convert base config to a function instead of const
gergoabraham Sep 19, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add @brokenInServerless to all "real endpoint" tests, until we got …
…a workaround
gergoabraham committed Sep 14, 2023
commit 5b3f9e53377eb7415d035b4dac69d220e3f32fe2
Original file line number Diff line number Diff line change
@@ -36,7 +36,7 @@ const yieldAppliedEndpointRevision = (): Cypress.Chainable<number> =>

const parseRevNumber = (revString: string) => Number(revString.match(/\d+/)?.[0]);

describe('Artifact pages', { tags: ['@ess', '@serverless'] }, () => {
describe('Artifact pages', { tags: ['@ess', '@serverless', '@brokenInServerless'] }, () => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be great if we could type this new property (tags) so that we get content assist. Can you add that to our cypress.d.ts file here:

interface SuiteConfigOverrides {
env?: {
ftrConfig: SecuritySolutionDescribeBlockFtrConfig;
};
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually - looks like tags is already defined via @cypress types so need to add to the our cypress.d.ts file. I do wonder if there is a way to make the type more specific in that it can suggest our "known" tags 🤔

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did some experiments with types to be able to have suggestions, but didn't work so far, exactly because tags are already defined in cypress.d.ts, and therefore it cannot be overriden by a more exact type.

one solution would be to use our own security_tags or something, but I'm not sure if grep can pick it up - I'll look into this, but this is definitely an open question.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no big deal. it would just help with DX. Don't hold this PR because of this.

let indexedPolicy: IndexedFleetEndpointPolicyResponse;
let policy: PolicyData;
let createdHost: CreateAndEnrollEndpointHostResponse;
Original file line number Diff line number Diff line change
@@ -20,86 +20,91 @@ import { createEndpointHost } from '../../tasks/create_endpoint_host';
import { deleteAllLoadedEndpointData } from '../../tasks/delete_all_endpoint_data';
import { enableAllPolicyProtections } from '../../tasks/endpoint_policy';

describe('Automated Response Actions', { tags: ['@ess', '@serverless'] }, () => {
let indexedPolicy: IndexedFleetEndpointPolicyResponse;
let policy: PolicyData;
let createdHost: CreateAndEnrollEndpointHostResponse;

before(() => {
getEndpointIntegrationVersion().then((version) =>
createAgentPolicyTask(version, 'automated_response_actions').then((data) => {
indexedPolicy = data;
policy = indexedPolicy.integrationPolicies[0];

return enableAllPolicyProtections(policy.id).then(() => {
// Create and enroll a new Endpoint host
return createEndpointHost(policy.policy_id).then((host) => {
createdHost = host as CreateAndEnrollEndpointHostResponse;
describe(
'Automated Response Actions',
{ tags: ['@ess', '@serverless', '@brokenInServerless'] },
() => {
let indexedPolicy: IndexedFleetEndpointPolicyResponse;
let policy: PolicyData;
let createdHost: CreateAndEnrollEndpointHostResponse;

before(() => {
getEndpointIntegrationVersion().then((version) =>
createAgentPolicyTask(version, 'automated_response_actions').then((data) => {
indexedPolicy = data;
policy = indexedPolicy.integrationPolicies[0];

return enableAllPolicyProtections(policy.id).then(() => {
// Create and enroll a new Endpoint host
return createEndpointHost(policy.policy_id).then((host) => {
createdHost = host as CreateAndEnrollEndpointHostResponse;
});
});
});
})
);
});
})
);
});

after(() => {
if (createdHost) {
cy.task('destroyEndpointHost', createdHost);
}
after(() => {
if (createdHost) {
cy.task('destroyEndpointHost', createdHost);
}

if (indexedPolicy) {
cy.task('deleteIndexedFleetEndpointPolicies', indexedPolicy);
}
if (indexedPolicy) {
cy.task('deleteIndexedFleetEndpointPolicies', indexedPolicy);
}

if (createdHost) {
deleteAllLoadedEndpointData({ endpointAgentIds: [createdHost.agentId] });
}
});
if (createdHost) {
deleteAllLoadedEndpointData({ endpointAgentIds: [createdHost.agentId] });
}
});

const hostname = new URL(Cypress.env('FLEET_SERVER_URL')).port;
const fleetHostname = `dev-fleet-server.${hostname}`;
const hostname = new URL(Cypress.env('FLEET_SERVER_URL')).port;
const fleetHostname = `dev-fleet-server.${hostname}`;

beforeEach(() => {
login();
disableExpandableFlyoutAdvancedSettings();
});
beforeEach(() => {
login();
disableExpandableFlyoutAdvancedSettings();
});

describe('From alerts', () => {
let ruleId: string;
let ruleName: string;
describe('From alerts', () => {
let ruleId: string;
let ruleName: string;

before(() => {
loadRule().then((data) => {
ruleId = data.id;
ruleName = data.name;
before(() => {
loadRule().then((data) => {
ruleId = data.id;
ruleName = data.name;
});
});
});

after(() => {
if (ruleId) {
cleanupRule(ruleId);
}
});
it.skip('should have generated endpoint and rule', () => {
loadPage(APP_ENDPOINTS_PATH);
cy.contains(createdHost.hostname).should('exist');
after(() => {
if (ruleId) {
cleanupRule(ruleId);
}
});

toggleRuleOffAndOn(ruleName);
it.skip('should have generated endpoint and rule', () => {
loadPage(APP_ENDPOINTS_PATH);
cy.contains(createdHost.hostname).should('exist');

visitRuleAlerts(ruleName);
closeAllToasts();
toggleRuleOffAndOn(ruleName);

changeAlertsFilter('event.category: "file"');
cy.getByTestSubj('expand-event').first().click();
cy.getByTestSubj('responseActionsViewTab').click();
cy.getByTestSubj('response-actions-notification').should('not.have.text', '0');
visitRuleAlerts(ruleName);
closeAllToasts();

cy.getByTestSubj(`response-results-${createdHost.hostname}-details-tray`)
.should('contain', 'isolate completed successfully')
.and('contain', createdHost.hostname);
changeAlertsFilter('event.category: "file"');
cy.getByTestSubj('expand-event').first().click();
cy.getByTestSubj('responseActionsViewTab').click();
cy.getByTestSubj('response-actions-notification').should('not.have.text', '0');

cy.getByTestSubj(`response-results-${fleetHostname}-details-tray`)
.should('contain', 'The host does not have Elastic Defend integration installed')
.and('contain', 'dev-fleet-server');
cy.getByTestSubj(`response-results-${createdHost.hostname}-details-tray`)
.should('contain', 'isolate completed successfully')
.and('contain', createdHost.hostname);

cy.getByTestSubj(`response-results-${fleetHostname}-details-tray`)
.should('contain', 'The host does not have Elastic Defend integration installed')
.and('contain', 'dev-fleet-server');
});
});
});
});
}
);
Original file line number Diff line number Diff line change
@@ -19,76 +19,80 @@ import { login } from '../../tasks/login';
import { EXECUTE_ROUTE } from '../../../../../common/endpoint/constants';
import { waitForActionToComplete } from '../../tasks/response_actions';

describe('Endpoint generated alerts', { tags: ['@ess', '@serverless'] }, () => {
let indexedPolicy: IndexedFleetEndpointPolicyResponse;
let policy: PolicyData;
let createdHost: CreateAndEnrollEndpointHostResponse;
describe(
'Endpoint generated alerts',
{ tags: ['@ess', '@serverless', '@brokenInServerless'] },
() => {
let indexedPolicy: IndexedFleetEndpointPolicyResponse;
let policy: PolicyData;
let createdHost: CreateAndEnrollEndpointHostResponse;

before(() => {
getEndpointIntegrationVersion().then((version) => {
createAgentPolicyTask(version, 'alerts test').then((data) => {
indexedPolicy = data;
policy = indexedPolicy.integrationPolicies[0];
before(() => {
getEndpointIntegrationVersion().then((version) => {
createAgentPolicyTask(version, 'alerts test').then((data) => {
indexedPolicy = data;
policy = indexedPolicy.integrationPolicies[0];

return enableAllPolicyProtections(policy.id).then(() => {
// Create and enroll a new Endpoint host
return createEndpointHost(policy.policy_id).then((host) => {
createdHost = host as CreateAndEnrollEndpointHostResponse;
return enableAllPolicyProtections(policy.id).then(() => {
// Create and enroll a new Endpoint host
return createEndpointHost(policy.policy_id).then((host) => {
createdHost = host as CreateAndEnrollEndpointHostResponse;
});
});
});
});
});
});

after(() => {
if (createdHost) {
cy.task('destroyEndpointHost', createdHost);
}
after(() => {
if (createdHost) {
cy.task('destroyEndpointHost', createdHost);
}

if (indexedPolicy) {
cy.task('deleteIndexedFleetEndpointPolicies', indexedPolicy);
}
if (indexedPolicy) {
cy.task('deleteIndexedFleetEndpointPolicies', indexedPolicy);
}

if (createdHost) {
deleteAllLoadedEndpointData({ endpointAgentIds: [createdHost.agentId] });
}
});
if (createdHost) {
deleteAllLoadedEndpointData({ endpointAgentIds: [createdHost.agentId] });
}
});

beforeEach(() => {
login();
});
beforeEach(() => {
login();
});

it('should create a Detection Engine alert from an endpoint alert', () => {
// Triggers a Malicious Behaviour alert on Linux system (`grep *` was added only to identify this specific alert)
const executeMaliciousCommand = `bash -c cat /dev/tcp/foo | grep ${Math.random()
.toString(16)
.substring(2)}`;
it('should create a Detection Engine alert from an endpoint alert', () => {
// Triggers a Malicious Behaviour alert on Linux system (`grep *` was added only to identify this specific alert)
const executeMaliciousCommand = `bash -c cat /dev/tcp/foo | grep ${Math.random()
.toString(16)
.substring(2)}`;

// Send `execute` command that triggers malicious behaviour using the `execute` response action
request<ResponseActionApiResponse>({
method: 'POST',
url: EXECUTE_ROUTE,
body: {
endpoint_ids: [createdHost.agentId],
parameters: {
command: executeMaliciousCommand,
},
},
})
.then((response) => waitForActionToComplete(response.body.data.id))
.then(() => {
return waitForEndpointAlerts(createdHost.agentId, [
{
term: { 'process.group_leader.args': executeMaliciousCommand },
// Send `execute` command that triggers malicious behaviour using the `execute` response action
request<ResponseActionApiResponse>({
method: 'POST',
url: EXECUTE_ROUTE,
body: {
endpoint_ids: [createdHost.agentId],
parameters: {
command: executeMaliciousCommand,
},
]);
},
})
.then(() => {
return navigateToAlertsList(
`query=(language:kuery,query:'agent.id: "${createdHost.agentId}" ')`
);
});
.then((response) => waitForActionToComplete(response.body.data.id))
.then(() => {
return waitForEndpointAlerts(createdHost.agentId, [
{
term: { 'process.group_leader.args': executeMaliciousCommand },
},
]);
})
.then(() => {
return navigateToAlertsList(
`query=(language:kuery,query:'agent.id: "${createdHost.agentId}" ')`
);
});

getAlertsTableRows().should('have.length.greaterThan', 0);
});
});
getAlertsTableRows().should('have.length.greaterThan', 0);
});
}
);
Original file line number Diff line number Diff line change
@@ -31,7 +31,7 @@ import { createEndpointHost } from '../../tasks/create_endpoint_host';
import { deleteAllLoadedEndpointData } from '../../tasks/delete_all_endpoint_data';
import { enableAllPolicyProtections } from '../../tasks/endpoint_policy';

describe.skip('Isolate command', { tags: ['@ess', '@serverless'] }, () => {
describe.skip('Isolate command', { tags: ['@ess', '@serverless', '@brokenInServerless'] }, () => {
let isolateComment: string;
let releaseComment: string;
let indexedPolicy: IndexedFleetEndpointPolicyResponse;
Original file line number Diff line number Diff line change
@@ -27,7 +27,7 @@ import { enableAllPolicyProtections } from '../../tasks/endpoint_policy';
import { createEndpointHost } from '../../tasks/create_endpoint_host';
import { deleteAllLoadedEndpointData } from '../../tasks/delete_all_endpoint_data';

describe('Response console', { tags: ['@ess', '@serverless'] }, () => {
describe('Response console', { tags: ['@ess', '@serverless', '@brokenInServerless'] }, () => {
beforeEach(() => {
login();
});