Skip to content

Commit

Permalink
[8.x] [Security Solution][Endpoint] Cypress test improvements to capt…
Browse files Browse the repository at this point in the history
…ure Agent diagnostics file when test fails (#202965) (#204485)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Security Solution][Endpoint] Cypress test improvements to capture
Agent diagnostics file when test fails
(#202965)](#202965)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Paul
Tavares","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-12-12T15:21:23Z","message":"[Security
Solution][Endpoint] Cypress test improvements to capture Agent
diagnostics file when test fails (#202965)\n\n## Summary\r\n\r\n- the
Cypress `parallel` runner was updated to set tooling logging
level\r\nfirst from Env. variables before falling back to the value
defined in\r\nthe Cypress configuration file\r\n- The env. value to set,
if wanting to enable a specific logging level,\r\nis
`TOOLING_LOG_LEVEL`. The values supported are the same as those
used\r\nwith
`ToolingLog`\r\n([here](https://github.com/elastic/kibana/blob/b6287708f687d4e3288851052c0c6ae4ade8ce60/packages/kbn-tooling-log/src/log_levels.ts#L10)):\r\n`silent`,
`error`, `warning`, `success`, `info`, `debug`, `verbose`\r\n- This
change makes it easier to run Cypress tests locally with
(for\r\nexample) a logging level of `verbose` for our tooling without
having to\r\nmodify the Cypress configuration file. Example:
`export\r\nTOOLING_LOG_LEVEL=verbose && yarn cypress:dw:open`\r\n- Added
two new methods to our scripting VM service clients (for Vagrant\r\nand
Multipass):\r\n- `download`: allow you to pull files out of the VM and
save them\r\nlocally\r\n- `upload`: uploads a local file to the VM.
(upload already existed as\r\n`transfer` - which has now been marked as
deprecated).\r\n- Added new service function on our Fleet scripting
module to enable us\r\nto set the logging level on a Fleet Agent\r\n-
Cypress tests were adjusted to automatically set the agent logging
to\r\ndebug when running in CI\r\n- A new Cypress task that allows for
an Agent Diagnostic file (which\r\nincludes the Endpoint Log) to be
retrieved from the host VM and stored\r\nwith the CI job (under the
artifacts tab)\r\n - A few tests were updated to include this step for
failed
test","sha":"2ab8a5ced075da08f3aeb9526a76b60b1a964602","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport
missing","v9.0.0","Team:Defend
Workflows","backport:prev-minor","v8.18.0"],"number":202965,"url":"https://github.com/elastic/kibana/pull/202965","mergeCommit":{"message":"[Security
Solution][Endpoint] Cypress test improvements to capture Agent
diagnostics file when test fails (#202965)\n\n## Summary\r\n\r\n- the
Cypress `parallel` runner was updated to set tooling logging
level\r\nfirst from Env. variables before falling back to the value
defined in\r\nthe Cypress configuration file\r\n- The env. value to set,
if wanting to enable a specific logging level,\r\nis
`TOOLING_LOG_LEVEL`. The values supported are the same as those
used\r\nwith
`ToolingLog`\r\n([here](https://github.com/elastic/kibana/blob/b6287708f687d4e3288851052c0c6ae4ade8ce60/packages/kbn-tooling-log/src/log_levels.ts#L10)):\r\n`silent`,
`error`, `warning`, `success`, `info`, `debug`, `verbose`\r\n- This
change makes it easier to run Cypress tests locally with
(for\r\nexample) a logging level of `verbose` for our tooling without
having to\r\nmodify the Cypress configuration file. Example:
`export\r\nTOOLING_LOG_LEVEL=verbose && yarn cypress:dw:open`\r\n- Added
two new methods to our scripting VM service clients (for Vagrant\r\nand
Multipass):\r\n- `download`: allow you to pull files out of the VM and
save them\r\nlocally\r\n- `upload`: uploads a local file to the VM.
(upload already existed as\r\n`transfer` - which has now been marked as
deprecated).\r\n- Added new service function on our Fleet scripting
module to enable us\r\nto set the logging level on a Fleet Agent\r\n-
Cypress tests were adjusted to automatically set the agent logging
to\r\ndebug when running in CI\r\n- A new Cypress task that allows for
an Agent Diagnostic file (which\r\nincludes the Endpoint Log) to be
retrieved from the host VM and stored\r\nwith the CI job (under the
artifacts tab)\r\n - A few tests were updated to include this step for
failed
test","sha":"2ab8a5ced075da08f3aeb9526a76b60b1a964602"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/202965","number":202965,"mergeCommit":{"message":"[Security
Solution][Endpoint] Cypress test improvements to capture Agent
diagnostics file when test fails (#202965)\n\n## Summary\r\n\r\n- the
Cypress `parallel` runner was updated to set tooling logging
level\r\nfirst from Env. variables before falling back to the value
defined in\r\nthe Cypress configuration file\r\n- The env. value to set,
if wanting to enable a specific logging level,\r\nis
`TOOLING_LOG_LEVEL`. The values supported are the same as those
used\r\nwith
`ToolingLog`\r\n([here](https://github.com/elastic/kibana/blob/b6287708f687d4e3288851052c0c6ae4ade8ce60/packages/kbn-tooling-log/src/log_levels.ts#L10)):\r\n`silent`,
`error`, `warning`, `success`, `info`, `debug`, `verbose`\r\n- This
change makes it easier to run Cypress tests locally with
(for\r\nexample) a logging level of `verbose` for our tooling without
having to\r\nmodify the Cypress configuration file. Example:
`export\r\nTOOLING_LOG_LEVEL=verbose && yarn cypress:dw:open`\r\n- Added
two new methods to our scripting VM service clients (for Vagrant\r\nand
Multipass):\r\n- `download`: allow you to pull files out of the VM and
save them\r\nlocally\r\n- `upload`: uploads a local file to the VM.
(upload already existed as\r\n`transfer` - which has now been marked as
deprecated).\r\n- Added new service function on our Fleet scripting
module to enable us\r\nto set the logging level on a Fleet Agent\r\n-
Cypress tests were adjusted to automatically set the agent logging
to\r\ndebug when running in CI\r\n- A new Cypress task that allows for
an Agent Diagnostic file (which\r\nincludes the Endpoint Log) to be
retrieved from the host VM and stored\r\nwith the CI job (under the
artifacts tab)\r\n - A few tests were updated to include this step for
failed
test","sha":"2ab8a5ced075da08f3aeb9526a76b60b1a964602"}},{"branch":"8.x","label":"v8.18.0","labelRegex":"^v8.18.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
  • Loading branch information
paul-tavares authored Dec 17, 2024
1 parent 665e336 commit 0350dd2
Show file tree
Hide file tree
Showing 12 changed files with 461 additions and 31 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@

import type { CasePostRequest } from '@kbn/cases-plugin/common/api';
import type { UsageRecord } from '@kbn/security-solution-serverless/server/types';
import type { HostVmTransferResponse } from '../../../scripts/endpoint/common/types';
import type {
DeletedEndpointHeartbeats,
IndexedEndpointHeartbeats,
Expand All @@ -30,6 +31,7 @@ import type {
UninstallAgentFromHostTaskOptions,
IsAgentAndEndpointUninstalledFromHostTaskOptions,
LogItTaskOptions,
CaptureHostVmAgentDiagnosticsOptions,
} from './types';
import type {
DeleteIndexedFleetEndpointPoliciesResponse,
Expand Down Expand Up @@ -267,6 +269,12 @@ declare global {
arg: LogItTaskOptions,
options?: Partial<Loggable & Timeoutable>
): Chainable<null>;

task(
name: 'captureHostVmAgentDiagnostics',
arg: CaptureHostVmAgentDiagnosticsOptions,
options?: Partial<Loggable & Timeoutable>
): Chainable<Omit<HostVmTransferResponse, 'delete'>>;
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,15 @@ describe('Response console', { tags: ['@ess', '@serverless'] }, () => {
}
});

afterEach(function () {
if (Cypress.env('IS_CI') && this.currentTest?.isFailed() && createdHost) {
cy.task('captureHostVmAgentDiagnostics', {
hostname: createdHost.hostname,
fileNamePrefix: this.currentTest?.fullTitle(),
});
}
});

it('"get-file --path" - should retrieve a file', () => {
const downloadsFolder = Cypress.config('downloadsFolder');

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,10 @@ import type { CasePostRequest } from '@kbn/cases-plugin/common';
import execa from 'execa';
import type { KbnClient } from '@kbn/test';
import type { ToolingLog } from '@kbn/tooling-log';
import { REPO_ROOT } from '@kbn/repo-info';
// This is a Cypress module and only used by Cypress, so disabling "should" be safe
// eslint-disable-next-line import/no-nodejs-modules
import { mkdir } from 'node:fs/promises';
import type { IndexedEndpointHeartbeats } from '../../../../common/endpoint/data_loaders/index_endpoint_hearbeats';
import {
deleteIndexedEndpointHeartbeats,
Expand Down Expand Up @@ -53,6 +57,7 @@ import type {
LoadUserAndRoleCyTaskOptions,
CreateUserAndRoleCyTaskOptions,
LogItTaskOptions,
CaptureHostVmAgentDiagnosticsOptions,
} from '../types';
import type {
DeletedIndexedEndpointRuleAlerts,
Expand All @@ -75,6 +80,7 @@ import {
deleteAgentPolicy,
fetchAgentPolicyEnrollmentKey,
getOrCreateDefaultAgentPolicy,
setAgentLoggingLevel,
} from '../../../../scripts/endpoint/common/fleet_services';
import { startElasticAgentWithDocker } from '../../../../scripts/endpoint/common/elastic_agent_service';
import type { IndexedFleetEndpointPolicyResponse } from '../../../../common/endpoint/data_loaders/index_fleet_endpoint_policy';
Expand Down Expand Up @@ -433,6 +439,7 @@ ${s1Info.status}
log,
kbnClient,
});
await setAgentLoggingLevel(kbnClient, newHost.agentId, 'debug', log);
await waitForEndpointToStreamData(kbnClient, newHost.agentId, 360000);
return newHost;
} catch (err) {
Expand Down Expand Up @@ -531,5 +538,65 @@ ${s1Info.status}
await startEndpointHost(hostName);
return null;
},

/**
* Generates an Agent Diagnostics archive (ZIP) directly on the Host VM and saves it to a directory
* that is then included with the list of Artifacts that are captured with Buildkite job.
*
* ### Usage:
*
* This task is best used from a `afterEach()` by checking if the test failed and if so (and it
* was a test that was running against a host VM), then capture the diagnostics file
*
* @param hostname
*
* @example
*
* describe('something', () => {
* let hostVm;
*
* afterEach(function() { // << Important: Note the use of `function()` here instead of arrow function
* if (this.currentTest?.isFailed() && hostVm) {
* cy.task('captureHostVmAgentDiagnostics', { hostname: hostVm.hostname });
* }
* });
*
* //...
* })
*/
captureHostVmAgentDiagnostics: async ({
hostname,
fileNamePrefix = '',
}: CaptureHostVmAgentDiagnosticsOptions) => {
const { log } = await stackServicesPromise;

log.info(`Capturing agent diagnostics for host VM [${hostname}]`);

const vmClient = getHostVmClient(hostname, undefined, undefined, log);
const fileName = `elastic-agent-diagnostics-${hostname}-${new Date()
.toISOString()
.replace(/:/g, '.')}.zip`;
const vmDiagnosticsFile = `/tmp/${fileName}`;
const localDiagnosticsDir = `${REPO_ROOT}/target/test_failures`;
const localDiagnosticsFile = `${localDiagnosticsDir}/${
fileNamePrefix
? // Insure the file name prefix does not have characters that can't be used in file names
`${fileNamePrefix.replace(/[><:"/\\|?*'`{} ]/g, '_')}-`
: ''
}${fileName}`;

await mkdir(localDiagnosticsDir, { recursive: true });

// generate diagnostics file on the host and then download it
await vmClient.exec(
`sudo /opt/Elastic/Agent/elastic-agent diagnostics --file ${vmDiagnosticsFile}`
);
return vmClient.download(vmDiagnosticsFile, localDiagnosticsFile).then((response) => {
log.info(`Agent diagnostic file for host [${hostname}] has been downloaded and is available at:
${response.filePath}
`);
return { filePath: response.filePath };
});
},
});
};
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,20 @@ export const setupToolingLogLevel = (config: Cypress.PluginConfigOptions) => {
const log = createToolingLogger();
const defaultToolingLogLevel = config.env.TOOLING_LOG_LEVEL;

log.info(`Cypress config 'env.TOOLING_LOG_LEVEL': ${defaultToolingLogLevel}`);
log.info(`
Cypress Configuration File: ${config.configFile}
'env.TOOLING_LOG_LEVEL' set to: ${defaultToolingLogLevel}
*** FYI: *** To help with test failures, an environmental variable named 'TOOLING_LOG_LEVEL' can be set
with a value of 'verbose' in order to capture more data in the logs. This environment
property can be set either in the runtime environment (ex. local shell or buildkite) or
directly in the Cypress configuration file \`env: {}\` section.
`);

if (defaultToolingLogLevel && defaultToolingLogLevel !== createToolingLogger.defaultLogLevel) {
createToolingLogger.defaultLogLevel = defaultToolingLogLevel;
log.info(`Default log level for 'createToolingLogger()' set to ${defaultToolingLogLevel}`);
}
};
Original file line number Diff line number Diff line change
Expand Up @@ -81,3 +81,8 @@ export interface LogItTaskOptions {
level: keyof Pick<ToolingLog, 'info' | 'debug' | 'verbose'>;
data: any;
}

export interface CaptureHostVmAgentDiagnosticsOptions {
hostname: string;
fileNamePrefix?: string;
}
Original file line number Diff line number Diff line change
Expand Up @@ -277,17 +277,19 @@ const startFleetServerWithDocker = async ({
const isServerless = await isServerlessKibanaFlavor(kbnClient);
const esURL = new URL(await getFleetElasticsearchOutputHost(kbnClient));
const containerName = `dev-fleet-server.${port}`;
let fleetServerVersionInfo = '';

log.info(
`Starting a new fleet server using Docker\n Agent version: ${agentVersion}\n Server URL: ${fleetServerUrl}`
);

let retryAttempt = isServerless ? 0 : 1;
const attemptServerlessFleetServerSetup = async (): Promise<StartedServer> => {
fleetServerVersionInfo = '';

return log.indent(4, async () => {
const hostname = `dev-fleet-server.${port}.${Math.random().toString(32).substring(2, 6)}`;
let containerId = '';
let fleetServerVersionInfo = '';

if (isLocalhost(esURL.hostname)) {
esURL.hostname = localhostRealIp;
Expand Down Expand Up @@ -361,8 +363,17 @@ const startFleetServerWithDocker = async ({
}

fleetServerVersionInfo = isServerless
? // `/usr/bin/fleet-server` process does not seem to support a `--version` type of argument
'Running latest standalone fleet server'
? (
await execa
.command(`docker exec ${containerName} /usr/bin/fleet-server --version`)
.catch((err) => {
log.verbose(
`Failed to retrieve fleet-server (serverless/standalone) version information from running instance.`,
err
);
return { stdout: 'Unable to retrieve version information (serverless)' };
})
).stdout
: (
await execa('docker', [
'exec',
Expand Down Expand Up @@ -424,7 +435,7 @@ Kill container: ${chalk.cyan(`docker kill ${containerId}`)}

const response: StartedServer = await attemptServerlessFleetServerSetup();

log.info(`Done. Fleet server up and running`);
log.info(`Done. Fleet server up and running (version: ${fleetServerVersionInfo})`);

return response;
};
Expand Down
Loading

0 comments on commit 0350dd2

Please sign in to comment.