Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution][Endpoint] Update to Endpoint List and associated supporting API to be space aware #194312

Merged
Changes from 44 commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
9382f7d
Endpoint list API updated to return hosts scopped to current space
paul-tavares Sep 23, 2024
79db62d
Enable space awareness for endpoint agent status API/Service
paul-tavares Sep 24, 2024
1b5e7e0
Fleet: added new `getByIds()` service method for Agents
paul-tavares Sep 26, 2024
04b3089
Change `EndpointFleetServicesFactory#asInternalUser()` method + adjus…
paul-tavares Sep 26, 2024
e18c9df
Host metadata API/service (single) updated to support spaces
paul-tavares Sep 26, 2024
0dfe39c
Added `ensureInCurrentSpace()` to fleet services and refactored endpo…
paul-tavares Sep 27, 2024
36af78c
Updated Policy response API to be space aware
paul-tavares Sep 27, 2024
33c0bc9
update to jsdoc comment
paul-tavares Sep 27, 2024
967b620
Fix calls to `createInternalScopedSoClient()` due to rebase from main
paul-tavares Sep 27, 2024
ce76ed0
Fleet: tests for agent service new method
paul-tavares Sep 30, 2024
aab3353
Tests for agent status api handler
paul-tavares Sep 30, 2024
339c569
Fleet: Add missing members to agent service mocks
paul-tavares Sep 30, 2024
c626af6
Tests for metadata api handler
paul-tavares Sep 30, 2024
97f44fa
Tests for policy response api + service
paul-tavares Oct 1, 2024
e02cd6b
Merge remote-tracking branch 'upstream/main' into task/olm-8538-endpo…
paul-tavares Oct 1, 2024
373b59e
Tests for the metadata service + mock and generator updates
paul-tavares Oct 2, 2024
6964f1a
Tests for Endpoint Agent Status
paul-tavares Oct 2, 2024
32ca590
Merge remote-tracking branch 'upstream/main' into task/olm-8538-endpo…
paul-tavares Oct 2, 2024
87f7093
Fix type in policy response route registration
paul-tavares Oct 2, 2024
ec157ac
Fix types in tests for policy response handlers
paul-tavares Oct 2, 2024
410ad02
Fix API integration test
paul-tavares Oct 3, 2024
5a496b5
Merge remote-tracking branch 'upstream/main' into task/olm-8538-endpo…
paul-tavares Oct 3, 2024
641711b
Improve data loader for fleet server and also formatted axios error
paul-tavares Oct 3, 2024
ee77054
Improve FTR endpoint test services to enable loading data by space
paul-tavares Oct 3, 2024
c41d971
Initial set of FTR test setup (not yet implemented) for spaces
paul-tavares Oct 3, 2024
81ea666
Rename variable
paul-tavares Oct 3, 2024
0a825b4
[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…
kibanamachine Oct 3, 2024
3fe80f3
Re-structure space awareness API integration tests
paul-tavares Oct 7, 2024
6743866
Fix up scripts `spaces` module to not use `KbnClient` internal `space…
paul-tavares Oct 7, 2024
7591d0f
Update test data loaders for endpoint hosts, fleet agent and fleet se…
paul-tavares Oct 7, 2024
d601588
FTR API integration tests for space awareness
paul-tavares Oct 7, 2024
b3ddf26
Merge branch 'main' into task/olm-8538-endpoint-list-api-support-spac…
paul-tavares Oct 7, 2024
d83ee04
Merge remote-tracking branch 'origin/task/olm-8538-endpoint-list-api-…
paul-tavares Oct 7, 2024
96a40a3
Enable serverless config for spaces
paul-tavares Oct 8, 2024
45f026f
Merge branch 'main' into task/olm-8538-endpoint-list-api-support-spac…
paul-tavares Oct 8, 2024
bdb7378
enable spaces in FTR serverless config
paul-tavares Oct 8, 2024
4a6bf16
Merge remote-tracking branch 'origin/task/olm-8538-endpoint-list-api-…
paul-tavares Oct 8, 2024
9fb15b4
Add utility for logging supertest errors
paul-tavares Oct 8, 2024
cf4ce36
Fix failing tests in serverless
paul-tavares Oct 8, 2024
c45b17b
Merge branch 'main' into task/olm-8538-endpoint-list-api-support-spac…
paul-tavares Oct 8, 2024
38b0a93
Merge branch 'main' into task/olm-8538-endpoint-list-api-support-spac…
paul-tavares Oct 9, 2024
c9f8c7e
Updated endpoint hosts indexer to now include the initial policy id r…
paul-tavares Oct 9, 2024
12f5b9e
Merge branch 'main' into task/olm-8538-endpoint-list-api-support-spac…
paul-tavares Oct 16, 2024
bfbb682
Merge branch 'main' into task/olm-8538-endpoint-list-api-support-spac…
paul-tavares Oct 16, 2024
ae5c0f4
Adjustments from code review
paul-tavares Oct 16, 2024
4f60af2
Merge remote-tracking branch 'origin/task/olm-8538-endpoint-list-api-…
paul-tavares Oct 16, 2024
b9b7f54
Merge remote-tracking branch 'upstream/main' into task/olm-8538-endpo…
paul-tavares Oct 16, 2024
081389b
Merge branch 'main' into task/olm-8538-endpoint-list-api-support-spac…
paul-tavares Oct 16, 2024
3b741e6
Merge branch 'main' into task/olm-8538-endpoint-list-api-support-spac…
paul-tavares Oct 17, 2024
9ef0bea
Merge branch 'main' into task/olm-8538-endpoint-list-api-support-spac…
paul-tavares Oct 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .buildkite/ftr_security_serverless_configs.yml
Original file line number Diff line number Diff line change
@@ -98,6 +98,7 @@ enabled:
- x-pack/test/security_solution_api_integration/test_suites/edr_workflows/policy_response/trial_license_complete_tier/configs/serverless.config.ts
- x-pack/test/security_solution_api_integration/test_suites/edr_workflows/resolver/trial_license_complete_tier/configs/serverless.config.ts
- x-pack/test/security_solution_api_integration/test_suites/edr_workflows/response_actions/trial_license_complete_tier/configs/serverless.config.ts
- x-pack/test/security_solution_api_integration/test_suites/edr_workflows/spaces/trial_license_complete_tier/configs/serverless.config.ts
- x-pack/test/security_solution_endpoint/configs/serverless.endpoint.config.ts
- x-pack/test/security_solution_endpoint/configs/serverless.integrations.config.ts
# serverless config files that run deployment-agnostic tests
1 change: 1 addition & 0 deletions .buildkite/ftr_security_stateful_configs.yml
Original file line number Diff line number Diff line change
@@ -81,6 +81,7 @@ enabled:
- x-pack/test/security_solution_api_integration/test_suites/edr_workflows/policy_response/trial_license_complete_tier/configs/ess.config.ts
- x-pack/test/security_solution_api_integration/test_suites/edr_workflows/resolver/trial_license_complete_tier/configs/ess.config.ts
- x-pack/test/security_solution_api_integration/test_suites/edr_workflows/response_actions/trial_license_complete_tier/configs/ess.config.ts
- x-pack/test/security_solution_api_integration/test_suites/edr_workflows/spaces/trial_license_complete_tier/configs/ess.config.ts
- x-pack/test/security_solution_endpoint/configs/endpoint.config.ts
- x-pack/test/security_solution_endpoint/configs/integrations.config.ts
- x-pack/test/api_integration/apis/cloud_security_posture/config.ts
4 changes: 2 additions & 2 deletions x-pack/plugins/fleet/common/errors.ts
Original file line number Diff line number Diff line change
@@ -8,9 +8,9 @@

import type { FleetErrorType } from './types';

export class FleetError extends Error {
export class FleetError<TMeta = unknown> extends Error {
attributes?: { type: FleetErrorType };
constructor(message?: string, public readonly meta?: unknown) {
constructor(message?: string, public readonly meta?: TMeta) {
super(message);
this.name = this.constructor.name; // for stack traces
}
9 changes: 6 additions & 3 deletions x-pack/plugins/fleet/server/errors/index.ts
Original file line number Diff line number Diff line change
@@ -95,7 +95,7 @@ export class FleetEncryptedSavedObjectEncryptionKeyRequired extends FleetError {
export class FleetSetupError extends FleetError {}
export class GenerateServiceTokenError extends FleetError {}
export class FleetUnauthorizedError extends FleetError {}
export class FleetNotFoundError extends FleetError {}
export class FleetNotFoundError<TMeta = unknown> extends FleetError<TMeta> {}
export class FleetTooManyRequestsError extends FleetError {}

export class OutputUnauthorizedError extends FleetError {}
@@ -105,7 +105,7 @@ export class DownloadSourceError extends FleetError {}
export class DeleteUnenrolledAgentsPreconfiguredError extends FleetError {}

// Not found errors
export class AgentNotFoundError extends FleetNotFoundError {}
export class AgentNotFoundError extends FleetNotFoundError<{ agentId: string }> {}
export class AgentPolicyNotFoundError extends FleetNotFoundError {}
export class AgentActionNotFoundError extends FleetNotFoundError {}
export class DownloadSourceNotFound extends FleetNotFoundError {}
@@ -115,7 +115,10 @@ export class SigningServiceNotFoundError extends FleetNotFoundError {}
export class InputNotFoundError extends FleetNotFoundError {}
export class OutputNotFoundError extends FleetNotFoundError {}
export class PackageNotFoundError extends FleetNotFoundError {}
export class PackagePolicyNotFoundError extends FleetNotFoundError {}
export class PackagePolicyNotFoundError extends FleetNotFoundError<{
/** The package policy ID that was not found */
packagePolicyId: string;
}> {}
export class StreamNotFoundError extends FleetNotFoundError {}

export class FleetServerHostUnauthorizedError extends FleetUnauthorizedError {}
Original file line number Diff line number Diff line change
@@ -15,10 +15,12 @@ const createClientMock = (): jest.Mocked<AgentClient> => ({
getAgentStatusForAgentPolicy: jest.fn(),
listAgents: jest.fn(),
getLatestAgentAvailableVersion: jest.fn(),
getByIds: jest.fn(async (..._) => []),
});

const createServiceMock = (): DeeplyMockedKeys<AgentService> => ({
asInternalUser: createClientMock(),
asInternalScopedUser: jest.fn().mockReturnValue(createClientMock()),
asScoped: jest.fn().mockReturnValue(createClientMock()),
});

Original file line number Diff line number Diff line change
@@ -185,6 +185,27 @@ describe('AgentService', () => {
() => new AgentServiceImpl(mockEsClient, mockSoClient).asInternalUser
);
});

describe('asInternalScopedUser', () => {
it('should throw error if no space id is passed', () => {
const agentService = new AgentServiceImpl(
elasticsearchServiceMock.createElasticsearchClient(),
savedObjectsClientMock.create()
);

expect(() => agentService.asInternalScopedUser('')).toThrowError(TypeError);
});

{
const mockEsClient = elasticsearchServiceMock.createElasticsearchClient();
const mockSoClient = savedObjectsClientMock.create();
expectApisToCallServicesSuccessfully(
mockEsClient,
() => mockSoClient,
() => new AgentServiceImpl(mockEsClient, mockSoClient).asInternalUser
);
}
});
});

function expectApisToCallServicesSuccessfully(
36 changes: 35 additions & 1 deletion x-pack/plugins/fleet/server/services/agents/agent_service.ts
Original file line number Diff line number Diff line change
@@ -27,7 +27,7 @@ import { FleetUnauthorizedError } from '../../errors';

import { getCurrentNamespace } from '../spaces/get_current_namespace';

import { getAgentsByKuery, getAgentById } from './crud';
import { getAgentsByKuery, getAgentById, getByIds } from './crud';
import { getAgentStatusById, getAgentStatusForAgentPolicy } from './status';
import { getLatestAvailableAgentVersion } from './versions';

@@ -42,6 +42,11 @@ export interface AgentService {
*/
asScoped(req: KibanaRequest): AgentClient;

/**
* Scoped services to a given space
*/
asInternalScopedUser(spaceId: string): AgentClient;

/**
* Only use for server-side usages (eg. telemetry), should not be used for end users unless an explicit authz check is
* done.
@@ -60,6 +65,12 @@ export interface AgentClient {
*/
getAgent(agentId: string): Promise<Agent>;

/**
* Get multiple agents by id
* @param agentIds
*/
getByIds(agentIds: string[], options?: { ignoreMissing?: boolean }): Promise<Agent[]>;

/**
* Return the status by the Agent's id
*/
@@ -128,6 +139,14 @@ class AgentClientImpl implements AgentClient {
return getAgentById(this.internalEsClient, this.soClient, agentId);
}

public async getByIds(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps rename this to getAgentsByIds?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the feedback @ashokaditya . I named getByIds() to mirror other APIs available from fleet - thus to remain consistent with their naming conventions.

agentIds: string[],
options?: Partial<{ ignoreMissing: boolean }>
): Promise<Agent[]> {
await this.#runPreflight();
return getByIds(this.internalEsClient, this.soClient, agentIds, options);
}

public async getAgentStatusById(agentId: string) {
await this.#runPreflight();
return getAgentStatusById(this.internalEsClient, this.soClient, agentId);
@@ -187,6 +206,21 @@ export class AgentServiceImpl implements AgentService {
);
}

public asInternalScopedUser(spaceId: string): AgentClient {
if (!spaceId) {
throw new TypeError(`spaceId argument is required!`);
}

const soClient = appContextService.getInternalUserSOClientForSpaceId(spaceId);

return new AgentClientImpl(
this.internalEsClient,
soClient,
undefined,
getCurrentNamespace(soClient)
);
}

public get asInternalUser() {
return new AgentClientImpl(this.internalEsClient, this.soClient);
}
63 changes: 61 additions & 2 deletions x-pack/plugins/fleet/server/services/agents/crud.test.ts
Original file line number Diff line number Diff line change
@@ -9,6 +9,10 @@ import type { ElasticsearchClient } from '@kbn/core/server';
import { elasticsearchServiceMock, savedObjectsClientMock } from '@kbn/core/server/mocks';
import { toElasticsearchQuery } from '@kbn/es-query';

import { isSpaceAwarenessEnabled as _isSpaceAwarenessEnabled } from '../spaces/helpers';

import { AgentNotFoundError } from '../..';

import { AGENTS_INDEX } from '../../constants';
import { createAppContextStartContractMock } from '../../mocks';
import type { Agent } from '../../types';
@@ -24,6 +28,7 @@ import {
openPointInTime,
updateAgent,
_joinFilters,
getByIds,
} from './crud';

jest.mock('../audit_logging');
@@ -41,6 +46,7 @@ jest.mock('./versions', () => {
jest.mock('../spaces/helpers');

const mockedAuditLoggingService = auditLoggingService as jest.Mocked<typeof auditLoggingService>;
const isSpaceAwarenessEnabledMock = _isSpaceAwarenessEnabled as jest.Mock;

describe('Agents CRUD test', () => {
const soClientMock = savedObjectsClientMock.create();
@@ -63,13 +69,22 @@ describe('Agents CRUD test', () => {
appContextService.start(mockContract);
});

function getEsResponse(ids: string[], total: number, status: AgentStatus) {
afterEach(() => {
isSpaceAwarenessEnabledMock.mockReset();
});

function getEsResponse(
ids: string[],
total: number,
status: AgentStatus,
generateSource: (id: string) => Partial<Agent> = () => ({})
) {
return {
hits: {
total,
hits: ids.map((id: string) => ({
_id: id,
_source: {},
_source: generateSource(id),
fields: {
status: [status],
},
@@ -513,4 +528,48 @@ describe('Agents CRUD test', () => {
});
});
});

describe(`getByIds()`, () => {
let searchResponse: ReturnType<typeof getEsResponse>;

beforeEach(() => {
searchResponse = getEsResponse(['1', '2'], 2, 'online', (id) => {
return { id, namespaces: ['foo'] };
});
(soClientMock.getCurrentNamespace as jest.Mock).mockReturnValue('foo');
searchMock.mockImplementation(async () => searchResponse);
});

it('should return a list of agents', async () => {
await expect(getByIds(esClientMock, soClientMock, ['1', '2'])).resolves.toEqual([
expect.objectContaining({ id: '1' }),
expect.objectContaining({ id: '2' }),
]);
});

it('should omit agents that are not found if `ignoreMissing` is true', async () => {
searchResponse.hits.hits = [searchResponse.hits.hits[0]];

await expect(
getByIds(esClientMock, soClientMock, ['1', '2'], { ignoreMissing: true })
).resolves.toEqual([expect.objectContaining({ id: '1' })]);
});

it('should error if agent is not found and `ignoreMissing` is false', async () => {
searchResponse.hits.hits = [searchResponse.hits.hits[0]];

await expect(getByIds(esClientMock, soClientMock, ['1', '2'])).rejects.toThrow(
AgentNotFoundError
);
});

it('should error if agent is not part of current space', async () => {
searchResponse.hits.hits[0]._source.namespaces = ['bar'];
isSpaceAwarenessEnabledMock.mockResolvedValue(true);

await expect(getByIds(esClientMock, soClientMock, ['1', '2'])).rejects.toThrow(
AgentNotFoundError
);
});
});
});
40 changes: 40 additions & 0 deletions x-pack/plugins/fleet/server/services/agents/crud.ts
Original file line number Diff line number Diff line change
@@ -415,6 +415,46 @@ export async function getAgentById(
return agentHit;
}

/**
* Get list of agents by `id`. service method performs space awareness checks.
* @param esClient
* @param soClient
* @param agentIds
* @param options
*
* @throws AgentNotFoundError
*/
export const getByIds = async (
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also perhaps rename this to getAgentsByIds

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comment as before - just following Fleet conventions as other getByIds() already exist

esClient: ElasticsearchClient,
soClient: SavedObjectsClientContract,
agentIds: string[],
options?: Partial<{ ignoreMissing: boolean }>
): Promise<Agent[]> => {
const agentsHits = await getAgentsById(esClient, soClient, agentIds);
const currentNamespace = getCurrentNamespace(soClient);
const response: Agent[] = [];

for (const agentHit of agentsHits) {
let throwError = false;

if ('notFound' in agentHit && !options?.ignoreMissing) {
throwError = true;
} else if ((await isAgentInNamespace(agentHit as Agent, currentNamespace)) !== true) {
throwError = true;
}

if (throwError) {
throw new AgentNotFoundError(`Agent ${agentHit.id} not found`, { agentId: agentHit.id });
}

if (!(`notFound` in agentHit)) {
response.push(agentHit);
}
}

return response;
};

async function _filterAgents(
esClient: ElasticsearchClient,
soClient: SavedObjectsClientContract,
4 changes: 3 additions & 1 deletion x-pack/plugins/fleet/server/services/package_policy.ts
Original file line number Diff line number Diff line change
@@ -771,7 +771,9 @@ class PackagePolicyClientImpl implements PackagePolicyClient {
if (options.ignoreMissing && so.error.statusCode === 404) {
return null;
} else if (so.error.statusCode === 404) {
throw new PackagePolicyNotFoundError(`Package policy ${so.id} not found`);
throw new PackagePolicyNotFoundError(`Package policy ${so.id} not found`, {
packagePolicyId: so.id,
});
} else {
throw new FleetError(so.error.message);
}
Original file line number Diff line number Diff line change
@@ -11,9 +11,15 @@ import type { DeepPartial } from 'utility-types';
import { merge } from 'lodash';
import { set } from '@kbn/safer-lodash-set';
import { gte } from 'semver';
import type { Agent } from '@kbn/fleet-plugin/common';
import type { EndpointCapabilities } from '../service/response_actions/constants';
import { BaseDataGenerator } from './base_data_generator';
import type { HostMetadataInterface, OSFields, HostInfoInterface } from '../types';
import type {
HostMetadataInterface,
OSFields,
HostInfoInterface,
UnitedAgentMetadataPersistedData,
} from '../types';
import { EndpointStatus, HostPolicyResponseActionStatus, HostStatus } from '../types';

export interface GetCustomEndpointMetadataGeneratorOptions {
@@ -226,6 +232,30 @@ export class EndpointMetadataGenerator extends BaseDataGenerator {
return merge(hostInfo, overrides);
}

generateUnitedAgentMetadata(
overrides: DeepPartial<UnitedAgentMetadataPersistedData> = {}
): UnitedAgentMetadataPersistedData {
const endpointMetadata = this.generate();

return merge(
{
agent: {
id: endpointMetadata.agent.id,
},
united: {
endpoint: endpointMetadata,
agent: {
agent: {
id: endpointMetadata.agent.id,
},
policy_id: this.seededUUIDv4(),
} as Agent,
},
} as UnitedAgentMetadataPersistedData,
overrides
);
}

protected randomOsFields(): OSFields {
return this.randomChoice([
EndpointMetadataGenerator.windowsOSFields,
Loading