Skip to content

Commit

Permalink
[Asset Manager] Creates baseline public asset client for use in publi…
Browse files Browse the repository at this point in the history
…c plugins (#167191)

Closes #167075 

## Summary

Adds a public asset client available in the `setup` lifecycle hook for
plugins that depend on this one. `getHosts` is the only method available
on this client for now.

TODO, before merge:
- [x] Add docs for the server client
- [x] Add docs for the public client
- [x] Remove REST docs from plugin docs, not needed
- [x] Add unit tests for public client


### Testing this PR

One way of testing this new client is to apply the attached
test-assets.patch file locally, adjust the date range in the getHosts
query that is added in the infra plugin, and then start Kibana and
navigate to the infra app. You should see print out in the browser
console.


[test-assets.patch](https://github.com/elastic/kibana/files/12718693/test-assets.patch)

---------

Co-authored-by: kibanamachine <[email protected]>
  • Loading branch information
jasonrhodes and kibanamachine authored Sep 28, 2023
1 parent 424dec6 commit 859ae9e
Show file tree
Hide file tree
Showing 42 changed files with 758 additions and 302 deletions.
3 changes: 1 addition & 2 deletions docs/developer/plugin-list.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -451,8 +451,7 @@ The plugin exposes the static DefaultEditorController class to consume.
|{kib-repo}blob/{branch}/x-pack/plugins/asset_manager/README.md[assetManager]
|This plugin provides access to the asset data stored in assets-* indices, primarily
for inventory and topology purposes.
|This plugin provides access to observed asset data, such as information about hosts, pods, containers, services, and more.
|{kib-repo}blob/{branch}/x-pack/plugins/banners/README.md[banners]
Expand Down
1 change: 1 addition & 0 deletions packages/kbn-optimizer/limits.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ pageLoadAssetSize:
aiops: 10000
alerting: 106936
apm: 64385
assetManager: 25000
banners: 17946
bfetch: 22837
canvas: 1066647
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,7 @@ export default function ({ getService }: PluginFunctionalProviderContext) {
'xpack.apm.featureFlags.sourcemapApiAvailable (any)',
'xpack.apm.featureFlags.storageExplorerAvailable (any)',
'xpack.apm.serverless.enabled (any)', // It's a boolean (any because schema.conditional)
'xpack.assetManager.alphaEnabled (boolean)',
'xpack.observability_onboarding.serverless.enabled (any)', // It's a boolean (any because schema.conditional)
'xpack.cases.files.allowedMimeTypes (array)',
'xpack.cases.files.maxSize (number)',
Expand Down
36 changes: 5 additions & 31 deletions x-pack/plugins/asset_manager/README.md
Original file line number Diff line number Diff line change
@@ -1,39 +1,13 @@
# Asset Manager Plugin

This plugin provides access to the asset data stored in assets-\* indices, primarily
for inventory and topology purposes.
This plugin provides access to observed asset data, such as information about hosts, pods, containers, services, and more.

## Documentation

See [docs for the provided APIs in the docs folder](./docs/index.md).
### User Docs

## Running Tests
For those interested in making use of the APIs provided by this plugin, see [our API docs](./docs/api.md).

There are integration tests for the endpoints implemented thus far as well as for
the sample data tests. There is also a small set of tests meant to ensure that the
plugin is not doing anything without the proper config value in place to enable
the plugin fully. For more on enabling the plugin, see [the docs page](./docs/index.md).
### Developer Docs

The "not enabled" tests are run by default in CI. To run them manually, do the following:

```shell
$ node scripts/functional_tests_server --config x-pack/test/api_integration/apis/asset_manager/config_when_disabled.ts
$ node scripts/functional_test_runner --config=x-pack/test/api_integration/apis/asset_manager/config_when_disabled.ts
```

The "enabled" tests are NOT run by CI yet, to prevent blocking Kibana development for a
test failure in this alpha, tech preview plugin. They will be moved into the right place
to make them run for CI before the plugin is enabled by default. To run them manually:

```shell
$ node scripts/functional_tests_server --config x-pack/test/api_integration/apis/asset_manager/config.ts
$ node scripts/functional_test_runner --config=x-pack/test/api_integration/apis/asset_manager/config.ts
```

## Using Sample Data

This plugin comes with a full "working set" of sample asset documents, meant
to provide enough data in the correct schema format so that all of the API
endpoints return expected values.

To create the sample data, follow [the instructions in the REST API docs](./docs/index.md#sample-data).
For those working on this plugin directly and developing it, please see [our development docs](./docs/development.md).
56 changes: 56 additions & 0 deletions x-pack/plugins/asset_manager/common/config.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import { schema, TypeOf } from '@kbn/config-schema';

export const INDEX_DEFAULTS = {
logs: 'filebeat-*,logs-*',
};

export const configSchema = schema.object({
alphaEnabled: schema.maybe(schema.boolean()),
// Designate where various types of data live.
// NOTE: this should be handled in a centralized way for observability, so
// that when a user configures these differently from the known defaults,
// that value is propagated everywhere. For now, we duplicate the value here.
sourceIndices: schema.object(
{
logs: schema.string({ defaultValue: INDEX_DEFAULTS.logs }),
},
{ defaultValue: INDEX_DEFAULTS }
),
// Choose an explicit source for asset queries.
// NOTE: This will eventually need to be able to cleverly switch
// between these values based on the availability of data in the
// indices, and possibly for each asset kind/type value.
// For now, we set this explicitly.
lockedSource: schema.oneOf([schema.literal('assets'), schema.literal('signals')], {
defaultValue: 'signals',
}),
});

export type AssetManagerConfig = TypeOf<typeof configSchema>;

/**
* The following map is passed to the server plugin setup under the
* exposeToBrowser: option, and controls which of the above config
* keys are allow-listed to be available in the browser config.
*
* NOTE: anything exposed here will be visible in the UI dev tools,
* and therefore MUST NOT be anything that is sensitive information!
*/
export const exposeToBrowserConfig = {
alphaEnabled: true,
} as const;

type ValidKeys = keyof {
[K in keyof typeof exposeToBrowserConfig as typeof exposeToBrowserConfig[K] extends true
? K
: never]: true;
};

export type AssetManagerPublicConfig = Pick<AssetManagerConfig, ValidKeys>;
18 changes: 18 additions & 0 deletions x-pack/plugins/asset_manager/common/constants_routes.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

export const ASSET_MANAGER_API_BASE = '/api/asset-manager';

function base(path: string) {
return `${ASSET_MANAGER_API_BASE}${path}`;
}

export const GET_ASSETS = base('/assets');
export const GET_RELATED_ASSETS = base('/assets/related');
export const GET_ASSETS_DIFF = base('/assets/diff');

export const GET_HOSTS = base('/assets/hosts');
181 changes: 117 additions & 64 deletions x-pack/plugins/asset_manager/common/types_api.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,77 +6,111 @@
*/

import * as rt from 'io-ts';

export const assetTypeRT = rt.union([
rt.literal('k8s.pod'),
rt.literal('k8s.cluster'),
rt.literal('k8s.node'),
]);
import {
dateRt,
inRangeFromStringRt,
datemathStringRt,
createLiteralValueFromUndefinedRT,
} from '@kbn/io-ts-utils';

export const assetTypeRT = rt.keyof({
'k8s.pod': null,
'k8s.cluster': null,
'k8s.node': null,
});

export type AssetType = rt.TypeOf<typeof assetTypeRT>;

export const assetKindRT = rt.union([
rt.literal('cluster'),
rt.literal('host'),
rt.literal('pod'),
rt.literal('container'),
rt.literal('service'),
rt.literal('alert'),
]);
export const assetKindRT = rt.keyof({
cluster: null,
host: null,
pod: null,
container: null,
service: null,
alert: null,
});

export type AssetKind = rt.TypeOf<typeof assetKindRT>;

export type AssetStatus =
| 'CREATING'
| 'ACTIVE'
| 'DELETING'
| 'FAILED'
| 'UPDATING'
| 'PENDING'
| 'UNKNOWN';
export type CloudProviderName = 'aws' | 'gcp' | 'azure' | 'other' | 'unknown' | 'none';

interface WithTimestamp {
'@timestamp': string;
}
export interface ECSDocument extends WithTimestamp {
'kubernetes.namespace'?: string;
'kubernetes.pod.name'?: string;
'kubernetes.pod.uid'?: string;
'kubernetes.pod.start_time'?: Date;
'kubernetes.node.name'?: string;
'kubernetes.node.start_time'?: Date;

'orchestrator.api_version'?: string;
'orchestrator.namespace'?: string;
'orchestrator.organization'?: string;
'orchestrator.type'?: string;
'orchestrator.cluster.id'?: string;
'orchestrator.cluster.name'?: string;
'orchestrator.cluster.url'?: string;
'orchestrator.cluster.version'?: string;

'cloud.provider'?: CloudProviderName;
'cloud.instance.id'?: string;
'cloud.region'?: string;
'cloud.service.name'?: string;

'service.environment'?: string;
}
export const assetStatusRT = rt.keyof({
CREATING: null,
ACTIVE: null,
DELETING: null,
FAILED: null,
UPDATING: null,
PENDING: null,
UNKNOWN: null,
});

export type AssetStatus = rt.TypeOf<typeof assetStatusRT>;

// https://github.com/gcanti/io-ts/blob/master/index.md#union-of-string-literals
export const cloudProviderNameRT = rt.keyof({
aws: null,
gcp: null,
azure: null,
other: null,
unknown: null,
none: null,
});

export type CloudProviderName = rt.TypeOf<typeof cloudProviderNameRT>;

const withTimestampRT = rt.type({
'@timestamp': rt.string,
});

export type WithTimestamp = rt.TypeOf<typeof withTimestampRT>;

export const ECSDocumentRT = rt.intersection([
withTimestampRT,
rt.partial({
'kubernetes.namespace': rt.string,
'kubernetes.pod.name': rt.string,
'kubernetes.pod.uid': rt.string,
'kubernetes.pod.start_time': rt.string,
'kubernetes.node.name': rt.string,
'kubernetes.node.start_time': rt.string,
'orchestrator.api_version': rt.string,
'orchestrator.namespace': rt.string,
'orchestrator.organization': rt.string,
'orchestrator.type': rt.string,
'orchestrator.cluster.id': rt.string,
'orchestrator.cluster.name': rt.string,
'orchestrator.cluster.url': rt.string,
'orchestrator.cluster.version': rt.string,
'cloud.provider': cloudProviderNameRT,
'cloud.instance.id': rt.string,
'cloud.region': rt.string,
'cloud.service.name': rt.string,
'service.environment': rt.string,
}),
]);

export interface Asset extends ECSDocument {
'asset.collection_version'?: string;
'asset.ean': string;
'asset.id': string;
'asset.kind': AssetKind;
'asset.name'?: string;
'asset.type'?: AssetType;
'asset.status'?: AssetStatus;
'asset.parents'?: string | string[];
'asset.children'?: string | string[];
'asset.references'?: string | string[];
'asset.namespace'?: string;
}
export type ECSDocument = rt.TypeOf<typeof ECSDocumentRT>;

export const assetRT = rt.intersection([
ECSDocumentRT,
rt.type({
'asset.ean': rt.string,
'asset.id': rt.string,
'asset.kind': assetKindRT,
}),
// mixed required and optional require separate hashes combined via intersection
// https://github.com/gcanti/io-ts/blob/master/index.md#mixing-required-and-optional-props
rt.partial({
'asset.collection_version': rt.string,
'asset.name': rt.string,
'asset.type': assetTypeRT,
'asset.status': assetStatusRT,
'asset.parents': rt.union([rt.string, rt.array(rt.string)]),
'asset.children': rt.union([rt.string, rt.array(rt.string)]),
'asset.references': rt.union([rt.string, rt.array(rt.string)]),
'asset.namespace': rt.string,
}),
]);

export type Asset = rt.TypeOf<typeof assetRT>;

export type AssetWithoutTimestamp = Omit<Asset, '@timestamp'>;

Expand Down Expand Up @@ -156,3 +190,22 @@ export type RelationField = keyof Pick<
Asset,
'asset.children' | 'asset.parents' | 'asset.references'
>;

export const sizeRT = rt.union([
inRangeFromStringRt(1, 100),
createLiteralValueFromUndefinedRT(10),
]);
export const assetDateRT = rt.union([dateRt, datemathStringRt]);
export const getHostAssetsQueryOptionsRT = rt.exact(
rt.partial({
from: assetDateRT,
to: assetDateRT,
size: sizeRT,
})
);
export type GetHostAssetsQueryOptions = rt.TypeOf<typeof getHostAssetsQueryOptionsRT>;

export const getHostAssetsResponseRT = rt.type({
hosts: rt.array(assetRT),
});
export type GetHostAssetsResponse = rt.TypeOf<typeof getHostAssetsResponseRT>;
17 changes: 17 additions & 0 deletions x-pack/plugins/asset_manager/common/types_client.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

export interface GetHostsOptionsPublic {
from: string;
to: string;
}

export interface GetServicesOptionsPublic {
from: string;
to: string;
parent?: string;
}
Loading

0 comments on commit 859ae9e

Please sign in to comment.