Skip to content

Commit

Permalink
Update tests, fixtures, and instructions
Browse files Browse the repository at this point in the history
  • Loading branch information
jen-huang committed Apr 23, 2020
1 parent d52f625 commit d30e087
Show file tree
Hide file tree
Showing 8 changed files with 110 additions and 99 deletions.
89 changes: 50 additions & 39 deletions x-pack/plugins/ingest_manager/README.md
Original file line number Diff line number Diff line change
@@ -1,60 +1,70 @@
# Ingest Manager

## Plugin
- The plugin is disabled by default. See the TypeScript type for the [the available plugin configuration options](https://github.com/elastic/kibana/blob/master/x-pack/plugins/ingest_manager/common/types/index.ts#L9-L27)
- Setting `xpack.ingestManager.enabled=true` enables the plugin including the EPM and Fleet features. It also adds the `DATASOURCE_API_ROUTES` and `AGENT_CONFIG_API_ROUTES` values in [`common/constants/routes.ts`](./common/constants/routes.ts)
- Adding `--xpack.ingestManager.epm.enabled=false` will disable the EPM API & UI
- Adding `--xpack.ingestManager.fleet.enabled=false` will disable the Fleet API & UI
- [code for adding the routes](https://github.com/elastic/kibana/blob/1f27d349533b1c2865c10c45b2cf705d7416fb36/x-pack/plugins/ingest_manager/server/plugin.ts#L115-L133)
- [Integration tests](server/integration_tests/router.test.ts)
- Both EPM and Fleet require `ingestManager` be enabled. They are not standalone features.

- The plugin is disabled by default. See the TypeScript type for the [the available plugin configuration options](https://github.com/elastic/kibana/blob/master/x-pack/plugins/ingest_manager/common/types/index.ts#L9-L27)
- Setting `xpack.ingestManager.enabled=true` enables the plugin including the EPM and Fleet features. It also adds the `DATASOURCE_API_ROUTES` and `AGENT_CONFIG_API_ROUTES` values in [`common/constants/routes.ts`](./common/constants/routes.ts)
- Adding `--xpack.ingestManager.epm.enabled=false` will disable the EPM API & UI
- Adding `--xpack.ingestManager.fleet.enabled=false` will disable the Fleet API & UI
- [code for adding the routes](https://github.com/elastic/kibana/blob/1f27d349533b1c2865c10c45b2cf705d7416fb36/x-pack/plugins/ingest_manager/server/plugin.ts#L115-L133)
- [Integration tests](server/integration_tests/router.test.ts)
- Both EPM and Fleet require `ingestManager` be enabled. They are not standalone features.

## Development

### Getting started
See the Kibana docs for [how to set up your dev environment](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#setting-up-your-development-environment), [run Elasticsearch](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#running-elasticsearch), and [start Kibana](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#running-kibana)

See the Kibana docs for [how to set up your dev environment](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#setting-up-your-development-environment), [run Elasticsearch](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#running-elasticsearch), and [start Kibana](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#running-kibana)

One common development workflow is:
- Bootstrap Kibana
```
yarn kbn bootstrap
```
- Start Elasticsearch in one shell
```
yarn es snapshot -E xpack.security.authc.api_key.enabled=true
```
- Start Kibana in another shell
```
yarn start --xpack.ingestManager.enabled=true --no-base-path --xpack.endpoint.enabled=true
```

- Bootstrap Kibana
```
yarn kbn bootstrap
```
- Start Elasticsearch in one shell
```
yarn es snapshot -E xpack.security.authc.api_key.enabled=true
```
- Start Kibana in another shell
```
yarn start --xpack.ingestManager.enabled=true --no-base-path --xpack.endpoint.enabled=true
```

This plugin follows the `common`, `server`, `public` structure from the [Architecture Style Guide
](https://github.com/elastic/kibana/blob/master/style_guides/architecture_style_guide.md#file-and-folder-structure). We also follow the pattern of developing feature branches under your personal fork of Kibana.

### API Tests

#### Ingest & Fleet
1. In one terminal, change to the `x-pack` directory and start the test server with
```
node scripts/functional_tests_server.js --config test/api_integration/config.ts
```
1. in a second terminal, run the tests from the Kibana root directory with
```
node scripts/functional_test_runner.js --config x-pack/test/api_integration/config.ts
```

1. In one terminal, change to the `x-pack` directory and start the test server with

```
node scripts/functional_tests_server.js --config test/api_integration/config.js
```

1. in a second terminal, run the tests from the Kibana root directory with
```
node scripts/functional_test_runner.js --config x-pack/test/api_integration/config.js
```

#### EPM
1. In one terminal, change to the `x-pack` directory and start the test server with
```
node scripts/functional_tests_server.js --config test/epm_api_integration/config.ts
```

1. in a second terminal, run the tests from the Kibana root directory with
```
node scripts/functional_test_runner.js --config x-pack/test/epm_api_integration/config.ts
```
1. In one terminal, change to the `x-pack` directory and start the test server with

```
node scripts/functional_tests_server.js --config test/epm_api_integration/config.ts
```

### Staying up-to-date with `master`
While we're developing in the `feature-ingest` feature branch, here's is more information on keeping up to date with upstream kibana.
1. in a second terminal, run the tests from the Kibana root directory with
```
node scripts/functional_test_runner.js --config x-pack/test/epm_api_integration/config.ts
```

### Staying up-to-date with `master`

While we're developing in the `feature-ingest` feature branch, here's is more information on keeping up to date with upstream kibana.

<details>
<summary>merge upstream <code>master</code> into <code>feature-ingest</code></summary>
Expand All @@ -80,6 +90,7 @@ git push origin
## push your changes to upstream feature branch from the terminal; not GitHub UI
git push upstream
```

</details>

See https://github.com/elastic/kibana/pull/37950 for an example.
6 changes: 3 additions & 3 deletions x-pack/test/api_integration/apis/fleet/agents/acks.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,12 +32,12 @@ export default function(providerContext: FtrProviderContext) {
body: { _source: agentDoc },
} = await esClient.get({
index: '.kibana',
id: 'agents:agent1',
id: 'fleet-agents:agent1',
});
agentDoc.agents.access_api_key_id = apiKey.id;
agentDoc['fleet-agents'].access_api_key_id = apiKey.id;
await esClient.update({
index: '.kibana',
id: 'agents:agent1',
id: 'fleet-agents:agent1',
refresh: 'true',
body: {
doc: agentDoc,
Expand Down
2 changes: 1 addition & 1 deletion x-pack/test/api_integration/apis/fleet/agents/actions.ts
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ export default function(providerContext: FtrProviderContext) {
},
})
.expect(404);
expect(apiResponse.message).to.eql('Saved object [agents/agent100] not found');
expect(apiResponse.message).to.eql('Saved object [fleet-agents/agent100] not found');
});
});
}
6 changes: 3 additions & 3 deletions x-pack/test/api_integration/apis/fleet/agents/checkin.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,12 +32,12 @@ export default function(providerContext: FtrProviderContext) {
body: { _source: agentDoc },
} = await esClient.get({
index: '.kibana',
id: 'agents:agent1',
id: 'fleet-agents:agent1',
});
agentDoc.agents.access_api_key_id = apiKey.id;
agentDoc['fleet-agents'].access_api_key_id = apiKey.id;
await esClient.update({
index: '.kibana',
id: 'agents:agent1',
id: 'fleet-agents:agent1',
refresh: 'true',
body: {
doc: agentDoc,
Expand Down
6 changes: 3 additions & 3 deletions x-pack/test/api_integration/apis/fleet/agents/enroll.ts
Original file line number Diff line number Diff line change
Expand Up @@ -33,13 +33,13 @@ export default function(providerContext: FtrProviderContext) {
body: { _source: enrollmentApiKeyDoc },
} = await esClient.get({
index: '.kibana',
id: 'enrollment_api_keys:ed22ca17-e178-4cfe-8b02-54ea29fbd6d0',
id: 'fleet-enrollment-api-keys:ed22ca17-e178-4cfe-8b02-54ea29fbd6d0',
});
// @ts-ignore
enrollmentApiKeyDoc.enrollment_api_keys.api_key_id = apiKey.id;
enrollmentApiKeyDoc['fleet-enrollment-api-keys'].api_key_id = apiKey.id;
await esClient.update({
index: '.kibana',
id: 'enrollment_api_keys:ed22ca17-e178-4cfe-8b02-54ea29fbd6d0',
id: 'fleet-enrollment-api-keys:ed22ca17-e178-4cfe-8b02-54ea29fbd6d0',
refresh: 'true',
body: {
doc: enrollmentApiKeyDoc,
Expand Down
10 changes: 5 additions & 5 deletions x-pack/test/api_integration/apis/fleet/unenroll_agent.ts
Original file line number Diff line number Diff line change
Expand Up @@ -40,18 +40,18 @@ export default function(providerContext: FtrProviderContext) {
body: { _source: agentDoc },
} = await esClient.get({
index: '.kibana',
id: 'agents:agent1',
id: 'fleet-agents:agent1',
});
// @ts-ignore
agentDoc.agents.access_api_key_id = accessAPIKeyId;
agentDoc.agents.default_api_key_id = outputAPIKeyBody.id;
agentDoc.agents.default_api_key = Buffer.from(
agentDoc['fleet-agents'].access_api_key_id = accessAPIKeyId;
agentDoc['fleet-agents'].default_api_key_id = outputAPIKeyBody.id;
agentDoc['fleet-agents'].default_api_key = Buffer.from(
`${outputAPIKeyBody.id}:${outputAPIKeyBody.api_key}`
).toString('base64');

await esClient.update({
index: '.kibana',
id: 'agents:agent1',
id: 'fleet-agents:agent1',
refresh: 'true',
body: {
doc: agentDoc,
Expand Down
66 changes: 33 additions & 33 deletions x-pack/test/functional/es_archives/fleet/agents/data.json
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
{
"type": "doc",
"value": {
"id": "agents:agent1",
"id": "fleet-agents:agent1",
"index": ".kibana",
"source": {
"type": "agents",
"agents": {
"type": "fleet-agents",
"fleet-agents": {
"access_api_key_id": "api-key-2",
"active": true,
"shared_id": "agent1_filebeat",
Expand All @@ -21,11 +21,11 @@
{
"type": "doc",
"value": {
"id": "agents:agent2",
"id": "fleet-agents:agent2",
"index": ".kibana",
"source": {
"type": "agents",
"agents": {
"type": "fleet-agents",
"fleet-agents": {
"access_api_key_id": "api-key-2",
"active": true,
"shared_id": "agent2_filebeat",
Expand All @@ -40,11 +40,11 @@
{
"type": "doc",
"value": {
"id": "agents:agent3",
"id": "fleet-agents:agent3",
"index": ".kibana",
"source": {
"type": "agents",
"agents": {
"type": "fleet-agents",
"fleet-agents": {
"access_api_key_id": "api-key-3",
"active": true,
"shared_id": "agent3_metricbeat",
Expand All @@ -59,11 +59,11 @@
{
"type": "doc",
"value": {
"id": "agents:agent4",
"id": "fleet-agents:agent4",
"index": ".kibana",
"source": {
"type": "agents",
"agents": {
"type": "fleet-agents",
"fleet-agents": {
"access_api_key_id": "api-key-4",
"active": true,
"shared_id": "agent4_metricbeat",
Expand All @@ -78,17 +78,17 @@
{
"type": "doc",
"value": {
"id": "enrollment_api_keys:ed22ca17-e178-4cfe-8b02-54ea29fbd6d0",
"id": "fleet-enrollment-api-keys:ed22ca17-e178-4cfe-8b02-54ea29fbd6d0",
"index": ".kibana",
"source": {
"enrollment_api_keys" : {
"fleet-enrollment-api-keys" : {
"created_at" : "2019-10-10T16:31:12.518Z",
"name": "FleetEnrollmentKey:1",
"api_key_id" : "key",
"config_id" : "policy:1",
"active" : true
},
"type" : "enrollment_api_keys",
"type" : "fleet-enrollment-api-keys",
"references": []
}
}
Expand All @@ -97,11 +97,11 @@
{
"type": "doc",
"value": {
"id": "events:event1",
"id": "fleet-agent-events:event1",
"index": ".kibana",
"source": {
"type": "agent_events",
"agent_events": {
"type": "fleet-agent-events",
"fleet-agent-events": {
"agent_id": "agent1",
"type": "STATE",
"subtype": "STARTED",
Expand All @@ -116,11 +116,11 @@
{
"type": "doc",
"value": {
"id": "events:event2",
"id": "fleet-agent-events:event2",
"index": ".kibana",
"source": {
"type": "agent_events",
"agent_events": {
"type": "fleet-agent-events",
"fleet-agent-events": {
"agent_id": "agent1",
"type": "STATE",
"subtype": "STOPPED",
Expand All @@ -135,11 +135,11 @@
{
"type": "doc",
"value": {
"id": "agent_actions:37ed51ff-e80f-4f2a-a62d-f4fa975e7d85",
"id": "fleet-agent-actions:37ed51ff-e80f-4f2a-a62d-f4fa975e7d85",
"index": ".kibana",
"source": {
"type": "agent_actions",
"agent_actions": {
"type": "fleet-agent-actions",
"fleet-agent-actions": {
"agent_id": "agent1",
"created_at": "2019-09-04T15:04:07+0000",
"type": "RESUME",
Expand All @@ -152,11 +152,11 @@
{
"type": "doc",
"value": {
"id": "agent_actions:b400439c-bbbf-43d5-83cb-cf8b7e32506f",
"id": "fleet-agent-actions:b400439c-bbbf-43d5-83cb-cf8b7e32506f",
"index": ".kibana",
"source": {
"type": "agent_actions",
"agent_actions": {
"type": "fleet-agent-actions",
"fleet-agent-actions": {
"agent_id": "agent1",
"type": "PAUSE",
"created_at": "2019-09-04T15:01:07+0000",
Expand All @@ -169,11 +169,11 @@
{
"type": "doc",
"value": {
"id": "agent_actions:48cebde1-c906-4893-b89f-595d943b72a1",
"id": "fleet-agent-actions:48cebde1-c906-4893-b89f-595d943b72a1",
"index": ".kibana",
"source": {
"type": "agent_actions",
"agent_actions": {
"type": "fleet-agent-actions",
"fleet-agent-actions": {
"agent_id": "agent1",
"type": "CONFIG_CHANGE",
"created_at": "2020-03-15T03:47:15.129Z",
Expand All @@ -186,11 +186,11 @@
{
"type": "doc",
"value": {
"id": "agent_actions:48cebde1-c906-4893-b89f-595d943b72a2",
"id": "fleet-agent-actions:48cebde1-c906-4893-b89f-595d943b72a2",
"index": ".kibana",
"source": {
"type": "agent_actions",
"agent_actions": {
"type": "fleet-agent-actions",
"fleet-agent-actions": {
"agent_id": "agent1",
"type": "CONFIG_CHANGE",
"created_at": "2020-03-15T03:47:15.129Z",
Expand Down
Loading

0 comments on commit d30e087

Please sign in to comment.