78024: move transform out of dataset #78216

nnamdifrankie · 2020-09-22T21:02:05Z

Summary

#78024

move transform install from dataset to package root under elasticsearch.
remove datastream configuration for metadata current, it is no longer a datastream.

Documentation

The transform is not related to a dataset and should be placed under the ElasticSearch directory which is the same level as the dataset. Along with other ElasticSearch assets like IndexTemplate. A transform under the dataset directory will be treated as a transform file under the dataset directory. In the future the package specification may be implemented to restrict it to the ElasticSearch directory.
A transform creates the destination index if it does not exist. With this version it is assumed that the lifespan of destination index is tied to the lifespan of the transform if it is managed by Ingest Manager. This means that when the transform is removed from the package the destination index is also deleted as a result. This cycle also occurs during upgrade of a package, and it is expected that transform will start from checkpoint zero. In the future this behavior may be handled by the transform service but will remain a key feature of the system.
Given that the index is created by the transform the best way to apply index mapping and settings to the index is to include an index template in the package that has an index pattern that matches the destination index of the transform. It is possible to have an existing index template with a pattern that matches the destination index for the template, but it is recommended to create new templates and patterns in order to avoid unexpected indexing and search outcomes on the destination index. In addition, if you remove the transform it is important to also remove the index template, as the index will be removed by Ingest Manager. Since the index template is a separate assets it has to be removed from the package to trigger its removal.
For the transform to start the source index must exist, if you are using a concrete index. If you are using a datastream then the source index has to be a wildcard pattern for the transform to be started successfully.

Checklist

Delete any items that are not applicable to this PR.

Unit or functional tests were updated or added to match the most common scenarios

ruflin · 2020-09-23T12:55:00Z

x-pack/plugins/security_solution/common/endpoint/constants.ts

@@ -8,7 +8,7 @@ export const eventsIndexPattern = 'logs-endpoint.events.*';
 export const alertsIndexPattern = 'logs-endpoint.alerts-*';
 export const metadataIndexPattern = 'metrics-endpoint.metadata-*';
 export const metadataCurrentIndexPattern = 'metrics-endpoint.metadata_current-*';


As this will not follow the index pattern, this should change. I'm starting to wonder if we should even put the data under metrics prefix or outside? If we put it under metrics-* it can't contain to -.

elasticmachine · 2020-09-24T17:44:16Z

Pinging @elastic/ingest-management (Team:Ingest Management)

kevinlog · 2020-09-24T20:06:42Z

x-pack/test/security_solution_endpoint/apps/endpoint/endpoint_list.ts

@@ -88,21 +85,21 @@ export default ({ getPageObjects, getService }: FtrProviderContext) => {
      });

      it('finds data after load and polling', async () => {
-        await esArchiver.load('endpoint/metadata/destination_index', { useCreate: true });
-        await pageObjects.endpoint.waitForTableToHaveData('endpointListTable', 1100);
+        await esArchiver.load('endpoint/metadata/api_feature', { useCreate: true });


these tests are skipped right now, so it's OK for now, but I'd rather we not wait for the transform in the integration tests because of the long wait times.

IMO, the UI integration tests can just test that we can call the AP when there's data there and the UI renders correctly

Well that should be tested in unit tests, the integration test should be integration. And ingesting into metadata current may not be a good test but I get the point. To get the test to pass again I had to change a few line that were removed.

Also the old setup will fail since we remove the matadata current stream.

neptunian · 2020-09-28T17:01:02Z

x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/transform/install.ts

-  path: string;
-  dataset: Dataset;
-}
-
 export const installTransformForDataset = async (


maybe this should be renamed to remove ForDataset?

Yes will remove

nnamdifrankie · 2020-09-28T20:55:36Z

@elasticmachine merge upstream

…kibana into 78024_fix_transform_error

kibanamachine · 2020-09-29T14:26:30Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 3ae674c

Metrics [docs]

async chunks size

id	value	diff	baseline
`securitySolution`	10.2MB	-16.0B	10.2MB

History

💚 Build #77712 succeeded 3f31f48
💚 Build #77698 succeeded df94043
💔 Build #77672 failed c11bdd2
💔 Build #76925 failed cf92665
💚 Build #76934 succeeded add0763

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* 78024: move transform out of dataset * Change index prefix * 78024: fix tests, remove vestiges * 78024: remove index defined in the transform when transform is removed. * 78024: clean up * 78024: fix build * 78024: add comment * 78024: remove test I added * 78024: more removal, will add in next PR Co-authored-by: Elastic Machine <[email protected]> # Conflicts: # x-pack/plugins/ingest_manager/server/services/epm/packages/install.ts

[7.x] 78024: move transform out of dataset (#78216)

…-to-timeline * 'master' of github.com:elastic/kibana: (22 commits) update apm index pattern (elastic#78732) 78024: move transform out of dataset (elastic#78216) [QA][Code Coverage] Upload the coverage static site before ingestion (elastic#78695) [Discover] Make _source field not clickable (elastic#78698) [Fleet] Rename Ingest Manager => Fleet, Fleet => Agents in the UI (elastic#78685) [APM] Review feedback from distribution + transaction metrics (elastic#78752) [Ingest pipelines] Add ability to stop pipeline simulation (elastic#78183) [CSM] Fix core vital legend background (elastic#78273) [Usage Collection] [schema] Support spreads + `canvas` definition (elastic#78481) fix lodash imports (elastic#78456) [Maps] Add layer type preview icons (elastic#78650) [APM] Use transaction metrics for distribution charts (elastic#78484) [Uptime] Ml anomaly alert edit (elastic#76909) [ML] Limit exposing shared static code through ml/public/index.ts. (elastic#77745) making expression debug info serializable (elastic#78727) fix lodahs imports in app-arch code (elastic#78582) Make Field a React.lazy export (elastic#78483) [Security Solution] Improves detections tests (elastic#77295) [TSVB] Different field format on different series is ignored (elastic#78138) RFC: Improve saved object migrations (elastic#66056) ...

nnamdifrankie · 2020-10-01T12:24:22Z

https://github.com/elastic/security-team/issues/73

78024: move transform out of dataset

d5ca023

nnamdifrankie added release_note:skip Skip the PR/issue when compiling release notes v7.10.0 labels Sep 22, 2020

ruflin reviewed Sep 23, 2020

View reviewed changes

nnamdifrankie added 3 commits September 23, 2020 10:30

Change index prefix

cf31379

78024: fix tests, remove vestiges

cf92665

Merge branch 'master' into 78024_fix_transform_error

add0763

nnamdifrankie marked this pull request as ready for review September 24, 2020 14:09

nnamdifrankie requested a review from a team as a code owner September 24, 2020 14:09

nnamdifrankie requested a review from a team September 24, 2020 14:09

nnamdifrankie requested a review from a team as a code owner September 24, 2020 14:09

paul-tavares approved these changes Sep 24, 2020

View reviewed changes

botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Sep 24, 2020

kevinlog reviewed Sep 24, 2020

View reviewed changes

neptunian reviewed Sep 28, 2020

View reviewed changes

neptunian approved these changes Sep 28, 2020

View reviewed changes

nnamdifrankie added 2 commits September 28, 2020 16:36

78024: remove index defined in the transform when transform is removed.

0d8e706

78024: clean up

cf033cc

elasticmachine and others added 4 commits September 28, 2020 16:55

Merge branch 'master' into 78024_fix_transform_error

c11bdd2

78024: fix build

d8e93a6

Merge branch '78024_fix_transform_error' of github.com:nnamdifrankie/…

df94043

…kibana into 78024_fix_transform_error

78024: add comment

3f31f48

kevinlog approved these changes Sep 29, 2020

View reviewed changes

nnamdifrankie added 2 commits September 29, 2020 08:43

78024: remove test I added

360487c

78024: more removal, will add in next PR

3ae674c

nnamdifrankie merged commit 9e9a48b into elastic:master Sep 29, 2020

nnamdifrankie deleted the 78024_fix_transform_error branch September 29, 2020 14:33

nnamdifrankie mentioned this pull request Sep 29, 2020

[7.x] 78024: move transform out of dataset (#78216) #78791

Merged

nnamdifrankie added a commit that referenced this pull request Sep 29, 2020

[7.x] 78024: move transform out of dataset (#78216) (#78791)

ce96439

[7.x] 78024: move transform out of dataset (#78216)

kevinlog mentioned this pull request Sep 30, 2020

[Fleet] Ingest Manager is showing an error on the Endpoint package and takes a few refreshes of browser before Ingest can be setup #78024

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

78024: move transform out of dataset #78216

78024: move transform out of dataset #78216

nnamdifrankie commented Sep 22, 2020 •

edited

Loading

ruflin Sep 23, 2020

elasticmachine commented Sep 24, 2020

kevinlog Sep 24, 2020

nnamdifrankie Sep 24, 2020

nnamdifrankie Sep 28, 2020

neptunian Sep 28, 2020

nnamdifrankie Sep 28, 2020

nnamdifrankie commented Sep 28, 2020

kibanamachine commented Sep 29, 2020

nnamdifrankie commented Oct 1, 2020

78024: move transform out of dataset #78216

78024: move transform out of dataset #78216

Conversation

nnamdifrankie commented Sep 22, 2020 • edited Loading

Summary

Documentation

Checklist

ruflin Sep 23, 2020

Choose a reason for hiding this comment

elasticmachine commented Sep 24, 2020

kevinlog Sep 24, 2020

Choose a reason for hiding this comment

nnamdifrankie Sep 24, 2020

Choose a reason for hiding this comment

nnamdifrankie Sep 28, 2020

Choose a reason for hiding this comment

neptunian Sep 28, 2020

Choose a reason for hiding this comment

nnamdifrankie Sep 28, 2020

Choose a reason for hiding this comment

nnamdifrankie commented Sep 28, 2020

kibanamachine commented Sep 29, 2020

💚 Build Succeeded

Metrics [docs]

async chunks size

History

nnamdifrankie commented Oct 1, 2020

nnamdifrankie commented Sep 22, 2020 •

edited

Loading