[Ingest] Add additional attributes to the Datasources Saved Object #66127

paul-tavares · 2020-05-11T21:07:21Z

Summary

Adds 4 additional attributes to the Datasources Saved Object (created/updated_by ++ created/updated_on). In support of this change, a datasources saved object migration was also created for v7.9.0

closes #65904

Checklist

Delete any items that are not applicable to this PR.

UI testing:
- Able to create Datasource from Integrations > [package] > create
- Able to create Datasource from Configurations > [config] > create
- Able to update Datasource from Configurations > [config] > edit data source
- Able to save Endpoint Policy changes (endpoint app)
Manual migration testing
(Elasticsearch + kibana was started using master. Datasources where created using the UI. Source was then switched to the branch containing these changed (Kibana server was restarted). Looking at the API calls made by the UI, I can see the new attributes being returned)

elasticmachine · 2020-05-11T21:07:23Z

Pinging @elastic/ingest-management (Team:Ingest Management)

elasticmachine · 2020-05-11T21:07:24Z

Pinging @elastic/endpoint-management (Team:Endpoint Management)

paul-tavares · 2020-05-11T21:08:44Z

Still draft - need to go through the code usages against the Datasources service and ensure that a user: AuthenticatedUser is being passed in (since its optional)

nchaulet · 2020-05-11T21:13:00Z

x-pack/plugins/ingest_manager/server/saved_objects/index.ts

@@ -218,8 +219,15 @@ const savedObjectTypes: { [key: string]: SavedObjectsType } = {
          },
        },
        revision: { type: 'integer' },
+        updated_on: { type: 'keyword' },
+        updated_by: { type: 'keyword' },
+        created_on: { type: 'keyword' },


Why not using the type date?

Good point @nchaulet . I copied the definition from the Agent Config SO which had keyword, but now see that Enrollment API Keys SO does use date. I will change it.
Thanks 👍

ruflin · 2020-05-12T06:49:39Z

x-pack/plugins/ingest_manager/server/saved_objects/index.ts

      },
    },
+    migrations: {


Even though I don't expect users to migrate from 7.8 to 7.9 I like that we get into the habit of providing a migration path! 👍

…-datasources-additional-fields

paul-tavares · 2020-05-12T14:30:05Z

Ok. I think I have captured all the needed updates for this PR. Its ready for review 😬

nchaulet · 2020-05-12T15:01:50Z

we have a mix of updated_on and updated_at|created_at looking at other plugins looks like *_at is more used what do you think of changing this? cc @jen-huang I think we should do some cleanup on our side too.

nchaulet · 2020-05-12T15:02:33Z

x-pack/plugins/endpoint/common/generate_data.ts

@@ -504,10 +504,15 @@ export class EndpointDocGenerator {
   * Generates an Ingest `datasource` that includes the Endpoint Policy data
   */
  public generatePolicyDatasource(): PolicyData {
+    const created = new Date(Date.now() - 8.64e7).toISOString(); // 24h ago


I imagine is just for testing purpose?

Yes, test purposes to mock up data,

paul-tavares · 2020-05-12T15:21:10Z

@nchaulet I'm good with changing the attributes you mentioned to the more commonly used names 👍

nchaulet

LGTM for the ingest part (if you can make the change to updated_at|created_at)

jen-huang · 2020-05-12T18:55:20Z

we have a mix of updated_on and updated_at|created_at looking at other plugins looks like *_at is more used what do you think of changing this? cc @jen-huang I think we should do some cleanup on our side too.

++ on using *_at. Looks like we only have one other *_on property (agent config's updated_on). We should normalize this sooner rather than later. Once this PR goes in, @nchaulet can you work on updating that property and adding a migration for it?

…-datasources-additional-fields

kibanamachine · 2020-05-12T21:22:13Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 1198481

History

💚 Build #47321 succeeded 78c8194
💚 Build #47119 succeeded 9eedd02

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…lastic#66127) * add additional properties to ingest-datasources SO * Adjusted Types and test generators * Added datasources migrations to SO * Add `user` object to calls to datasource.create()

…66127) (#66313) * add additional properties to ingest-datasources SO * Adjusted Types and test generators * Added datasources migrations to SO * Add `user` object to calls to datasource.create()

* upstream/master: (223 commits) [Ingest] Support root level yaml variables in agent stream template (elastic#66120) [Snapshot Restore] Fix error when deleting snapshots behind reverse proxy (elastic#66147) [Lens] fix empty state for pie (elastic#66206) [APM] Improve e2e tests (elastic#66373) [ML] Data Frame Analytics: Fix steps to be named phases. (elastic#65855) [Discover] Encode context link filter part (elastic#63831) [APM] Scope APM alert creation to environment (elastic#65681) Move Kibana Usage collectors outside the telemetry plugin (elastic#65663) [ML] Data Frame Analytics: Fix confusion matrix data grid width. (elastic#65818) Switch to core application service (elastic#63443) Removes use of prefer_v2_templates (elastic#66316) [Maps] handle case where fit to bounds does not match any documents (elastic#66307) log error instead of throw (elastic#66254) [plugin-helpers] remove outdated postinstall task (elastic#66324) Spaces - migrate default space and enter space view to KP (elastic#66098) [APM] Change plugin id for `apm_oss` to `apmOss` (elastic#66164) [Maps] convert map_selectors to TS (elastic#65905) [uptime/usage-collector] add missing await (elastic#66079) [Ingest] Add additional attributes to the Datasources Saved Object (elastic#66127) [Endpoint]EMT-339: add new policy response schema (elastic#66264) ...

paul-tavares added 3 commits May 8, 2020 09:41

add additinoal properties to ingest-datasources SO

4c5bd86

Adjusted Types and test generators

5f545ff

Added datasources migrations to SO

9eedd02

paul-tavares added v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Team:Endpoint Management v7.9.0 Team:Fleet Team label for Observability Data Collection Fleet team labels May 11, 2020

paul-tavares self-assigned this May 11, 2020

nchaulet reviewed May 11, 2020

View reviewed changes

ruflin reviewed May 12, 2020

View reviewed changes

paul-tavares added 6 commits May 12, 2020 09:07

change date attributes to type: 'date'

98a75ff

Add user info. to datasource.create()

be8535d

Merge remote-tracking branch 'upstream/master' into task/ingest-65904…

433eedb

…-datasources-additional-fields

Fix datasource update to remove extra attributes

375ea63

Fix endpoint policy save to remove extra attributes from update

74c3cd8

Merge remote-tracking branch 'upstream/master' into task/ingest-65904…

78c8194

…-datasources-additional-fields

paul-tavares marked this pull request as ready for review May 12, 2020 14:29

paul-tavares requested a review from a team as a code owner May 12, 2020 14:29

paul-tavares requested a review from a team May 12, 2020 14:29

paul-tavares requested a review from a team as a code owner May 12, 2020 14:29

nchaulet reviewed May 12, 2020

View reviewed changes

nchaulet approved these changes May 12, 2020

View reviewed changes

paul-tavares added 2 commits May 12, 2020 15:26

renamed created/update attributes to be *_at

cd9d9b3

Merge remote-tracking branch 'upstream/master' into task/ingest-65904…

1198481

…-datasources-additional-fields

parkiino approved these changes May 12, 2020

View reviewed changes

paul-tavares merged commit ad39997 into elastic:master May 12, 2020

paul-tavares deleted the task/ingest-65904-datasources-additional-fields branch May 12, 2020 21:35

paul-tavares mentioned this pull request May 12, 2020

[7.x] [Ingest] Add additional attributes to the Datasources Saved Object (#66127) #66313

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Ingest] Add additional attributes to the Datasources Saved Object #66127

[Ingest] Add additional attributes to the Datasources Saved Object #66127

paul-tavares commented May 11, 2020 •

edited

Loading

elasticmachine commented May 11, 2020

elasticmachine commented May 11, 2020

paul-tavares commented May 11, 2020

nchaulet May 11, 2020

paul-tavares May 12, 2020

ruflin May 12, 2020

paul-tavares commented May 12, 2020

nchaulet commented May 12, 2020

nchaulet May 12, 2020

paul-tavares May 12, 2020

paul-tavares commented May 12, 2020

nchaulet left a comment

jen-huang commented May 12, 2020

kibanamachine commented May 12, 2020

[Ingest] Add additional attributes to the Datasources Saved Object #66127

[Ingest] Add additional attributes to the Datasources Saved Object #66127

Conversation

paul-tavares commented May 11, 2020 • edited Loading

Summary

Checklist

elasticmachine commented May 11, 2020

elasticmachine commented May 11, 2020

paul-tavares commented May 11, 2020

nchaulet May 11, 2020

Choose a reason for hiding this comment

paul-tavares May 12, 2020

Choose a reason for hiding this comment

ruflin May 12, 2020

Choose a reason for hiding this comment

paul-tavares commented May 12, 2020

nchaulet commented May 12, 2020

nchaulet May 12, 2020

Choose a reason for hiding this comment

paul-tavares May 12, 2020

Choose a reason for hiding this comment

paul-tavares commented May 12, 2020

nchaulet left a comment

Choose a reason for hiding this comment

jen-huang commented May 12, 2020

kibanamachine commented May 12, 2020

💚 Build Succeeded

History

paul-tavares commented May 11, 2020 •

edited

Loading