Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Content management] POC for the core (server) #148791

Closed

Conversation

sebelga
Copy link
Contributor

@sebelga sebelga commented Jan 12, 2023

WIP for the content management core implementation (server side)

Screenshot 2023-01-13 at 12 02 32

Note for reviewers

This is still a work in Progress! 😊

Navigate to "management > kibana > content management" to test the UI

  • There are 2 contents to play with: foo (with an In memory DB) and maps. To index the content for maps it currently only works when creating a new map. To index the content for foo use the form in the UI.
  • If need be, delete the Search index. For that go to DevTools and send DELETE .kibana-content-mgt. It will be re-created on the next Kibana server restart.
  • The search box does not work (yet). The table is only to demo the content being indexed.

Upcoming work:

  • Handle indexing the content

I am thinking in requiring a method on the Storage class to be able to retrieve the full content from the DB.
Then, when registering the content we provide the Kibana version we want the content to be indexed from (e.g. indexContentFrom: "8.8.0"). --> When doing a stack upgrade we only index the content to the Search index when the new stack equal or greater version declared here.

  • Search api

I am thinking in allowing 5 custom metrics (number fields in mapping) and 5 custom terms (keyword fields in mappings) to be passed in the documents. Those fields ("metric1", "metric2", "metric3" ...) will then allow us to render any table list view with custom column for each content (e.g. Files table will be able to sort by file size)

@Dosant
Copy link
Contributor

Dosant commented Jan 16, 2023

@elasticmachine merge upstream

@TinaHeiligers
Copy link
Contributor

FYI @jloleysens @TinaHeiligers

@kibana-ci
Copy link
Collaborator

kibana-ci commented Feb 2, 2023

💔 Build Failed

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #38 / Advanced Settings security feature controls no advanced_settings privileges does not allow navigation to advanced settings; shows "not found" error
  • [job] [logs] FTR Configs #38 / Advanced Settings security feature controls no advanced_settings privileges does not allow navigation to advanced settings; shows "not found" error
  • [job] [logs] FTR Configs #38 / Advanced Settings security feature controls no advanced_settings privileges does not show Management navlink
  • [job] [logs] FTR Configs #38 / Advanced Settings security feature controls no advanced_settings privileges does not show Management navlink
  • [job] [logs] FTR Configs #33 / API Keys app feature controls security global dashboard read with manage_security should render the "Security" section with API Keys
  • [job] [logs] FTR Configs #33 / API Keys app feature controls security global dashboard read with manage_security should render the "Security" section with API Keys
  • [job] [logs] FTR Configs #49 / apps transform basic license transform feature controls security global dashboard read with transform_user should render the "Data" section with Transform
  • [job] [logs] FTR Configs #49 / apps transform basic license transform feature controls security global dashboard read with transform_user should render the "Data" section with Transform
  • [job] [logs] FTR Configs #14 / Canvas Canvas app security feature controls global canvas all privileges shows canvas navlink
  • [job] [logs] FTR Configs #14 / Canvas Canvas app security feature controls global canvas all privileges shows canvas navlink
  • [job] [logs] FTR Configs #14 / Canvas Canvas app security feature controls global canvas read-only privileges shows canvas navlink
  • [job] [logs] FTR Configs #14 / Canvas Canvas app security feature controls global canvas read-only privileges shows canvas navlink
  • [job] [logs] FTR Configs #11 / Cross Cluster Replication app feature controls security global dashboard read with ccr_user should render the "Data" section with CCR
  • [job] [logs] FTR Configs #11 / Cross Cluster Replication app feature controls security global dashboard read with ccr_user should render the "Data" section with CCR
  • [job] [logs] FTR Configs #10 / dashboard feature controls dashboard feature controls security global dashboard read-only privileges shows dashboard navlink
  • [job] [logs] FTR Configs #10 / dashboard feature controls dashboard feature controls security global dashboard read-only privileges shows dashboard navlink
  • [job] [logs] FTR Configs #10 / dashboard feature controls dashboard feature controls security global dashboard read-only with url_create privileges shows dashboard navlink
  • [job] [logs] FTR Configs #10 / dashboard feature controls dashboard feature controls security global dashboard read-only with url_create privileges shows dashboard navlink
  • [job] [logs] FTR Configs #14 / Data Views feature controls security no data views privileges does not show Management navlink
  • [job] [logs] FTR Configs #14 / Data Views feature controls security no data views privileges does not show Management navlink
  • [job] [logs] FTR Configs #14 / Data Views feature controls security no data views privileges doesn't show Data Views in management side-nav
  • [job] [logs] FTR Configs #14 / Data Views feature controls security no data views privileges doesn't show Data Views in management side-nav
  • [job] [logs] FTR Configs #32 / Dev Tools feature controls security global dev_tools all privileges shows Dev Tools navlink
  • [job] [logs] FTR Configs #32 / Dev Tools feature controls security global dev_tools all privileges shows Dev Tools navlink
  • [job] [logs] FTR Configs #32 / Dev Tools feature controls security global dev_tools read-only privileges shows 'Dev Tools' navlink
  • [job] [logs] FTR Configs #32 / Dev Tools feature controls security global dev_tools read-only privileges shows 'Dev Tools' navlink
  • [job] [logs] FTR Configs #46 / discover feature controls discover feature controls security global discover read-only privileges shows discover navlink
  • [job] [logs] FTR Configs #46 / discover feature controls discover feature controls security global discover read-only privileges shows discover navlink
  • [job] [logs] FTR Configs #46 / discover feature controls discover feature controls security global discover read-only privileges with url_create shows discover navlink
  • [job] [logs] FTR Configs #46 / discover feature controls discover feature controls security global discover read-only privileges with url_create shows discover navlink
  • [job] [logs] Jest Tests #9 / FetchHistoricalSummary Calendar Aligned and Occurrences SLOs returns the summary
  • [job] [logs] Jest Tests #9 / FetchHistoricalSummary Calendar Aligned and Timeslices SLOs returns the summary
  • [job] [logs] FTR Configs #4 / Fleet Endpoints EPM Endpoints EPM - get allows user with only package level permission to access corresponding packages
  • [job] [logs] FTR Configs #24 / graph app feature controls security global graph all privileges shows graph navlink
  • [job] [logs] FTR Configs #24 / graph app feature controls security global graph all privileges shows graph navlink
  • [job] [logs] FTR Configs #24 / graph app feature controls security global graph read-only privileges shows graph navlink
  • [job] [logs] FTR Configs #24 / graph app feature controls security global graph read-only privileges shows graph navlink
  • [job] [logs] FTR Configs #22 / License app feature controls security global dashboard read with license_management_user and upgrade assistant [SkipCloud] global dashboard with license management user and upgrade assistant : skip cloud should render the "Stack" section with License Management and Upgrade Assistant
  • [job] [logs] FTR Configs #22 / License app feature controls security global dashboard read with license_management_user and upgrade assistant [SkipCloud] global dashboard with license management user and upgrade assistant : skip cloud should render the "Stack" section with License Management and Upgrade Assistant
  • [job] [logs] FTR Configs #44 / logstash feature controls security global dashboard read with logstash_read_user should render the "Ingest" section with Logstash Pipelines
  • [job] [logs] FTR Configs #44 / logstash feature controls security global dashboard read with logstash_read_user should render the "Ingest" section with Logstash Pipelines
  • [job] [logs] FTR Configs #50 / management feature controls security global all privileges (aka kibana_admin) should only render management entries controllable via Kibana privileges
  • [job] [logs] FTR Configs #50 / management feature controls security global all privileges (aka kibana_admin) should only render management entries controllable via Kibana privileges
  • [job] [logs] FTR Configs #50 / management feature controls security no management privileges should not show the Stack Management nav link
  • [job] [logs] FTR Configs #50 / management feature controls security no management privileges should not show the Stack Management nav link
  • [job] [logs] FTR Configs #50 / management feature controls security no management privileges should render the "application not found" view when navigating to management directly
  • [job] [logs] FTR Configs #50 / management feature controls security no management privileges should render the "application not found" view when navigating to management directly
  • [job] [logs] FTR Configs #35 / maps app embeddable maps add-to-dashboard save flow should allow existing maps be added by reference to a new dashboard
  • [job] [logs] FTR Configs #35 / maps app embeddable maps add-to-dashboard save flow should allow existing maps be added by reference to a new dashboard
  • [job] [logs] FTR Configs #35 / maps app embeddable maps add-to-dashboard save flow should allow existing maps be added by reference to an existing dashboard
  • [job] [logs] FTR Configs #35 / maps app embeddable maps add-to-dashboard save flow should allow existing maps be added by reference to an existing dashboard
  • [job] [logs] FTR Configs #35 / maps app embeddable save and return work flow edit existing map save as and uncheck return to origin switch should cut the originator and stay in maps application
  • [job] [logs] FTR Configs #35 / maps app embeddable save and return work flow edit existing map save as and uncheck return to origin switch should cut the originator and stay in maps application
  • [job] [logs] FTR Configs #35 / maps app embeddable save and return work flow edit existing map save as should return to dashboard and add new panel
  • [job] [logs] FTR Configs #35 / maps app embeddable save and return work flow edit existing map save as should return to dashboard and add new panel
  • [job] [logs] FTR Configs #21 / maps app maps security feature controls global maps read-only privileges shows Maps navlink
  • [job] [logs] FTR Configs #21 / maps app maps security feature controls global maps read-only privileges shows Maps navlink
  • [job] [logs] FTR Configs #14 / Remote Clusters app feature controls security global dashboard read with license_management_user should render the "Data" section with Remote Clusters
  • [job] [logs] FTR Configs #14 / Remote Clusters app feature controls security global dashboard read with license_management_user should render the "Data" section with Remote Clusters
  • [job] [logs] FTR Configs #16 / saved objects tagging - functional tests maps integration creating allows to create a tag from the tag selector
  • [job] [logs] FTR Configs #16 / saved objects tagging - functional tests maps integration creating allows to create a tag from the tag selector
  • [job] [logs] FTR Configs #16 / saved objects tagging - functional tests maps integration creating allows to select tags for a new map
  • [job] [logs] FTR Configs #16 / saved objects tagging - functional tests maps integration creating allows to select tags for a new map
  • [job] [logs] FTR Configs #39 / search sessions management Search Sessions Management UI permissions Sessions management is available if one app enables search sessions
  • [job] [logs] FTR Configs #39 / search sessions management Search Sessions Management UI permissions Sessions management is available if one app enables search sessions
  • [job] [logs] FTR Configs #39 / search sessions management Search Sessions Management UI permissions Sessions management is not available if no apps enable search sessions
  • [job] [logs] FTR Configs #39 / search sessions management Search Sessions Management UI permissions Sessions management is not available if no apps enable search sessions
  • [job] [logs] FTR Configs #37 / transform feature controls security global dashboard read with transform_user should render the "Data" section with Transform
  • [job] [logs] FTR Configs #37 / transform feature controls security global dashboard read with transform_user should render the "Data" section with Transform
  • [job] [logs] FTR Configs #42 / Upgrade Assistant feature controls security global dashboard read with global_upgrade_assistant_role [SkipCloud] global dashboard read with global_upgrade_assistant_role should render the "Stack" section with Upgrade Assistant
  • [job] [logs] FTR Configs #42 / Upgrade Assistant feature controls security global dashboard read with global_upgrade_assistant_role [SkipCloud] global dashboard read with global_upgrade_assistant_role should render the "Stack" section with Upgrade Assistant
  • [job] [logs] FTR Configs #23 / Visualize visualize feature controls security global visualize read-only privileges shows visualize navlink
  • [job] [logs] FTR Configs #23 / Visualize visualize feature controls security global visualize read-only privileges shows visualize navlink
  • [job] [logs] FTR Configs #23 / Visualize visualize feature controls security global visualize read-only with url_create privileges shows visualize navlink
  • [job] [logs] FTR Configs #23 / Visualize visualize feature controls security global visualize read-only with url_create privileges shows visualize navlink

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
contentManagement - 150 +150
maps 963 966 +3
total +153

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
contentManagement - 30.8KB +30.8KB
maps 2.7MB 2.7MB +1.2KB
total +32.0KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
contentManagement - 210.0KB ⚠️ +210.0KB
maps 53.1KB 53.1KB +64.0B
total +210.1KB
Unknown metric groups

async chunk count

id before after diff
contentManagement - 1 +1

miscellaneous assets size

id before after diff
contentManagement - 161.8KB +161.8KB

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream


export interface GetItemStart {
type: 'getItemStart';
contentId: string;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the contentId in any way related to a Saved Object id if the content is a saved object? If so, given that we're using the exClient directly, would the Saved Object id be the raw ES id?

Another question related to saved object content types: How are spaces and space-specific objects handled? What about shared saved objects? I guess with shared types we can assume uniqueness.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the contentId in any way related to a Saved Object id if the content is a saved object? If so, given that we're using the exClient directly, would the Saved Object id be the raw ES id?

The id could be directly SO id and thus the raw ES id, yes. For content management it is just a string passed down to the storage layer to retrieve a content.

How are spaces and space-specific objects handled? What about shared saved objects? I guess with shared types we can assume uniqueness.

The CM layer is DB agnostic it does not know about SO. What you are describing is a logic handled by the application. If an application ask to fetch id 1234, CM will forward that to the storage layer and fetch that content on the storage layer.
For example, we are on space A and the application fetches object with id 1234 that only exist in space B, that's an error at the application level. It should not fetch that object. Now, the storage layer has access to the core http request context so which, from what I understand, provide security and authorization for SO. That context can be leveraged to prevent the above scenario.

Copy link
Contributor

@TinaHeiligers TinaHeiligers Feb 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

directly SO id and thus the raw ES id,

Saved object IDs go through a SavedObjectsSerializer class that transforms the raw es doc id to an id in the context of Kibana. Hence, a saved object id isn't necessarily the raw es doc Id but you probably know that already.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is internal to SO and CM does not know (or need to know) about it. Here we are talking about the SO id, if it is equal or not to the raw ES id is internal to SO.

Dosant added a commit that referenced this pull request Feb 7, 2023
## Summary

Partially addresses #149216,
#149939
Closes #149217


Setup basic client-side content client (client, cache, react content and
hooks) based on the latest API from @sebelga's [server-side and rpc
POC](#148791). We assume that the
POC isn't far from the initial version that we are going to merge to
main

Initial content client implementation scope: 
- We start from using tanstack/query for client side caching, content
queuing and mutating . Also use it as a basis for the API. We start from
their defaults and will adjust defaults as we go. Basically our content
client would be a helpful wrapper around tanstack/query (at least
initially)
- We start from exposing minimal query lib API surface, but we will
gradually expose more as needed
- For content retrieval we support promise based, RXJS observable based
and react hook based methods
- For the RPC/api I copied over the latest from [server-side and rpc
POC](#148791) assuming it won't
change much.
- no optimistic update or automatic invalidation after updates yet (this
will be later)
- just get and create for now
@legrego legrego closed this Jul 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants