Merge branch 'main' into fix/dw-response-actions-history-tray-back-pa…

…ge-6603

.buildkite/scripts/steps/checks.sh

@@ -6,7 +6,6 @@ export DISABLE_BOOTSTRAP_VALIDATION=false
 .buildkite/scripts/bootstrap.sh
 
 .buildkite/scripts/steps/checks/precommit_hook.sh
-.buildkite/scripts/steps/checks/packages.sh
 .buildkite/scripts/steps/checks/ts_projects.sh
 .buildkite/scripts/steps/checks/packages.sh
 .buildkite/scripts/steps/checks/bazel_packages.sh
@@ -22,4 +21,4 @@ export DISABLE_BOOTSTRAP_VALIDATION=false
 .buildkite/scripts/steps/checks/test_projects.sh
 .buildkite/scripts/steps/checks/test_hardening.sh
 .buildkite/scripts/steps/checks/ftr_configs.sh
-.buildkite/scripts/steps/checks/saved_objects_compat_changes.sh
+.buildkite/scripts/steps/checks/saved_objects_compat_changes.sh

.github/workflows/deploy-my-kibana.yml

@@ -2,7 +2,7 @@
 ##
 ## This the automation to let Observability team members to deploy a Kibana instance
 ## using the Observability test environments.
-## It will deploy a new instance for those who add a comment /oblt-deploy
+## It will deploy a new instance for those who add a comment /oblt-deploy or /oblt-deploy-serverless
 ## only supported for Elasticians.
 ##
 ## Owner: @elastic/observablt-robots
@@ -28,3 +28,15 @@ jobs:
           vaultUrl: ${{ secrets.OBLT_VAULT_ADDR }}
           vaultRoleId: ${{ secrets.OBLT_VAULT_ROLE_ID }}
           vaultSecretId: ${{ secrets.OBLT_VAULT_SECRET_ID }}
+          serverless: false
+
+  deploy-my-kibana-serverless:
+    if: ${{ github.event.issue.pull_request && github.event.comment.body == '/oblt-deploy-serverless'}}
+    runs-on: ubuntu-latest
+    steps:
+      - uses: elastic/apm-pipeline-library/.github/actions/deploy-my-kibana@current
+        with:
+          vaultUrl: ${{ secrets.OBLT_VAULT_ADDR }}
+          vaultRoleId: ${{ secrets.OBLT_VAULT_ROLE_ID }}
+          vaultSecretId: ${{ secrets.OBLT_VAULT_SECRET_ID }}
+          serverless: true

.github/workflows/oblt-github-commands.yml

@@ -50,6 +50,7 @@ jobs:
 
               Just comment with:
               - \`/oblt-deploy\` : Deploy a Kibana instance using the Observability test environments.
+              - \`/oblt-deploy-serverless\` : Deploy a Kibana instance using the Observability `serverless` test environment [only for main].
               - \`run\` \`elasticsearch-ci/docs\` : Re-trigger the docs validation. (use unformatted text in the comment!)
 
               </p>

dev_docs/getting_started/setting_up_a_development_env.mdx

@@ -75,11 +75,13 @@ In another terminal tab/window you can start Kibana.
 yarn start
 ```
 
-Include developer examples](https://github.com/elastic/kibana/tree/main/examples) by adding an optional `--run-examples` flag. Read more about the advanced options for [Running Kibana](https://www.elastic.co/guide/en/kibana/current/running-kibana-advanced.html).
+Include developer [examples](https://github.com/elastic/kibana/tree/main/examples) by adding an optional `--run-examples` flag. Read more about the advanced options for [Running Kibana](https://www.elastic.co/guide/en/kibana/current/running-kibana-advanced.html).
+You will find the development server running on (http://localhost:5601) - and you can log in with the `elastic:changeme` credential pair.
 
 ## Code away!
 
-You are now ready to start developing. Changes to the source files should be picked up automatically and either cause the server to restart, or be served to the browser on the next page refresh.
+You are now ready to start developing. 
+Changes to the source files should be picked up automatically and either cause the server to restart, or be served to the browser on the next page refresh.
 
 ## Install pre-commit hook (optional)
 

docs/concepts/kuery.asciidoc

@@ -178,13 +178,17 @@ http.request.method: (GET OR POST OR DELETE)
 === Matching multiple fields
 
 Wildcards can also be used to query multiple fields. For example, to search for
-documents where any sub-field of `http.response` contains “error”, use the following:
+documents where any sub-field of `datastream` contains “logs”, use the following:
 
 [source,yaml]
 -------------------
-http.response.*: error
+datastream.*: logs
 -------------------
 
+NOTE: When using wildcards to query multiple fields, errors might occur if the fields are of
+different types. For example, if `datastream.*` matches both numeric and string fields, the
+above query will result in an error because numeric fields cannot be queried for string values.
+
 [discrete]
 === Querying nested fields
 

docs/setup/upgrade/resolving-migration-failures.asciidoc

@@ -8,8 +8,8 @@ with the new version.
 ==== Saved object migration failures
 
 If {kib} unexpectedly terminates while migrating a saved object index, {kib} automatically attempts to
-perform the migration again when the process restarts. Do not delete any saved objects indices to 
-fix a failed migration. Unlike previous versions, {kib} 7.12.0 and later does not require deleting 
+perform the migration again when the process restarts. Do not delete any saved objects indices to
+fix a failed migration. Unlike previous versions, {kib} 7.12.0 and later does not require deleting
 indices to release a failed migration lock.
 
 If upgrade migrations fail repeatedly, refer to
@@ -19,14 +19,6 @@ When you address the root cause for the migration failure,
 If you're unable to resolve a failed migration, contact Support.
 
 
-[float]
-[[upgrade-migrations-old-indices]]
-==== Old `.kibana_N` indices
-
-After the migrations complete, multiple {kib} indices are created in {es}: (`.kibana_1`, `.kibana_2`, `.kibana_7.12.0` etc).
-{kib} only uses the index that the `.kibana` and `.kibana_task_manager` aliases point to.
-The other {kib} indices can be safely deleted, but are left around as a matter of historical record, and to facilitate rolling {kib} back to a previous version.
-
 [float]
 ==== Known issues with {fleet} beta
 If you see a`timeout_exception` or `receive_timeout_transport_exception` error,

docs/setup/upgrade/saved-objects-migration.asciidoc

@@ -8,34 +8,66 @@ To access the assistant, go to *Stack Management > Upgrade Assistant*.
 
 WARNING: {kib} 7.12.0 and later uses a new migration process and index naming scheme. Before you upgrade, read the documentation for your version of {kib}.
 
-WARNING: The following instructions assumes {kib} is using the default index names. If the `kibana.index` or `xpack.tasks.index` configuration settings are different from the default, adapt the instructions accordingly.
+WARNING: The `kibana.index` and `xpack.tasks.index` configuration settings are obsolete and no longer taken into account in 8.x. If you are using custom index names, please perform the necessary adaptations before attempting to upgrade to 8.x.
 
 [float]
 [[upgrade-migrations-process]]
 ==== How saved objects migrations work
 
-Saved objects are stored in two indices:
-
-* `.kibana_{kibana_version}_001`, e.g. for {kib} 7.12.0 `.kibana_7.12.0_001`.
-* `.kibana_task_manager_{kibana_version}_001`, e.g. for {kib} 7.12.0 `.kibana_task_manager_7.12.0_001`.
-
-The index aliases `.kibana` and `.kibana_task_manager` always point to
-the most up-to-date saved object indices.
-
 When you start a new {kib} installation, an upgrade migration is performed before starting plugins or serving HTTP traffic.
 Before you upgrade, shut down old nodes to prevent losing acknowledged writes.
 To reduce the likelihood of old nodes losing acknowledged writes, {kib} 7.12.0 and later
 adds a write block to the outdated index.
 
-The following tables lists the saved objects indices used by previous {kib} versions.
+Saved objects are stored in multiple indices. Whilst all of them start with the `.kibana*` prefix, other `.kibana*` indices exist, which are not used to store saved objects.  The following tables lists the saved objects indices used by each {kib} version.
 
 .Saved object indices and aliases per {kib} version
 [options="header"]
 |=======================
-|Upgrading from version | Outdated index (alias)
-| 6.5.0 through 7.3.x    | `.kibana_N` (`.kibana` alias)
+|Upgrading from version | Index | Aliases
+| 6.5.0 through 7.3.x
+| `.kibana_N`
+| `.kibana`
 | 7.4.0 through 7.11.x
-| `.kibana_N` (`.kibana` alias)
-
-`.kibana_task_manager_N` (`.kibana_task_manager` alias)
+| `.kibana_N` +
+`.kibana_task_manager_N`
+| `.kibana` +
+`.kibana_task_manager`
+| 7.11.x through 8.7.x
+| `.kibana_{kibana_version}_001` +
+`.kibana_task_manager_{kibana_version}_001`
+| `.kibana`, `.kibana_{kibana_version}` +
+`.kibana_task_manager`, `.kibana_task_manager_{kibana_version}`
+| 8.8.0+
+| `.kibana_{kibana_version}_001` +
+`.kibana_alerting_cases_{kibana_version}_001`
+`.kibana_analytics_{kibana_version}_001`
+`.kibana_ingest_{kibana_version}_001`
+`.kibana_task_manager_{kibana_version}_001`
+`.kibana_security_solution_{kibana_version}_001`
+| `.kibana`, `.kibana_{kibana_version}` +
+`.kibana_alerting_cases`, `.kibana_alerting_cases_{kibana_version}`
+`.kibana_analytics`, `.kibana_analytics_{kibana_version}`
+`.kibana_ingest`, `.kibana_ingest_{kibana_version}`
+`.kibana_task_manager`, `.kibana_task_manager_{kibana_version}`
+`.kibana_security_solution`, `.kibana_security_solution_{kibana_version}`
 |=======================
+
+Starting on 7.11.0, each of the saved objects indices has a couple of aliases, e.g. the `.kibana_8.8.0_001` index has a _default_ alias `.kibana` and a _version_ alias `.kibana_8.8.0`. The _default_ aliases (e.g. `.kibana` and `.kibana_task_manager`) always point to
+the most up-to-date saved object indices. Then, _version_ aliases are aligned with the deployed {kib} version.
+
+
+Starting on 8.6.0, index names aren't necessarily aligned with the deployed {kib} version. When updates on a certain index are compatible, {kib} will keep the existing index instead of creating a new one. This allows for a more efficient upgrade process. The following example illustrates a completely valid state for a 8.8.0 deployment:
+
+* `.kibana_8.8.0_001` index, with `.kibana` and `.kibana_8.8.0` aliases.
+* `.kibana_task_manager_8.7.0_001` index, with `.kibana_task_manager` and `.kibana_task_manager_8.8.0` aliases.
+
+Starting on 8.8.0, {kib} splits the main saved object index into multiple ones, as depicted on the table above. When upgrading from a previous version, the {kib} migration process will reindex some saved objects from the `.kibana` index into the new indices, depending on their types. Note that the `.kibana` index still exists, and it continues to store multiple saved object types.
+
+[float]
+[[upgrade-migrations-old-indices]]
+==== Old {kib} indices
+
+As a deployment is gradually upgraded, multiple {kib} indices are created in {es}: (`.kibana_1`, `.kibana_2`, `.kibana_7.12.0_001`, `.kibana_7.13.0_001`, `.kibana_8.0.0_001` etc).
+{kib} only uses those indices that the _default_ and _version_ aliases point to.
+The other, older {kib} saved object indices can be safely deleted, but are left around as a matter of historical record, and to facilitate rolling {kib} back to a previous version.

docs/user/dashboard/lens.asciidoc

@@ -175,18 +175,35 @@ Compare your real-time data to the results that are offset by a time increment.
 For a time shift example, refer to <<compare-time-ranges>>.
 
 [float]
-[[multi-metric-partition-chart]]
-==== Build a partition chart from multiple metrics
+[[create-partition-charts-with-multiple-metrics]]
+==== Create partition charts with multiple metrics
 
-By default, partition charts (e.g. pie) are built from one or more "slice-by" dimensions to define the partitions and a single metric dimension to define their size. However, you can also build a partition chart from multiple metric dimensions.
+To create partition charts, such as pie charts, configure one or more *Slice by* dimensions to define the partitions, and a *Metric* dimension to define the size. 
+To create partition charts with multiple metrics, use the layer settings. Multiple metrics are unsupported for mosaic visualizations.
 
-. Open the layer context menu at the top right of the layer panel.
+. In the layer pane, click image:dashboard/images/lens_layerActions_8.5.0.png[Actions menu for the partition visualization layer], then select *Layer settings*.
 
-. Click *Layer settings*.
+. Select *Multiple metrics*.
 
-. Click the switch labeled *Multiple metrics*.
+. Click *X*.
 
-Note: this option is not available for mosaic charts.
+[float]
+[[improve-visualization-loading-time]]
+=== Improve visualization loading time
+
+preview::[]
+
+Data sampling allows you to improve the visualization loading time. To decrease the loading time, use a lower sampling percentage, which also decreases the accuracy. 
+Use low sampling percentages on large datasets.
+
+. In the layer pane, click image:dashboard/images/lens_layerActions_8.5.0.png[Actions menu for the partition visualization layer], then select *Layer settings*.
+
+. To select the *Sampling* percentage, use the slider.
+
+. Click *X*.
++
+When you add the visualization to dashboards, image:dashboard/images/lens_visualizationModifierPopup_8.8.0.png[Visualization modifier popup] appears, 
+which allows you to view settings changes to the visualization.
 
 [float]
 [[add-annotations]]
@@ -214,6 +231,9 @@ Create the annotation layer.
 . From the fields list, drag a field to the *Add an annotation* field.
 
 . To use global filters in the annotation, click image:dashboard/images/lens_layerActions_8.5.0.png[Actions menu for the annotations layer], then select *Keep global filters* from the dropdown.
++
+When you add the visualization to dashboards, image:dashboard/images/lens_visualizationModifierPopup_8.8.0.png[Visualization modifier popup] appears, 
+which allows you to view settings changes to the visualization.
 
 Create static annotations.
 

docs/user/reporting/index.asciidoc

@@ -13,19 +13,21 @@
 
 You access the options from the *Share* menu in the toolbar. The sharing options include the following:
 
-* *PDF Reports* — Generate and download a PDF file of a dashboard, visualization, or *Canvas* workpad. PDF reports are a link:https://www.elastic.co/subscriptions[subscription feature].
+* *PDF Reports* — Generate and download PDF files of dashboards, visualizations, and *Canvas* workpads. PDF reports are a link:https://www.elastic.co/subscriptions[subscription feature].
 
-* *PNG Reports* — Generate and download a PNG file of a dashboard or visualization. PNG reports are a link:https://www.elastic.co/subscriptions[subscription feature].
+* *PNG Reports* — Generate and download PNG files of dashboards and visualizations. PNG reports are a link:https://www.elastic.co/subscriptions[subscription feature].
 
-* *CSV Reports* — Generate and download a CSV file of a *Discover* saved search.
+* *CSV Reports* — Generate and download CSV files of saved searches.
 
-* *Get links* — Share a direct link to a *Discover* saved search, dashboard, or visualization.
+* *CSV Download* — Generate and download CSV files of *Lens* visualizations.
 
-* *Download as JSON* — Generate and download a JSON file of a *Canvas* workpad.
+* *Get links* — Share direct links to saved searches, dashboards, and visualizations.
+
+* *Download as JSON* — Generate and download JSON files of *Canvas* workpads.
 
 * beta[] *Share on a website* — Download and securely share *Canvas* workpads on any website.
 
-* *Embed code* — Embed a fully interactive dashboard as an iframe on a web page.
+* *Embed code* — Embed fully interactive dashboards as an iframe on web pages.
 
 [[reporting-on-cloud-resource-requirements]]
 NOTE: For Elastic Cloud deployments, {kib} instances require a minimum of 2GB RAM to generate PDF or PNG reports. To
@@ -51,19 +53,13 @@ image::user/reporting/images/shareable-container.png["Shareable Container"]
 
 . Open the main menu, then open the saved search, dashboard, visualization, or workpad you want to share.
 
-. From the toolbar, click *Share*, then select one of the following options:
-
-** **PDF Reports** — Generates a PDF file of the dashboard, visualization, or workpad.
-
-** **PNG Reports** — Generates a PNG file of the dashboard or visualization.
+. From the toolbar, click *Share*, then select the report option. 
 
-** **CSV Reports** — Generates a CSV report of the saved search.
-
-. If you are creating a PDF report of a dashboard, select *Optimize for printing* to create a printer-friendly PDF with multiple A4 portrait pages and two visualizations per page.
+* If you are creating dashboard PDFs, select *Optimize for printing* to create printer-friendly PDFs with multiple A4 portrait pages and two visualizations per page.
 +
 NOTE: When you create a dashboard report that includes a data table or saved search, the PDF includes only the visible data.
 
-. If you are creating a PDF report of a workpad, select *Full page layout* to create a PDF without margins that surround the workpad.
+* If you are creating workpad PDFs, select *Full page layout* to create PDFs without margins that surround the workpad.
 
 . Generate the report.
 
@@ -78,9 +74,9 @@ more about {ilm-init} policies, refer to the {es}
 
 [float]
 [[share-a-direct-link]]
-== Share a direct link
+== Share direct links
 
-Share a direct link to a saved search, dashboard, or visualization. To access the shared object, authentication is required.
+Create and share direct links to saved searches, dashboards, and visualizations. To access the shared object, authentication is required.
 
 . Open the main menu, then open the saved search, dashboard, or visualization you want to share.
 
@@ -105,9 +101,9 @@ NOTE: *Public URL* is available only when anonymous access is configured and you
 
 [float]
 [[download-as-json]]
-== Create a JSON file
+== Create JSON files
 
-Create a JSON file for a workpad.
+Create and share JSON files for workpads.
 
 . Open the main menu, then click *Canvas*.
 
@@ -119,17 +115,14 @@ Create a JSON file for a workpad.
 [[add-workpad-website]]
 == Share workpads on a website
 
-beta[] *Canvas* allows you to create _shareables_, which are workpads that you download and securely share on a website.
-To customize the behavior of the workpad on your website, you can choose to autoplay the pages or hide the workpad toolbar.
+beta[] Create and securely share static *Canvas* workpads on a website. To customize the behavior of the workpad on your website, you can choose to autoplay the pages or hide the workpad toolbar.
 
 . Open the main menu, then click *Canvas*.
 
 . Open the workpad you want to share.
 
 . Click *Share > Share on a website*.
 
-. Follow the instructions.
-
 . To customize the workpad behavior to autoplay the pages or hide the toolbar, use the inline parameters.
 +
 To make sure that your data remains secure, the data in the JSON file is not connected to {kib}. *Canvas* does not display elements that manipulate the data on the workpad.
@@ -142,21 +135,21 @@ NOTE: Shareable workpads encode the current state of the workpad in a JSON file.
 [[embed-code]]
 == Embed code
 
-Display your dashboard on an internal company website or personal web page with an iframe. Embedding other {kib} objects is generally supported, but you might need to manually craft the proper HTML code.
+Display your dashboards on an internal company website or personal web page with an iframe. To embed other {kib} objects, manually create the HTML code.
 
-Some users might not have access to the dashboard or visualization. For more information, refer to <<anonymous-access-and-embedding>> and <<embedded-content-authentication>>.
+For information about granting access to embedded dashboards, refer to <<kibana-authentication>>.
 
 . Open the main menu, then open the dashboard you want to share.
 
 . Click *Share > Embed code*.
 
 . Specify how you want to generate the code:
 
-* To display only the current state, select *Snapshot*.
+* To display the current state, select *Snapshot*.
 
 * To display up-to-date changes, select *Saved object*.
 
-* Select the dashboard components you want to include.
+* Select the dashboard components you want to share.
 
 * To generate a shortened link, select *Short URL*.
 

packages/core/saved-objects/core-saved-objects-base-server-internal/index.ts

@@ -39,6 +39,7 @@ export type {
   KibanaMigratorStatus,
   MigrationResult,
   MigrationStatus,
+  MigrateDocumentOptions,
 } from './src/migration';
 export { parseObjectKey, getObjectKey, getIndexForType } from './src/utils';
 export {
@@ -64,4 +65,5 @@ export {
   getModelVersionDelta,
   buildModelVersionTransformFn,
   aggregateMappingAdditions,
+  convertModelVersionBackwardConversionSchema,
 } from './src/model_version';

packages/core/saved-objects/core-saved-objects-base-server-internal/src/migration/index.ts

@@ -11,4 +11,5 @@ export type {
   KibanaMigratorStatus,
   MigrationStatus,
   MigrationResult,
+  MigrateDocumentOptions,
 } from './kibana_migrator';