Merge branch 'master' into moving-file-data-visualizer-UI-to-file-upl…

…oad-plugin

.bazelrc.common

@@ -10,12 +10,13 @@
 build --experimental_guard_against_concurrent_changes
 run --experimental_guard_against_concurrent_changes
 test --experimental_guard_against_concurrent_changes
+query --experimental_guard_against_concurrent_changes
 
 ## Cache action outputs on disk so they persist across output_base and bazel shutdown (eg. changing branches)
-build --disk_cache=~/.bazel-cache/disk-cache
+common --disk_cache=~/.bazel-cache/disk-cache
 
 ## Bazel repo cache settings
-build --repository_cache=~/.bazel-cache/repository-cache
+common --repository_cache=~/.bazel-cache/repository-cache
 
 # Bazel will create symlinks from the workspace directory to output artifacts.
 # Build results will be placed in a directory called "bazel-bin"
@@ -35,13 +36,16 @@ build --experimental_inprocess_symlink_creation
 # Incompatible flags to run with
 build --incompatible_no_implicit_file_export
 build --incompatible_restrict_string_escapes
+query --incompatible_no_implicit_file_export
+query --incompatible_restrict_string_escapes
 
 # Log configs
 ## different from default
 common --color=yes
-build --show_task_finish
-build --noshow_progress
+common --noshow_progress
+common --show_task_finish
 build --noshow_loading_progress
+query --noshow_loading_progress
 build --show_result=0
 
 # Specifies desired output mode for running tests.
@@ -82,7 +86,7 @@ test:debug --test_output=streamed --test_strategy=exclusive --test_timeout=9999
 run:debug --define=VERBOSE_LOGS=1 -- --node_options=--inspect-brk
 # The following option will change the build output of certain rules such as terser and may not be desirable in all cases
 # It will also output both the repo cache and action cache to a folder inside the repo
-build:debug --compilation_mode=dbg --show_result=1
+build:debug --compilation_mode=dbg --show_result=0 --noshow_loading_progress --noshow_progress --show_task_finish
 
 # Turn off legacy external runfiles
 # This prevents accidentally depending on this feature, which Bazel will remove.

WORKSPACE.bazel

@@ -52,9 +52,6 @@ node_repositories(
 # NOTE: FORCE_COLOR env var forces colors on non tty mode
 yarn_install(
   name = "npm",
-  environment = {
-    "FORCE_COLOR": "True",
-  },
   package_json = "//:package.json",
   yarn_lock = "//:yarn.lock",
   data = [

docs/concepts/index.asciidoc

@@ -0,0 +1,149 @@
+[[kibana-concepts-analysts]]
+== {kib} concepts for analysts
+**_Learn the shared concepts for analyzing and visualizing your data_**
+
+As an analyst, you will use a combination of {kib} apps to analyze and
+visualize your data. {kib} contains both general-purpose apps and apps for the
+https://www.elastic.co/guide/en/enterprise-search/current/index.html[*Enterprise Search*],
+{observability-guide}/observability-introduction.html[*Elastic Observability*],
+and {security-guide}/es-overview.html[*Elastic Security*] solutions.
+These apps share a common set of concepts.
+
+[float]
+=== Three things to know about {es}
+
+You don't need to know everything about {es} to use {kib}, but the most important concepts follow:
+
+* *{es} makes JSON documents searchable and aggregatable.* The documents are
+stored in an {ref}/documents-indices.html[index] or {ref}/data-streams.html[data stream], which represent one type of data.
+
+* **_Searchable_ means that you can filter the documents for conditions.**
+For example, you can filter for data "within the last 7 days" or data that "contains the word {kib}".
+{kib} provides many ways for you to construct filters, which are also called queries or search terms.
+
+* **_Aggregatable_ means that you can extract summaries from matching documents.**
+The simplest aggregation is *count*, and it is frequently used in combination
+with the *date histogram*, to see count over time. The *terms* aggregation shows the most frequent values.
+
+[float]
+=== Finding your apps and objects
+
+{kib} offers a <<kibana-navigation-search,global search bar>> on every page that you can use to find any app or saved object.
+Open the search bar using the keyboard shortcut Ctrl+/ on Windows and Linux, Command+/ on MacOS.
+
+[role="screenshot"]
+image:concepts/images/global-search.png["Global search showing matches to apps and saved objects for the word visualize"]
+
+[float]
+=== Accessing data with index patterns
+
+{kib} requires an index pattern to tell it which {es} data you want to access,
+and whether the data is time-based. An index pattern can point to one or more {es}
+data streams, indices, or index aliases by name.
+For example, `logs-elasticsearch-prod-*` is an index pattern,
+and it is time-based with a time field of `@timestamp`. The time field is not editable.
+
+Index patterns are typically created by an administrator when sending data to {es}.
+You can <<index-patterns,create or update index patterns>> in *Stack Management*, or by using a script
+that accesses the {kib} API.
+
+{kib} uses the index pattern to show you a list of fields, such as
+`event.duration`. You can customize the display name and format for each field.
+For example, you can tell Kibana to display `event.duration` in seconds.
+{kib} has <<managing-fields,field formatters>> for strings,
+dates, geopoints,
+and numbers.
+
+[float]
+=== Searching your data
+
+{kib} provides you several ways to build search queries,
+which will reduce the number of document matches that you get from {es}.
+Each app in {kib} provides a time filter, and most apps also include semi-structured search and extra filters.
+
+[role="screenshot"]
+image:concepts/images/top-bar.png["Time filter, semi-structured search, and filters in a {kib} app"]
+
+If you frequently use any of the search options, you can click the
+save icon
+image:concepts/images/save-icon.png["save icon"] next to the
+semi-structured search to save or load a previously saved query.
+The saved query will always contain the semi-structured search query,
+and can optionally contain the time filter and extra filters.
+
+[float]
+==== Time filter
+
+The <<set-time-filter, global time filter>> limits the time range of data displayed.
+In most cases, the time filter applies to the time field in the index pattern,
+but some apps allow you to use a different time field.
+
+Using the time filter, you can configure a refresh rate to periodically
+resubmit your searches. You can also click *Refresh* to resubmit the search.
+This might be useful if you use {kib} to monitor the underlying data.
+
+[role="screenshot"]
+image:concepts/images/refresh-every.png["section of time filter where you can configure a refresh rate"]
+
+
+[float]
+==== Semi-structured search
+
+Combine free text search with field-based search using the Kibana Query Language (KQL).
+Type a search term to match across all fields, or start typing a field name to
+get suggestions for field names and operators that you can use to build a structured query.
+The semi-structured search will filter documents for matches, and only return matching documents.
+
+Following are some example KQL queries.  For more detailed examples, refer to <<kuery-query,Kibana Query Language>>.
+
+[cols=2*]
+|===
+| Exact phrase query
+| `http.response.body.content.text:"quick brown fox"`
+
+| Terms query
+| http.response.status_code:400 401 404
+
+| Boolean query
+| `response:200 or extension:php`
+
+| Range query
+| `account_number >= 100 and items_sold <= 200`
+
+| Wildcard query
+| `machine.os:win*`
+|===
+
+[float]
+==== Additional filters with AND
+
+Structured filters are a more interactive way to create {es} queries,
+and are commonly used when building dashboards that are shared by multiple analysts.
+Each filter can be disabled, inverted, or pinned across all apps.
+The structured filters are the only way to use the {es} Query DSL in JSON form,
+or to target a specific index pattern for filtering. Each of the structured
+filters is combined with AND logic on the rest of the query.
+
+[role="screenshot"]
+image:concepts/images/add-filter-popup.png["Add filter popup"]
+
+[float]
+=== Saving objects
+{kib} lets you save objects for your own future use or for sharing with others.
+Each <<managing-saved-objects,saved object>> type has different abilities. For example, you can save
+your search queries made with *Discover*, which lets you:
+
+* Share a link to your search
+* Download the full search results in CSV form
+* Start an aggregated visualization using the same search query
+* Embed the *Discover* search results into a dashboard
+* Embed the *Discover* search results into a Canvas workpad
+
+For organization, every saved object can have a name, <<kibana-navigation-search,tags>>, and type.
+Use the global search to quickly open a saved object.
+
+[float]
+=== What's next?
+
+* Try the {kib} <<get-started,Quick start>>, which shows you how to put these concepts into action.
+* Go to <<discover, Discover>> for instructions on searching your data.

docs/concepts/save-query.asciidoc

@@ -0,0 +1,39 @@
+[[save-load-delete-query]]
+== Save a query
+A saved query is a collection of query text and filters that you can
+reuse in any app with a query bar, like <<discover, *Discover*>> and <<dashboard, *Dashboard*>>. Save a query when you want to:
+
+* Retrieve results from the same query at a later time without having to reenter the query text, add the filters or set the time filter
+* View the results of the same query in multiple apps
+* Share your query
+
+Saved queries don't include information specific to *Discover*,
+such as the currently selected columns in the document table, the sort order, and the index pattern.
+To save your current view of *Discover* for later retrieval and reuse,
+create a <<save-open-search, saved search>> instead.
+
+NOTE::
+
+If you have insufficient privileges to save queries, the *Save current query*
+button isn't visible in the saved query management popover.
+For more information, see <<xpack-security-authorization, Granting access to Kibana>>
+
+. Click *#* in the query bar.
+. In the popover, click *Save current query*.
++
+[role="screenshot"]
+image::discover/images/saved-query-management-component-all-privileges.png["Example of the saved query management popover with a list of saved queries with write access",width="80%"]
++
+. Enter a name, a description, and then select the filter options.
+By default, filters are automatically included, but the time filter is not.
++
+[role="screenshot"]
+image::discover/images/saved-query-save-form-default-filters.png["Example of the saved query management save form with the filters option included and the time filter option excluded",width="80%"]
+. Click *Save*.
+. To load a saved query into *Discover* or *Dashboard*, open the *Saved search* popover, and select the query.
+. To manage your saved queries, use these actions in the popover:
++
+* Save as new:  Save changes to the current query.
+* Clear.  Clear a query that is currently loaded in an app.
+* Delete.  You can’t recover a deleted query.
+. To import and export saved queries, go to <<managing-saved-objects, Saved Objects in Management>>.

docs/developer/getting-started/index.asciidoc

@@ -66,7 +66,8 @@ yarn kbn bootstrap --force-install
 
 (You can also run `yarn kbn` to see the other available commands. For
 more info about this tool, see
-{kib-repo}tree/{branch}/packages/kbn-pm[{kib-repo}tree/{branch}/packages/kbn-pm].)
+{kib-repo}tree/{branch}/packages/kbn-pm[{kib-repo}tree/{branch}/packages/kbn-pm]. If you want more 
+information about how to actively develop over packages please read <<monorepo-packages>>)
 
 When switching branches which use different versions of npm packages you
 may need to run:
@@ -169,3 +170,5 @@ include::debugging.asciidoc[leveloffset=+1]
 include::building-kibana.asciidoc[leveloffset=+1]
 
 include::development-plugin-resources.asciidoc[leveloffset=+1]
+
+include::monorepo-packages.asciidoc[leveloffset=+1]

docs/developer/getting-started/monorepo-packages.asciidoc

@@ -0,0 +1,66 @@
+[[monorepo-packages]]
+== {kib} Monorepo Packages
+
+Currently {kib} works as a monorepo composed by a core, plugins and packages.
+The latest are located in a folder called `packages` and are pieces of software that 
+composes a set of features that can be isolated and reused across the entire repository.
+They are also supposed to be able to imported just like any other `node_module`.
+
+Previously we relied solely on `@kbn/pm` to manage the development tools of those packages, but we are 
+now in the middle of migrating those responsibilities into Bazel. Every package already migrated 
+will contain in its root folder a `BUILD.bazel` file and other `build` and `watching` strategies should be used.
+
+Remember that any time you need to make sure the monorepo is ready to be used just run:
+
+[source,bash]
+----
+yarn kbn bootstrap
+----
+
+[discrete]
+=== Building Non Bazel Packages
+
+Non Bazel packages can be built independently with
+
+[source,bash]
+----
+yarn kbn run build -i PACKAGE_NAME
+----
+
+[discrete]
+=== Watching Non Bazel Packages
+
+Non Bazel packages can be watched independently with
+
+[source,bash]
+----
+yarn kbn watch -i PACKAGE_NAME
+----
+
+[discrete]
+=== Building Bazel Packages
+
+Bazel packages are built as a whole for now. You can use:
+
+[source,bash]
+----
+yarn kbn build-bazel
+----
+
+[discrete]
+=== Watching Bazel Packages
+
+Bazel packages are watched as a whole for now. You can use:
+
+[source,bash]
+----
+yarn kbn watch-bazel
+----
+
+
+[discrete]
+=== List of Already Migrated Packages to Bazel
+
+- @elastic/datemath
+
+

docs/development/plugins/embeddable/server/kibana-plugin-plugins-embeddable-server.embeddablestart.md

@@ -0,0 +1,11 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-plugins-embeddable-server](./kibana-plugin-plugins-embeddable-server.md) &gt; [EmbeddableStart](./kibana-plugin-plugins-embeddable-server.embeddablestart.md)
+
+## EmbeddableStart type
+
+<b>Signature:</b>
+
+```typescript
+export declare type EmbeddableStart = PersistableStateService<EmbeddableStateWithType>;
+```

docs/development/plugins/embeddable/server/kibana-plugin-plugins-embeddable-server.md

@@ -18,3 +18,9 @@
 |  --- | --- |
 |  [plugin](./kibana-plugin-plugins-embeddable-server.plugin.md) |  |
 
+## Type Aliases
+
+|  Type Alias | Description |
+|  --- | --- |
+|  [EmbeddableStart](./kibana-plugin-plugins-embeddable-server.embeddablestart.md) |  |
+

docs/management/index-patterns.asciidoc

@@ -125,6 +125,11 @@ pattern:
 *:logstash-*
 ```
 
+You can use exclusions to exclude indices that might contain mapping errors.
+To match indices starting with `logstash-`, and exclude those starting with `logstash-old` from 
+all clusters having a name starting with `cluster_`, you can use `cluster_*:logstash-*,cluster*:logstash-old*`.
+To exclude a cluster, use `cluster_*:logstash-*,cluster_one:-*`.
+
 Once an index pattern is configured using the {ccs} syntax, all searches and
 aggregations using that index pattern in {kib} take advantage of {ccs}.
 

docs/settings/dev-settings.asciidoc

@@ -29,3 +29,14 @@ They are enabled by default.
   | Set to `true` to enable the <<xpack-profiler,{searchprofiler}>>. Defaults to `true`.
 
 |===
+
+[float]
+[[painless_lab-settings]]
+==== Painless Lab settings
+
+[cols="2*<"]
+|===
+| `xpack.painless_lab.enabled`
+  | When set to `true`, enables the <<painlesslab, Painless Lab>>. Defaults to `true`.
+
+|===