Skip to content

Commit

Permalink
Merge remote-tracking branch 'upstream/main' into test-es-81400
Browse files Browse the repository at this point in the history
  • Loading branch information
Tyler Smalley committed Dec 9, 2021
2 parents 3ccb661 + 3672215 commit f9757e6
Show file tree
Hide file tree
Showing 915 changed files with 24,540 additions and 11,677 deletions.
10 changes: 5 additions & 5 deletions dev_docs/tutorials/expressions.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ const result = await executionContract.getData();
```
<DocCallOut>
Check the full spec of execute function [here](https://github.com/elastic/kibana/blob/main/docs/development/plugins/expressions/public/kibana-plugin-plugins-expressions-public.execution.md)
Check the full spec of execute function <DocLink id="kibExpressionsPluginApi" section="def-common.ExpressionsService.execute" text="here" />
</DocCallOut>
In addition, on the browser side, there are two additional ways to run expressions and render the results.
Expand All @@ -71,7 +71,7 @@ This is the easiest way to get expressions rendered inside your application.
```
<DocCallOut>
Check the full spec of ReactExpressionRenderer component props [here](https://github.com/elastic/kibana/blob/main/docs/development/plugins/expressions/public/kibana-plugin-plugins-expressions-public.reactexpressionrendererprops.md)
Check the full spec of ReactExpressionRenderer component props <DocLink id="kibExpressionsPluginApi" section="def-public.ReactExpressionRendererProps" text="here" />
</DocCallOut>
#### Expression loader
Expand All @@ -83,7 +83,7 @@ const handler = loader(domElement, expression, params);
```
<DocCallOut>
Check the full spec of expression loader params [here](https://github.com/elastic/kibana/blob/main/docs/development/plugins/expressions/public/kibana-plugin-plugins-expressions-public.iexpressionloaderparams.md)
Check the full spec of expression loader params <DocLink id="kibExpressionsPluginApi" section="def-public.IExpressionLoaderParams" text="here" />
</DocCallOut>
### Creating new expression functions
Expand All @@ -106,7 +106,7 @@ expressions.registerFunction(functionDefinition);
```
<DocCallOut>
Check the full interface of ExpressionFuntionDefinition [here](https://github.com/elastic/kibana/blob/main/docs/development/plugins/expressions/public/kibana-plugin-plugins-expressions-public.expressionfunctiondefinition.md)
Check the full interface of ExpressionFuntionDefinition <DocLink id="kibExpressionsPluginApi" section="def-common.ExpressionFunctionDefinition" text="here" />
</DocCallOut>
### Creating new expression renderers
Expand All @@ -128,5 +128,5 @@ expressions.registerRenderer(rendererDefinition);
```
<DocCallOut>
Check the full interface of ExpressionRendererDefinition [here](https://github.com/elastic/kibana/blob/main/docs/development/plugins/expressions/public/kibana-plugin-plugins-expressions-public.expressionrenderdefinition.md)
Check the full interface of ExpressionRendererDefinition <DocLink id="kibExpressionsPluginApi" section="def-common.ExpressionRenderDefinition" text="here" />
</DocCallOut>
68 changes: 0 additions & 68 deletions docs/developer/getting-started/debugging.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -130,71 +130,3 @@ Once you're finished, you can stop Kibana normally, then stop the {es} and APM s
----
./scripts/compose.py stop
----

=== Using {kib} server logs
{kib} Logs is a great way to see what's going on in your application and to debug performance issues. Navigating through a large number of generated logs can be overwhelming, and following are some techniques that you can use to optimize the process.

Start by defining a problem area that you are interested in. For example, you might be interested in seeing how a particular {kib} Plugin is performing, so no need to gather logs for all of {kib}. Or you might want to focus on a particular feature, such as requests from the {kib} server to the {es} server.
Depending on your needs, you can configure {kib} to generate logs for a specific feature.
[source,yml]
----
logging:
appenders:
file:
type: file
fileName: ./kibana.log
layout:
type: json
### gather all the Kibana logs into a file
logging.root:
appenders: [file]
level: all
### or gather a subset of the logs
logging.loggers:
### responses to an HTTP request
- name: http.server.response
level: debug
appenders: [file]
### result of a query to the Elasticsearch server
- name: elasticsearch.query
level: debug
appenders: [file]
### logs generated by my plugin
- name: plugins.myPlugin
level: debug
appenders: [file]
----
WARNING: Kibana's `file` appender is configured to produce logs in https://www.elastic.co/guide/en/ecs/master/ecs-reference.html[ECS JSON] format. It's the only format that includes the meta information necessary for https://www.elastic.co/guide/en/apm/agent/nodejs/current/log-correlation.html[log correlation] out-of-the-box.

The next step is to define what https://www.elastic.co/observability[observability tools] are available.
For a better experience, set up an https://www.elastic.co/guide/en/apm/get-started/current/observability-integrations.html[Observability integration] provided by Elastic to debug your application with the <<debugging-logs-apm-ui, APM UI.>>
To debug something quickly without setting up additional tooling, you can work with <<plain-kibana-logs, the plain {kib} logs.>>

[[debugging-logs-apm-ui]]
==== APM UI
*Prerequisites* {kib} logs are configured to be in https://www.elastic.co/guide/en/ecs/master/ecs-reference.html[ECS JSON] format to include tracing identifiers.

To debug {kib} with the APM UI, you must set up the APM infrastructure. You can find instructions for the setup process
https://www.elastic.co/guide/en/apm/get-started/current/observability-integrations.html[on the Observability integrations page].

Once you set up the APM infrastructure, you can enable the APM agent and put {kib} under load to collect APM events. To analyze the collected metrics and logs, use the APM UI as demonstrated https://www.elastic.co/guide/en/kibana/master/transactions.html#transaction-trace-sample[in the docs].

[[plain-kibana-logs]]
==== Plain {kib} logs
*Prerequisites* {kib} logs are configured to be in https://www.elastic.co/guide/en/ecs/master/ecs-reference.html[ECS JSON] format to include tracing identifiers.

Open {kib} Logs and search for an operation you are interested in.
For example, suppose you want to investigate the response times for queries to the `/api/telemetry/v2/clusters/_stats` {kib} endpoint.
Open Kibana Logs and search for the HTTP server response for the endpoint. It looks similar to the following (some fields are omitted for brevity).
[source,json]
----
{
"message":"POST /api/telemetry/v2/clusters/_stats 200 1014ms - 43.2KB",
"log":{"level":"DEBUG","logger":"http.server.response"},
"trace":{"id":"9b99131a6f66587971ef085ef97dfd07"},
"transaction":{"id":"d0c5bbf14f5febca"}
}
----
You are interested in the https://www.elastic.co/guide/en/ecs/current/ecs-tracing.html#field-trace-id[trace.id] field, which is a unique identifier of a trace. The `trace.id` provides a way to group multiple events, like transactions, which belong together. You can search for `"trace":{"id":"9b99131a6f66587971ef085ef97dfd07"}` to get all the logs that belong to the same trace. This enables you to see how many {es} requests were triggered during the `9b99131a6f66587971ef085ef97dfd07` trace, what they looked like, what {es} endpoints were hit, and so on.
6 changes: 3 additions & 3 deletions docs/settings/apm-settings.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -101,8 +101,8 @@ Changing these settings may disable features of the APM App.
| `xpack.apm.indices.sourcemap` {ess-icon}
| Matcher for all source map indices. Defaults to `apm-*`.

| `xpack.apm.autocreateApmIndexPattern` {ess-icon}
| Set to `false` to disable the automatic creation of the APM index pattern when the APM app is opened. Defaults to `true`.
| `xpack.apm.autoCreateApmDataView` {ess-icon}
| Set to `false` to disable the automatic creation of the APM data view when the APM app is opened. Defaults to `true`.
|===

// end::general-apm-settings[]
// end::general-apm-settings[]
76 changes: 42 additions & 34 deletions docs/settings/task-manager-settings.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,51 +9,59 @@ Task Manager runs background tasks by polling for work on an interval. You can

[float]
[[task-manager-settings]]
==== Task Manager settings
==== Task Manager settings

[cols="2*<"]
|===
| `xpack.task_manager.max_attempts`
| The maximum number of times a task will be attempted before being abandoned as failed. Defaults to 3.

| `xpack.task_manager.poll_interval`
| How often, in milliseconds, the task manager will look for more work. Defaults to 3000 and cannot be lower than 100.

| `xpack.task_manager.request_capacity`
| How many requests can Task Manager buffer before it rejects new requests. Defaults to 1000.
`xpack.task_manager.max_attempts`::
The maximum number of times a task will be attempted before being abandoned as failed. Defaults to 3.

| `xpack.task_manager.max_workers`
| The maximum number of tasks that this Kibana instance will run simultaneously. Defaults to 10.
Starting in 8.0, it will not be possible to set the value greater than 100.
`xpack.task_manager.poll_interval`::
How often, in milliseconds, the task manager will look for more work. Defaults to 3000 and cannot be lower than 100.

| `xpack.task_manager.`
`monitored_stats_health_verbose_log.enabled`
| This flag will enable automatic warn and error logging if task manager self detects a performance issue, such as the time between when a task is scheduled to execute and when it actually executes. Defaults to false.
`xpack.task_manager.request_capacity`::
How many requests can Task Manager buffer before it rejects new requests. Defaults to 1000.

| `xpack.task_manager.`
`monitored_stats_health_verbose_log.`
`warn_delayed_task_start_in_seconds`
| The amount of seconds we allow a task to delay before printing a warning server log. Defaults to 60.
`xpack.task_manager.max_workers`::
The maximum number of tasks that this Kibana instance will run simultaneously. Defaults to 10.
Starting in 8.0, it will not be possible to set the value greater than 100.

| `xpack.task_manager.ephemeral_tasks.enabled`
| Enables an experimental feature that executes a limited (and configurable) number of actions in the same task as the alert which triggered them.
These action tasks will reduce the latency of the time it takes an action to run after it's triggered, but are not persisted as SavedObjects.
These non-persisted action tasks have a risk that they won't be run at all if the Kibana instance running them exits unexpectedly. Defaults to false.
`xpack.task_manager.monitored_stats_health_verbose_log.enabled`::
This flag will enable automatic warn and error logging if task manager self detects a performance issue, such as the time between when a task is scheduled to execute and when it actually executes. Defaults to false.

`xpack.task_manager.monitored_stats_health_verbose_log.warn_delayed_task_start_in_seconds`::
The amount of seconds we allow a task to delay before printing a warning server log. Defaults to 60.

`xpack.task_manager.ephemeral_tasks.enabled`::
Enables an experimental feature that executes a limited (and configurable) number of actions in the same task as the alert which triggered them.
These action tasks will reduce the latency of the time it takes an action to run after it's triggered, but are not persisted as SavedObjects.
These non-persisted action tasks have a risk that they won't be run at all if the Kibana instance running them exits unexpectedly. Defaults to false.

`xpack.task_manager.ephemeral_tasks.request_capacity`::
Sets the size of the ephemeral queue defined above. Defaults to 10.

| `xpack.task_manager.ephemeral_tasks.request_capacity`
| Sets the size of the ephemeral queue defined above. Defaults to 10.
|===

[float]
[[task-manager-health-settings]]
==== Task Manager Health settings
==== Task Manager Health settings

Settings that configure the <<task-manager-health-monitoring>> endpoint.

[cols="2*<"]
|===
| `xpack.task_manager.`
`monitored_task_execution_thresholds`
| Configures the threshold of failed task executions at which point the `warn` or `error` health status is set under each task type execution status (under `stats.runtime.value.execution.result_frequency_percent_as_number[${task type}].status`). This setting allows configuration of both the default level and a custom task type specific level. By default, this setting is configured to mark the health of every task type as `warning` when it exceeds 80% failed executions, and as `error` at 90%. Custom configurations allow you to reduce this threshold to catch failures sooner for task types that you might consider critical, such as alerting tasks. This value can be set to any number between 0 to 100, and a threshold is hit when the value *exceeds* this number. This means that you can avoid setting the status to `error` by setting the threshold at 100, or hit `error` the moment any task fails by setting the threshold to 0 (as it will exceed 0 once a single failure occurs).

|===
`xpack.task_manager.monitored_task_execution_thresholds`::
Configures the threshold of failed task executions at which point the `warn` or
`error` health status is set under each task type execution status
(under `stats.runtime.value.execution.result_frequency_percent_as_number[${task type}].status`).
+
This setting allows configuration of both the default level and a
custom task type specific level. By default, this setting is configured to mark
the health of every task type as `warning` when it exceeds 80% failed executions,
and as `error` at 90%.
+
Custom configurations allow you to reduce this threshold to catch failures sooner
for task types that you might consider critical, such as alerting tasks.
+
This value can be set to any number between 0 to 100, and a threshold is hit
when the value *exceeds* this number. This means that you can avoid setting the
status to `error` by setting the threshold at 100, or hit `error` the moment
any task fails by setting the threshold to 0 (as it will exceed 0 once a
single failure occurs).
23 changes: 10 additions & 13 deletions docs/settings/telemetry-settings.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,29 +17,26 @@ See our https://www.elastic.co/legal/privacy-statement[Privacy Statement] to lea
[[telemetry-general-settings]]
==== General telemetry settings

[cols="2*<"]
|===
|[[telemetry-enabled]] `telemetry.enabled`
| Set to `true` to send cluster statistics to Elastic. Reporting your

[[telemetry-enabled]] `telemetry.enabled`::
Set to `true` to send cluster statistics to Elastic. Reporting your
cluster statistics helps us improve your user experience. Your data is never
shared with anyone. Set to `false` to disable statistics reporting from any
browser connected to the {kib} instance. Defaults to `true`.

| `telemetry.sendUsageFrom`
| Set to `'server'` to report the cluster statistics from the {kib} server.
`telemetry.sendUsageFrom`::
Set to `'server'` to report the cluster statistics from the {kib} server.
If the server fails to connect to our endpoint at https://telemetry.elastic.co/, it assumes
it is behind a firewall and falls back to `'browser'` to send it from users' browsers
when they are navigating through {kib}. Defaults to `'server'`.

|[[telemetry-optIn]] `telemetry.optIn`
| Set to `true` to automatically opt into reporting cluster statistics. You can also opt out through
[[telemetry-optIn]] `telemetry.optIn`::
Set to `true` to automatically opt into reporting cluster statistics. You can also opt out through
*Advanced Settings* in {kib}. Defaults to `true`.

| `telemetry.allowChangingOptInStatus`
| Set to `true` to allow overwriting the <<telemetry-optIn, `telemetry.optIn`>> setting via the {kib} UI. Defaults to `true`. +

|===

`telemetry.allowChangingOptInStatus`::
Set to `true` to allow overwriting the <<telemetry-optIn, `telemetry.optIn`>> setting via the {kib} UI. Defaults to `true`. +
+
[NOTE]
============
When `false`, <<telemetry-optIn, `telemetry.optIn`>> must be `true`. To disable telemetry and not allow users to change that parameter, use <<telemetry-enabled, `telemetry.enabled`>>.
Expand Down
13 changes: 10 additions & 3 deletions docs/setup/upgrade.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,13 +44,20 @@ a|
[[upgrade-before-you-begin]]
=== Before you begin

WARNING: {kib} automatically runs upgrade migrations when required. To roll back to an earlier version in case of an upgrade failure, you **must** have a {ref}/snapshot-restore.html[backup snapshot] available. This snapshot must include the `kibana` feature state or all `kibana*` indices. For more information see <<upgrade-migrations, upgrade migrations>>.
[WARNING]
====
{kib} automatically runs upgrade migrations when required. To roll back to an
earlier version in case of an upgrade failure, you **must** have a
{ref}/snapshot-restore.html[backup snapshot] that includes the `kibana` feature
state. Snapshots include this feature state by default.
For more information, refer to <<upgrade-migrations, upgrade migrations>>.
====

Before you upgrade {kib}:

* Consult the <<breaking-changes,breaking changes>>.
* {ref}/snapshots-take-snapshot.html[Take a snapshot] of your data. To roll back to an earlier version, the snapshot must include the `kibana` feature state or all `.kibana*` indices.
* Although not a requirement for rollbacks, we recommend taking a snapshot of all {kib} indices created by the plugins you use such as the `.reporting*` indices created by the reporting plugin.
* {ref}/snapshots-take-snapshot.html[Take a snapshot] of your data. To roll back to an earlier version, the snapshot must include the `kibana` feature state.
* Before you upgrade production servers, test the upgrades in a dev environment.
* See <<preventing-migration-failures, preventing migration failures>> for common reasons upgrades fail and how to prevent these.
* If you are using custom plugins, check that a compatible version is
Expand Down
7 changes: 4 additions & 3 deletions docs/setup/upgrade/upgrade-migrations.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -151,17 +151,18 @@ In order to rollback after a failed upgrade migration, the saved object indices
[float]
===== Rollback by restoring a backup snapshot:

1. Before proceeding, {ref}/snapshots-take-snapshot.html[take a snapshot] that contains the `kibana` feature state or all `.kibana*` indices.
1. Before proceeding, {ref}/snapshots-take-snapshot.html[take a snapshot] that contains the `kibana` feature state.
Snapshots include this feature state by default.
2. Shutdown all {kib} instances to be 100% sure that there are no instances currently performing a migration.
3. Delete all saved object indices with `DELETE /.kibana*`
4. {ref}/snapshots-restore-snapshot.html[Restore] the `kibana` feature state or all `.kibana* indices and their aliases from the snapshot.
4. {ref}/snapshots-restore-snapshot.html[Restore] the `kibana` feature state from the snapshot.
5. Start up all {kib} instances on the older version you wish to rollback to.

[float]
===== (Not recommended) Rollback without a backup snapshot:

1. Shutdown all {kib} instances to be 100% sure that there are no {kib} instances currently performing a migration.
2. {ref}/snapshots-take-snapshot.html[Take a snapshot] that includes the `kibana` feature state or all `.kibana*` indices.
2. {ref}/snapshots-take-snapshot.html[Take a snapshot] that includes the `kibana` feature state. Snapshots include this feature state by default.
3. Delete the version specific indices created by the failed upgrade migration. E.g. if you wish to rollback from a failed upgrade to v7.12.0 `DELETE /.kibana_7.12.0_*,.kibana_task_manager_7.12.0_*`
4. Inspect the output of `GET /_cat/aliases`. If either the `.kibana` and/or `.kibana_task_manager` alias is missing, these will have to be created manually. Find the latest index from the output of `GET /_cat/indices` and create the missing alias to point to the latest index. E.g. if the `.kibana` alias was missing and the latest index is `.kibana_3` create a new alias with `POST /.kibana_3/_aliases/.kibana`.
5. Remove the write block from the rollback indices. `PUT /.kibana,.kibana_task_manager/_settings {"index.blocks.write": false}`
Expand Down
Loading

0 comments on commit f9757e6

Please sign in to comment.