diff --git a/docs/configuring.asciidoc b/docs/configuring.asciidoc index 0def3f96690..390de5b4c97 100644 --- a/docs/configuring.asciidoc +++ b/docs/configuring.asciidoc @@ -27,7 +27,7 @@ include::./copied-from-beats/shared-ssl-config.asciidoc[] include::./template-config.asciidoc[] -include::./ilm-setup.asciidoc[] +include::./ilm.asciidoc[] include::./copied-from-beats/loggingconfig.asciidoc[] diff --git a/docs/copied-from-beats/command-reference.asciidoc b/docs/copied-from-beats/command-reference.asciidoc index e1ed7c3146f..9efa2c6678e 100644 --- a/docs/copied-from-beats/command-reference.asciidoc +++ b/docs/copied-from-beats/command-reference.asciidoc @@ -17,11 +17,11 @@ :deploy-command-short-desc: Deploys the specified function to your serverless environment ifndef::no_dashboards[] -:export-command-short-desc: Exports the configuration, index template, or a dashboard to stdout +:export-command-short-desc: Exports the configuration, index template, ilm-policy or a dashboard to stdout endif::no_dashboards[] ifdef::no_dashboards[] -:export-command-short-desc: Exports the configuration or index template to stdout +:export-command-short-desc: Exports the configuration, index template, or ilm-policy to stdout endif::no_dashboards[] :help-command-short-desc: Shows help for any command @@ -32,15 +32,15 @@ endif::no_dashboards[] :run-command-short-desc: Runs {beatname_uc}. This command is used by default if you start {beatname_uc} without specifying a command ifdef::has_ml_jobs[] -:setup-command-short-desc: Sets up the initial environment, including the index template, {kib} dashboards (when available), and machine learning jobs (when available) +:setup-command-short-desc: Sets up the initial environment, including the index template, ilm policy and write alias, {kib} dashboards (when available), and machine learning jobs (when available) endif::[] ifdef::no_dashboards[] -:setup-command-short-desc: Sets up the initial environment, including the ES index template +:setup-command-short-desc: Sets up the initial environment, including the ES index template, ilm policy and write alias endif::no_dashboards[] ifndef::has_ml_jobs,no_dashboards[] -:setup-command-short-desc: Sets up the initial environment, including the index template and {kib} dashboards (when available) +:setup-command-short-desc: Sets up the initial environment, including the index template, ilm policy and write alias, and {kib} dashboards (when available) endif::[] :update-command-short-desc: Updates the specified function @@ -82,23 +82,23 @@ endif::[] [options="header"] |======================= |Commands | -ifeval::[("{beatname_lc}"=="functionbeat")] +ifeval::["{beatname_lc}"=="functionbeat"] |<> | {deploy-command-short-desc}. endif::[] |<> |{export-command-short-desc}. |<> |{help-command-short-desc}. |<> |{keystore-command-short-desc}. -ifeval::[("{beatname_lc}"=="functionbeat")] +ifeval::["{beatname_lc}"=="functionbeat"] |<> |{package-command-short-desc}. |<> |{remove-command-short-desc}. endif::[] -ifeval::[("{beatname_lc}"=="filebeat") or ("{beatname_lc}"=="metricbeat")] +ifdef::has_modules_command[] |<> |{modules-command-short-desc}. endif::[] |<> |{run-command-short-desc}. |<> |{setup-command-short-desc}. |<> |{test-command-short-desc}. -ifeval::[("{beatname_lc}"=="functionbeat")] +ifeval::["{beatname_lc}"=="functionbeat"] |<> |{update-command-short-desc}. endif::[] |<> |{version-command-short-desc}. @@ -106,7 +106,7 @@ endif::[] Also see <>. -ifeval::[("{beatname_lc}"=="functionbeat")] +ifeval::["{beatname_lc}"=="functionbeat"] [[deploy-command]] ==== `deploy` command @@ -145,13 +145,13 @@ endif::[] ifndef::no_dashboards[] {export-command-short-desc}. You can use this command to quickly view your configuration, see the contents of the index -template, or export a dashboard from {kib}. +template and the ilm policy, or export a dashboard from {kib}. endif::no_dashboards[] ifdef::no_dashboards[] {export-command-short-desc}. You can use this command to quickly view your configuration or see the contents of the index -template. +template or the ilm policy. endif::no_dashboards[] *SYNOPSIS* @@ -194,19 +194,21 @@ endif::no_dashboards[] [[template-subcommand]]*`template`*:: Exports the index template to stdout. You can specify the `--es.version` and -`--index` flags to further define what gets exported. +`--index` flags to further define what gets exported. Furthermore you can export +the template to a file instead of `stdout` by defining a directory via `--dir`. -ifndef::apm-server[] [[ilm-policy-subcommand]] *`ilm-policy`*:: -Exports ILM policy to stdout. -endif::apm-server[] +Exports ILM policy to stdout. You can specify the `--es.version` and a `--dir` +to which the policy should be exported as a file rather than exporting to `stdout`. *FLAGS* *`--es.version VERSION`*:: When used with <>, exports an index template that is compatible with the specified version. +When used with <>, exports the ilm policy +if the specified ES version is enabled for ILM. *`-h, --help`*:: Shows help for the `export` command. @@ -216,6 +218,10 @@ When used with <>, sets the base name to use for the index template. If this flag is not specified, the default base name is +{beatname_lc}+. +*`--dir DIRNAME`*:: +Define a directory to which the template and ilm-policy should be exported to +as files instead of printing them to `stdout`. + ifndef::no_dashboards[] *`--id DASHBOARD_ID`*:: When used with <>, specifies the dashboard ID. @@ -327,7 +333,7 @@ Shows help for the `keystore` command. See <> for more examples. -ifeval::[("{beatname_lc}"=="functionbeat")] +ifeval::["{beatname_lc}"=="functionbeat"] [[package-command]] ==== `package` command @@ -389,7 +395,7 @@ Shows help for the `remove` command. ----- endif::[] -ifeval::[("{beatname_lc}"=="filebeat") or ("{beatname_lc}"=="metricbeat")] +ifdef::has_modules_command[] [[modules-command]] ==== `modules` command @@ -584,6 +590,10 @@ Or: {setup-command-short-desc} * The index template ensures that fields are mapped correctly in Elasticsearch. +If index lifecycle management is enabled it also ensures that the defined ILM policy +and write alias are connected to the indices matching the index template. +The ILM policy takes care of the lifecycle of an index, when to do a rollover, +when to move an index from the hot phase to the next phase etc. ifndef::no_dashboards[] * The {kib} dashboards make it easier for you to visualize {beatname_uc} data @@ -636,8 +646,19 @@ enabled modules in the +{beatname_lc}.yml+ file. If you used the directory, also specify the `--modules` flag. endif::[] +*`--index-management`*:: +Sets up components related to Elasticsearch index management including +template, ilm policy, and write alias. + *`--template`*:: +deprecated[7.2] Sets up the index template only. +It is recommended to use `--index-management` instead. + +*`--ilm-policy`*:: +deprecated[7.2] +Sets up the index lifecycle policy. +It is recommended to use `--index-management` instead. {global-flags} @@ -650,7 +671,7 @@ ifeval::["{beatname_lc}"=="filebeat"] {beatname_lc} setup --machine-learning {beatname_lc} setup --pipelines {beatname_lc} setup --pipelines --modules system,nginx,mysql <1> -{beatname_lc} setup --template +{beatname_lc} setup --index-management ----- <1> If you used the <> command to enable modules in the `modules.d` directory, also specify the `--modules` flag to indicate which @@ -664,14 +685,14 @@ ifndef::no_dashboards[] ----- {beatname_lc} setup --dashboards {beatname_lc} setup --machine-learning -{beatname_lc} setup --template +{beatname_lc} setup --index-management ----- endif::no_dashboards[] ifdef::no_dashboards[] ["source","sh",subs="attributes"] ----- {beatname_lc} setup --machine-learning -{beatname_lc} setup --template +{beatname_lc} setup --index-management ----- endif::no_dashboards[] @@ -732,7 +753,7 @@ ifeval::["{beatname_lc}"=="metricbeat"] ----- endif::[] -ifeval::[("{beatname_lc}"=="functionbeat")] +ifeval::["{beatname_lc}"=="functionbeat"] [[update-command]] ==== `update` command diff --git a/docs/copied-from-beats/keystore.asciidoc b/docs/copied-from-beats/keystore.asciidoc index d5588073008..45eb297ac9c 100644 --- a/docs/copied-from-beats/keystore.asciidoc +++ b/docs/copied-from-beats/keystore.asciidoc @@ -64,7 +64,7 @@ To create a secrets keystore, use: ---------------------------------------------------------------- -{beatname_uc} creates the keystore in the directory defined by the `path.config` +{beatname_uc} creates the keystore in the directory defined by the `path.data` configuration setting. [float] diff --git a/docs/copied-from-beats/monitoring/monitoring-beats.asciidoc b/docs/copied-from-beats/monitoring/monitoring-beats.asciidoc index f9dee1ed6f6..c750eefbea5 100644 --- a/docs/copied-from-beats/monitoring/monitoring-beats.asciidoc +++ b/docs/copied-from-beats/monitoring/monitoring-beats.asciidoc @@ -38,9 +38,13 @@ configured the {es} output and want to send {beatname_uc} monitoring events to the same {es} cluster, specify the following minimal configuration: + -- -[source, yml] +["source","yml",subs="attributes"] -------------------- -monitoring.enabled: true +monitoring: + enabled: true + elasticsearch: + username: {beat_monitoring_user} + password: somepassword -------------------- If you configured a different output, such as {ls} or you want to send {beatname_uc} diff --git a/docs/copied-from-beats/outputconfig.asciidoc b/docs/copied-from-beats/outputconfig.asciidoc index 8f1742a9588..1fbdc591152 100644 --- a/docs/copied-from-beats/outputconfig.asciidoc +++ b/docs/copied-from-beats/outputconfig.asciidoc @@ -1568,8 +1568,11 @@ endif::[] ++++ ifdef::apm-server[] -NOTE: This page refers to using a separate instance of APM Server with an existing Elasticsearch Service deployment. -APM Server is not yet supported on Elasticsearch Service. +NOTE: This page refers to using a separate instance of APM Server with an existing +https://www.elastic.co/cloud/elasticsearch-service[Elasticsearch Service deployment]. +If you want to use APM on Elastic Cloud, see the cloud docs: +{cloud}/ec-create-deployment.html[Create your deployment] or +{cloud}/ec-manage-apm-settings.html[Add APM user settings]. endif::apm-server[] {beatname_uc} comes with two settings that simplify the output configuration diff --git a/docs/copied-from-beats/security/basic-auth.asciidoc b/docs/copied-from-beats/security/basic-auth.asciidoc index 986b277d630..02f276f17d3 100644 --- a/docs/copied-from-beats/security/basic-auth.asciidoc +++ b/docs/copied-from-beats/security/basic-auth.asciidoc @@ -6,168 +6,57 @@ When sending data to a secured cluster through the `elasticsearch` output, {beatname_uc} must either provide basic authentication credentials or present a client certificate. -To configure authentication credentials for {beatname_uc}: +*Before you begin:* <>. -. Create a writer role that has the following privileges: -+ --- -ifeval::["{beatname_lc}"!="filebeat"] -* *Cluster*: `manage_index_templates` and `monitor` -endif::[] -ifeval::["{beatname_lc}"=="filebeat"] -* *Cluster*: `manage_index_templates`, `monitor`, and -`manage_ingest_pipelines` -endif::[] -* *Index*: `write` and `create_index` on the {beatname_uc} indices --- -+ -You can create roles from the **Management / Roles** UI in {kib} or through the -`role` API. For example, the following request creates a role named -++{beat_default_index_prefix}_writer++: -+ --- -ifeval::["{beatname_lc}"!="filebeat"] -["source","sh",subs="attributes,callouts"] ---------------------------------------------------------------- -POST _security/role/{beat_default_index_prefix}_writer -{ - "cluster": ["manage_index_templates","monitor"], - "indices": [ - { - "names": [ "{beat_default_index_prefix}-*" ], <1> - "privileges": ["write","create_index"] - } - ] -} ---------------------------------------------------------------- -// CONSOLE -<1> If you use a custom {beatname_uc} index pattern, specify that pattern -instead of the default ++{beat_default_index_prefix}-*++ pattern. -endif::[] -ifeval::["{beatname_lc}"=="filebeat"] -["source","sh",subs="attributes,callouts"] ---------------------------------------------------------------- -POST _security/role/{beat_default_index_prefix}_writer -{ - "cluster": ["manage_index_templates","monitor","manage_ingest_pipelines"], <1> - "indices": [ - { - "names": [ "{beat_default_index_prefix}-*" ], <2> - "privileges": ["write","create_index"] - } - ] -} ---------------------------------------------------------------- -// CONSOLE -<1> The `manage_ingest_pipelines` cluster privilege is required to run -{beatname_uc} modules. -<2> If you use a custom {beatname_uc} index pattern, specify that pattern -instead of the default ++{beat_default_index_prefix}-*++ pattern. -endif::[] --- - -ifndef::no_ilm[] -. If you plan to use {ref}/getting-started-index-lifecycle-management.html[index -lifecycle management], create a role that has the following privileges. These -privileges are required to load index lifecycle policies and create and manage -rollover indices: -+ -* *Cluster:* `manage_ilm` -* *Index:* `write`, `create_index`, `manage`, and `manage_ilm` on the -{beatname_uc} indices -+ --- -["source","sh",subs="attributes"] ---------------------------------------------------------------- -POST _xpack/security/role/{beat_default_index_prefix}_ilm -{ - "cluster": ["manage_ilm"], - "indices": [ - { - "names": [ "{beat_default_index_prefix}-*","shrink-{beat_default_index_prefix}-*"], - "privileges": ["write","create_index","manage","manage_ilm"] - } - ] -} ---------------------------------------------------------------- -// CONSOLE --- -endif::no_ilm[] +You specify authentication credentials in the {beatname_uc} configuration +file: -. Assign the writer role to the user that {beatname_uc} will use to connect to -{es}. Make sure you also assign any roles that are required for specific -features. For the list of features and required roles, see <>. - -.. To authenticate as a native user, create a user for {beatname_uc} to use -internally and assign it the writer role, plus any other roles that are -needed. -+ -You can create users from the **Management / Users** UI in {kib} or through the -`user` API. For example, following request creates a user -named ++{beat_default_index_prefix}_internal++ that has the -++{beat_default_index_prefix}_writer++ and `kibana_user` roles: -+ --- -["source","sh",subs="attributes,callouts"] ---------------------------------------------------------------- -POST /_security/user/{beat_default_index_prefix}_internal -{ - "password" : "{pwd}", - "roles" : [ "{beat_default_index_prefix}_writer","kibana_user"], - "full_name" : "Internal {beatname_uc} User" -} ---------------------------------------------------------------- -// CONSOLE - --- - -.. To use PKI authentication, assign the writer role, plus any other roles that are -needed, in the `role_mapping.yml` configuration file. Specify the user by the -distinguished name that appears in its certificate: +* To use basic authentication, specify the `username` and `password` settings +under `output.elasticsearch`. For example: + -- ["source","yaml",subs="attributes,callouts"] ---------------------------------------------------------------- -{beat_default_index_prefix}_writer: - - "cn=Internal {beatname_uc} User,ou=example,o=com" -kibana_user: - - "cn=Internal {beatname_uc} User,ou=example,o=com" ---------------------------------------------------------------- - - -For more information, see -{xpack-ref}/mapping-roles.html#mapping-roles-file[Using Role Mapping Files]. --- - -. In the {beatname_uc} configuration file, specify authentication credentials -for the `elasticsearch` output: - - -.. To use basic authentication, configure the `username` and `password` settings. -For example, the following {beatname_uc} output configuration uses the native -++{beat_default_index_prefix}_internal++ user to connect to {es}: -+ -["source","js",subs="attributes,callouts"] --------------------------------------------------- +---- output.elasticsearch: hosts: ["localhost:9200"] username: "{beat_default_index_prefix}_internal" <1> password: "{pwd}" <2> --------------------------------------------------- -<1> You created this user earlier. +---- +<1> Let's assume this user has the privileges required to publish events to +{es}. <2> The example shows a hard-coded password, but you should store sensitive values in the <>. - -.. To use PKI authentication, configure the `certificate` and `key` settings: +-- +ifndef::apm-server[] ++ +If you've configured the {kib} endpoint, also specify credentials for +authenticating with {kib}. For example: + -["source","js",subs="attributes,callouts"] +["source","yaml",subs="attributes,callouts"] +---- +setup.kibana: + host: "mykibanahost:5601" + username: "{beat_default_index_prefix}_internal" <1> + password: "{pwd}" +---- +<1> Let's assume this user has the privileges required to set up dashboards. +endif::apm-server[] + +* To use Public Key Infrastructure (PKI) certificates to authenticate users, +configure the `certificate` and `key` settings. These settings assume that the +distinguished name (DN) in the certificate is mapped to the appropriate roles in +the `role_mapping.yml` file on each node in the {es} cluster. For more +information, see {xpack-ref}/mapping-roles.html#mapping-roles-file[Using Role +Mapping Files]. ++ +["source","yaml",subs="attributes,callouts"] -------------------------------------------------- output.elasticsearch: hosts: ["localhost:9200"] - ssl.certificate: "/etc/pki/client/cert.pem" <1> + ssl.certificate: "/etc/pki/client/cert.pem" ssl.key: "/etc/pki/client/cert.key" -------------------------------------------------- -<1> The distinguished name (DN) in the certificate must be mapped to -the ++{beat_default_index_prefix}_writer++ and `kibana_user` roles in the -`role_mapping.yml` configuration file on each node in the {es} cluster. +To learn more about {stack} security features and other types of +authentication, see {stack-ov}/elasticsearch-security.html[Securing the +{stack}]. diff --git a/docs/copied-from-beats/security/securing-beats.asciidoc b/docs/copied-from-beats/security/securing-beats.asciidoc index 4e1fa86996f..13b55dc1596 100644 --- a/docs/copied-from-beats/security/securing-beats.asciidoc +++ b/docs/copied-from-beats/security/securing-beats.asciidoc @@ -2,6 +2,7 @@ [[securing-beats]] == Configure {beatname_uc} to use {security} +[subs="attributes"] ++++ Use {security} ++++ @@ -10,25 +11,15 @@ If you want {beatname_uc} to connect to a cluster that has {stack-ov}/elasticsearch-security.html[{security}] enabled, there are extra configuration steps: -. <>. +. <>. + -ifeval::["{beatname_lc}"=="filebeat"] -To send data to a secured cluster through the `elasticsearch` output, -{beatname_uc} needs to authenticate as a user who can manage index templates, -monitor the cluster, create indices, read and write to the indices -it creates, and manage ingest pipelines. -endif::[] -ifeval::["{beatname_lc}"!="filebeat"] -To send data to a secured cluster through the `elasticsearch` output, -{beatname_uc} needs to authenticate as a user who can manage index templates, -monitor the cluster, create indices, and read and write to the indices -it creates. -endif::[] +You can use role-based access control to grant {beatname_uc} users access to +secured resources. -. <>. +. <>. + -To search the indexed {beatname_uc} data and visualize it in {kib}, users need -access to the indices {beatname_uc} creates. +To interact with a secured cluster, {beatname_uc} must either provide basic +authentication credentials or present a client certificate. . <>. + @@ -44,54 +35,10 @@ password, set it up now. For more information about {security}, see {xpack-ref}/elasticsearch-security.html[Securing the {stack}]. -[[feature-roles]] -=== {beatname_uc} features that require authorization - -After securing {beatname_uc}, make sure your users have the roles (or associated -privileges) required to use these {beatname_uc} features. Note that some of the -roles shown here are {xpack-ref}/built-in-roles.html[built-in], and some -are user-defined. - -[options="header"] -|======= -|Feature | Role -|Send data to a secured cluster | ++{beat_default_index_prefix}_writer++ footnoteref:[noteA,These roles are user-defined.] -ifeval::["{beatname_lc}"=="filebeat"] -|Run Filebeat modules | ++{beat_default_index_prefix}_writer++ footnoteref:[noteA] -endif::[] -|Load index templates | ++{beat_default_index_prefix}_writer++ footnoteref:[noteA] and `kibana_user` -ifndef::no_dashboards[] -|Load {beatname_uc} dashboards into {kib} | ++{beat_default_index_prefix}_writer++ footnoteref:[noteA] and `kibana_user` -endif::[] -ifdef::has_ml_jobs[] -|Load machine learning jobs | `machine_learning_admin` -endif::[] -ifndef::apm-server[] -|Read indices created by {beatname_uc} | ++{beat_default_index_prefix}_reader++ footnoteref:[noteA] -endif::[] -ifdef::apm-server[] -|Read indices created by {beatname_uc} | ++{beat_default_index_prefix}_user++ -|View {beatname_uc} dashboards in {kib} | `kibana_user` -endif::[] -ifndef::no_dashboards[] -|View {beatname_uc} dashboards in {kib} | `kibana_user` -endif::[] -ifdef::has_central_config[] -|Store and manage configurations in a central location in {kib} | `beats_admin` -endif::[] -ifndef::no_ilm[] -|Load index lifecycle policies and use index lifecycle management | +{beatname_lc}_ilm+ footnoteref:[noteA] -endif::[] -|======= - -To create the user-defined roles shown here, see <> and -<>. You may want to define additional roles to provide more -restrictive access. +include::users.asciidoc[] include::basic-auth.asciidoc[] -include::user-access.asciidoc[] - include::tls.asciidoc[] include::beats-system.asciidoc[] diff --git a/docs/copied-from-beats/security/user-access.asciidoc b/docs/copied-from-beats/security/user-access.asciidoc deleted file mode 100644 index fe77d4596fa..00000000000 --- a/docs/copied-from-beats/security/user-access.asciidoc +++ /dev/null @@ -1,80 +0,0 @@ -[role="xpack"] -[[beats-user-access]] -=== Grant users access to {beatname_uc} indices - -To enable users to access the indices {beatname_uc} creates, grant them `read` -and `view_index_metadata` privileges on the {beatname_uc} indices. If they're -using {kib}, they also need the `kibana_user` role. - -ifdef::apm-server[] -X-Pack security provides a built-in role called `apm_user` that you can explicitly assign to users. -This role grants them the necessary `read` and `view_index_metadata` privileges on the {beatname_uc} indices. -endif::apm-server[] - -ifndef::apm-server[] -. Create a role that has the `read` and `view_index_metadata` privileges -on the {beatname_uc} indices. -+ -You can create roles from the **Management > Roles** UI in {kib} or through the -`role` API. For example, the following request creates a role named -++{access_role}++: -+ --- -["source","sh",subs="attributes,callouts"] ---------------------------------------------------------------- -POST _security/role/{access_role} -{ - "indices": [ - { - "names": [ "{beat_default_index_prefix}-*" ], <1> - "privileges": ["read","view_index_metadata"] - } - ] -} ---------------------------------------------------------------- -// CONSOLE -<1> If you use a custom {beatname_uc} index pattern, specify that pattern -instead of the default ++{beat_default_index_prefix}-*++ pattern. --- -endif::apm-server[] - -. Assign your users the ++{access_role}++ -role so they can access the {beatname_uc} indices. -For {kib} users who need to visualize the data, -also assign the `kibana_user` role: - -.. If you're using the `native` realm, you can assign roles with the -**Management > Users** UI in {kib} or through the `user` API. For example, the -following request grants ++{beat_default_index_prefix}_account++ the -++{access_role}++ and `kibana_user` roles: -+ --- -["source", "sh", subs="attributes,callouts"] ---------------------------------------------------------------- -POST /_security/user/{beat_default_index_prefix}_account -{ - "password" : "{pwd}", - "roles" : [ "{access_role}","kibana_user"], - "full_name" : "{beatname_uc} account" -} ---------------------------------------------------------------- -// CONSOLE --- -.. If you're using the LDAP, Active Directory, or PKI realms, -you assign the roles in the `role_mapping.yml` configuration file. -For example, the following snippet grants -++{beat_default_index_prefix}_account++ the -++{access_role}++ and `kibana_user` roles: -+ --- -["source", "yaml", subs="attributes,callouts"] ---------------------------------------------------------------- -{access_role}: - - "cn={beat_default_index_prefix}_account,dc=example,dc=com" -kibana_user: - - "cn={beat_default_index_prefix}_account,dc=example,dc=com" ---------------------------------------------------------------- - -For more information, see -{xpack-ref}/mapping-roles.html#mapping-roles-file[Using Role Mapping Files]. --- diff --git a/docs/copied-from-beats/security/users.asciidoc b/docs/copied-from-beats/security/users.asciidoc new file mode 100644 index 00000000000..3d05773c0ba --- /dev/null +++ b/docs/copied-from-beats/security/users.asciidoc @@ -0,0 +1,209 @@ +[role="xpack"] +[[feature-roles]] +=== Grant users access to secured resources + +You can use role-based access control to grant users access to secured +resources. The roles that you set up depend on your organization's security +requirements and the minimum privileges required to use specific features. + +{beatname_uc} users typically perform these main roles: they do the initial +setup, publish monitoring information, and publish events. If they're using +{kib}, they view and sometimes create visualizations that access {beatname_uc} +indices. + +{security} provides pre-built roles that grant _some_ of the privileges needed +by {beatname_uc} users. When possible, use the built-in roles to minimize the +affect of future changes on your security strategy. + +For privileges not granted by existing roles, create new roles. At a minimum, +create a role for setting up {beatname_uc}, a role for publishing events, and a +role for reading {beatname_uc} indices. Assign these new roles, along with the +pre-built roles, to grant the full set of privileges required by {beatname_uc} +users. + +The following sections describe the privileges and roles required to perform +specific job roles. + +[[privileges-to-setup-beats]] +==== Privileges needed for initial setup + +Users who set up {beatname_uc} typically need to load mappings, dashboards, and +other objects used to index data into {es} and visualize it in {kib}. The +privileges required depend on the setup tasks users need to perform. + +NOTE: These instructions assume that you are using the default name for +{beatname_uc} indices. If you are using a custom name, modify the privileges to +match your index naming pattern. + +[options="header"] +|==== +|Task | Required privileges and roles + +.3+|Set up index templates +|`manage_index_templates` and `monitor` on cluster +|`manage_ilm` on cluster (if cluster supports index lifecycle management) +|`manage` on +{beat_default_index_prefix}-*+ indices (if cluster supports index lifecycle management) + +ifndef::no_dashboards[] +|Set up example dashboards +|`kibana_user` role +endif::no_dashboards[] + +ifdef::has_ml_jobs[] +.3+|Set up machine learning job configurations +|`manage_ml` and `monitor` on cluster +|`read` on +{beat_default_index_prefix}-*+ indices +|`kibana_user` role +endif::has_ml_jobs[] + +ifeval::["{beatname_lc}"=="filebeat"] +.2+|Set up ingest pipelines +|`monitor` on cluster +|`ingest_admin` role +endif::[] + +ifdef::apm-server[] +.2+|Set up ingest pipelines +|`monitor` on cluster +|`ingest_admin` role +endif::apm-server[] + +.2+|Set up index lifecycle policies +|`manage_ilm`, `manage_index_templates`, and `monitor` on cluster +|`manage` on +{beat_default_index_prefix}-*+ indices + +ifdef::has_central_config[] +|Enroll and manage configurations in Beats central management +|`beats_admin` and `kibana_user` roles +endif::has_central_config[] +|==== + +[[privileges-to-publish-monitoring]] +==== Privileges needed to publish and view monitoring information + +{security} provides the +{beat_default_index_prefix}_system+ +{stack-ov}/built-in-users.html[built-in user] and ++{beat_default_index_prefix}_system+ {stack-ov}/built-in-roles.html[built-in +role] for sending monitoring information. You can use the built-in user, or +create a user who has the privileges needed to send monitoring information. +If you use the +{beat_default_index_prefix}_system+ user, make sure you +<>. + +[options="header"] +|==== +|Task | Required privileges and roles + +|Send monitoring info +|`monitor` on cluster + +|Use *Stack Monitoring* in {kib} to monitor {beatname_uc} +|`monitoring_user` and `kibana_user` roles +|==== + + +[[privileges-to-publish-events]] +==== Privileges needed to publish events + +Users who publish events to {es} need to create and read from {beatname_uc} +indices. The privileges required for this role depend on the tasks users +need to perform: + +[options="header"] +|==== +|Task | Required privileges and roles + +ifndef::apm-server[] +.3+|Send data to a secured cluster without index lifecycle management +|`monitor` on cluster +ifeval::["{beatname_lc}"=="filebeat"] +(and `manage_pipeline` if {beatname_uc} modules are used) +endif::[] +|`create_index` and `index` on +{beat_default_index_prefix}-*+ indices +|also requires privileges to <> +unless you've disabled automatic template loading + +.2+|Send data to a secured cluster that supports index lifecycle management +|`manage_index_templates`, `manage_ilm` +footnote:[Use `read_ilm` instead of `manage_ilm` if you pre-loaded the lifecycle policy] +, and `monitor` on cluster +ifeval::["{beatname_lc}"=="filebeat"] +(and `manage_pipeline` if {beatname_uc} modules are used) +endif::[] +| `index` and `manage` on +{beat_default_index_prefix}-*+ indices +endif::apm-server[] + +ifdef::apm-server[] +.3+|Send data to a secured cluster without index lifecycle management +|`monitor` on cluster +|`create_index` and `write` on +{beat_default_index_prefix}-*+ indices +|also requires privileges to <> +unless you've disabled automatic template loading: `setup.template.enabled=false` + +.3+|Send data to a secured cluster that supports index lifecycle management +|`manage_ilm` and `monitor` on cluster +| `index` and `manage` on +{beat_default_index_prefix}-*+ indices +|also requires privileges to <> +unless you've disabled automatic template loading: `setup.template.enabled=false` +endif::apm-server[] + +ifdef::has_central_config[] +.2+|Read configurations from Beats central management +| `monitor` on cluster +|`beats_admin` role +endif::has_central_config[] +|==== + +// REVIEWERS: Do users need `index` and `manage` on `shrink-beatname-*`, too? +// Are there other privileges that might be required as indices move through the +// lifecycle stages? + +[[kibana-user-privileges]] +==== Privileges needed by {kib} users + +{kib} users typically need to view dashboards and visualizations that contain +{beatname_uc} data. These users might also need to create and edit dashboards +and visualizations. +ifdef::has_central_config[] +If you're using Beats central management, they need to create and manage +configurations. +endif::has_central_config[] + +The privileges required for {kib} users depend on the tasks they need to +perform: + +[options="header"] +|==== +|Task | Required privileges and roles + +ifndef::no_dashboards[] +.2+|View {beatname_uc} dashboards +|`read` on +{beat_default_index_prefix}-*+ indices +|`kibana_dashboard_only_user` role + +.2+|View and edit {beatname_uc} dashboards +|`read` on +{beat_default_index_prefix}-*+ indices +|`kibana_user` role +endif::no_dashboards[] + +ifdef::apm-server[] +|Use the APM UI +|`kibana_user` and `apm_user` roles +endif::apm-server[] + +ifdef::has_central_config[] +.2+|Create and manage configurations in Beats central management +|`beats_admin` role +|`kibana_user` role +endif::[] +|==== + +[[learn-more-security]] +==== Learn more about users and roles + +Want to learn more about creating users and roles? See +{stack-ov}/elasticsearch-security.html[Securing the {stack}]. Also see: + +* {stack-ov}/security-privileges.html[Security privileges] for a description of +available privileges +* {stack-ov}/built-in-roles.html[Built-in roles] for a description of roles that +you can assign to users \ No newline at end of file diff --git a/docs/copied-from-beats/shared-docker.asciidoc b/docs/copied-from-beats/shared-docker.asciidoc index 6da7376d0e4..24520f212ea 100644 --- a/docs/copied-from-beats/shared-docker.asciidoc +++ b/docs/copied-from-beats/shared-docker.asciidoc @@ -50,7 +50,37 @@ ifndef::no_dashboards[] endif::no_dashboards[] and machine learning jobs. Run this command: -ifeval::[("{beatname_lc}"=="filebeat") or ("{beatname_lc}"=="metricbeat") or ("{beatname_lc}"=="heartbeat") or ("{beatname_lc}"=="journalbeat")] +ifeval::["{beatname_lc}"=="filebeat"] +["source", "sh", subs="attributes"] +-------------------------------------------- +docker run \ +{dockerimage} \ +setup -E setup.kibana.host=kibana:5601 \ +-E output.elasticsearch.hosts=["elasticsearch:9200"] <1> <2> +-------------------------------------------- +endif::[] + +ifeval::["{beatname_lc}"=="metricbeat"] +["source", "sh", subs="attributes"] +-------------------------------------------- +docker run \ +{dockerimage} \ +setup -E setup.kibana.host=kibana:5601 \ +-E output.elasticsearch.hosts=["elasticsearch:9200"] <1> <2> +-------------------------------------------- +endif::[] + +ifeval::["{beatname_lc}"=="heartbeat"] +["source", "sh", subs="attributes"] +-------------------------------------------- +docker run \ +{dockerimage} \ +setup -E setup.kibana.host=kibana:5601 \ +-E output.elasticsearch.hosts=["elasticsearch:9200"] <1> <2> +-------------------------------------------- +endif::[] + +ifeval::["{beatname_lc}"=="journalbeat"] ["source", "sh", subs="attributes"] -------------------------------------------- docker run \ @@ -115,9 +145,9 @@ curl -L -O {dockerconfig} ===== Volume-mounted configuration One way to configure {beatname_uc} on Docker is to provide +{beatname_lc}.docker.yml+ via a volume mount. -With +docker run+, the volume mount can be specified like this: +With +docker run+, the volume mount can be specified like this. -ifeval::[("{beatname_lc}"=="filebeat") or ("{beatname_lc}"=="journalbeat")] +ifeval::["{beatname_lc}"=="filebeat"] ["source", "sh", subs="attributes"] -------------------------------------------- docker run -d \ @@ -131,6 +161,24 @@ docker run -d \ -------------------------------------------- endif::[] +ifeval::["{beatname_lc}"=="journalbeat"] +Make sure you include the path to the host's journal. The path might be +`/var/log/journal` or `/run/log/journal`. + +["source", "sh", subs="attributes"] +-------------------------------------------- +sudo docker run -d \ + --name={beatname_lc} \ + --user=root \ + --volume="/var/log/journal:/var/log/journal" \ + --volume="/etc/machine-id:/etc/machine-id" \ + --volume="/run/systemd:/run/systemd" \ + --volume="/etc/hostname:/etc/hostname:ro" \ + {dockerimage} {beatname_lc} -e -strict.perms=false \ + -E output.elasticsearch.hosts=["elasticsearch:9200"] <1> <2> +-------------------------------------------- +endif::[] + ifeval::["{beatname_lc}"=="metricbeat"] ["source", "sh", subs="attributes"] -------------------------------------------- @@ -179,7 +227,20 @@ docker run -d \ -------------------------------------------- endif::[] -ifeval::[("{beatname_lc}"=="heartbeat") or ("{beatname_lc}"=="apm-server")] +ifeval::["{beatname_lc}"=="heartbeat"] +["source", "sh", subs="attributes"] +-------------------------------------------- +docker run -d \ + --name={beatname_lc} \ + --user={beatname_lc} \ + --volume="$(pwd)/{beatname_lc}.docker.yml:/usr/share/{beatname_lc}/{beatname_lc}.yml:ro" \ + {dockerimage} \ + --strict.perms=false -e \ + -E output.elasticsearch.hosts=["elasticsearch:9200"] <1> <2> +-------------------------------------------- +endif::[] + +ifeval::["{beatname_lc}"=="apm-server"] ["source", "sh", subs="attributes"] -------------------------------------------- docker run -d \ @@ -199,8 +260,7 @@ using the syntax shown earlier. ===== Customize your configuration -ifeval::[("{beatname_lc}"=="filebeat") or ("{beatname_lc}"=="metricbeat")] - +ifdef::has_docker_label_ex[] The +{beatname_lc}.docker.yml+ file you downloaded earlier is configured to deploy Beats modules based on the Docker labels applied to your containers. See <> for more details. Add labels to your application Docker containers, and they will be picked up by the Beats autodiscover feature when they are deployed. Here is an example command for an Apache HTTP Server container with labels to configure the Filebeat and Metricbeat modules for the Apache HTTP Server: ["source", "sh", subs="attributes"] @@ -217,11 +277,9 @@ docker run \ -p 8080:80 \ httpd:2.4 -------------------------------------------- - endif::[] -ifeval::[("{beatname_lc}"!="filebeat") and ("{beatname_lc}"!="metricbeat")] - +ifndef::has_docker_label_ex[] The +{beatname_lc}.docker.yml+ downloaded earlier should be customized for your environment. See <> for more details. Edit the configuration file and customize it to match your environment then re-deploy your {beatname_uc} container. endif::[] diff --git a/docs/copied-from-beats/shared-path-config.asciidoc b/docs/copied-from-beats/shared-path-config.asciidoc index 117c5aad34b..981a64f23df 100644 --- a/docs/copied-from-beats/shared-path-config.asciidoc +++ b/docs/copied-from-beats/shared-path-config.asciidoc @@ -17,7 +17,7 @@ The `path` section of the +{beatname_lc}.yml+ config file contains configuration options that define where {beatname_uc} looks for its files. For example, {beatname_uc} looks for the Elasticsearch template file in the configuration path and writes log files in the logs path. -ifeval::["{beatname_lc}"=="filebeat" or "{beatname_lc}"=="winlogbeat"] +ifdef::has_registry[] {beatname_uc} looks for its registry files in the data path. endif::[] diff --git a/docs/guide/apm-release-notes.asciidoc b/docs/guide/apm-release-notes.asciidoc index 3c9bb13eb7a..73be25c1302 100644 --- a/docs/guide/apm-release-notes.asciidoc +++ b/docs/guide/apm-release-notes.asciidoc @@ -7,6 +7,7 @@ For a full list of changes, see the {apm-server-ref-v}/release-notes.html[APM Server Release Notes] or the {kibana-ref}/release-notes.html[Kibana Release Notes]. +* <> * <> * <> * <> @@ -16,6 +17,16 @@ For a full list of changes, see the * <> * <> +[[release-highlights-7.2.0]] +=== APM version 7.2.0 + +[float] +==== New features + +*APM Server* + +* Support for {apm-server-ref-v}/ilm.html[index lifecycle management (ILM)] + [[release-highlights-7.1.0]] === APM version 7.1.0 diff --git a/docs/guide/overview.asciidoc b/docs/guide/overview.asciidoc index 27c5ba730a9..a83abf0706e 100644 --- a/docs/guide/overview.asciidoc +++ b/docs/guide/overview.asciidoc @@ -35,7 +35,7 @@ Elasticsearch is used to store APM performance metrics and make use of its aggre {kibana-ref}/xpack-apm.html[*Kibana*] is an open source analytics and visualization platform designed to work with Elasticsearch. You use Kibana to search, view, and interact with data stored in Elasticsearch. -You an use Kibana to visualize APM data by utilizing the dedicated APM UI bundled in Basic license, +You can use Kibana to visualize APM data by utilizing the dedicated APM UI bundled in Basic license, or the pre-built, open source, Kibana dashboards that can be loaded directly via the {kibana-ref}/apm-getting-started.html[APM Kibana UI]. @@ -83,4 +83,4 @@ Together, transactions and spans form a `Trace`. Traces are not events, but group together events that have a common root. All of our APM agents support <> out of the box. -Distributed tracing enables you to analyze performance throughout your microservices architecture all in one view. \ No newline at end of file +Distributed tracing enables you to analyze performance throughout your microservices architecture all in one view. diff --git a/docs/ilm-setup.asciidoc b/docs/ilm-setup.asciidoc index 231dc3ab7ee..6c80e267f4b 100644 --- a/docs/ilm-setup.asciidoc +++ b/docs/ilm-setup.asciidoc @@ -1,13 +1,9 @@ [role="xpack"] [[manual-ilm-setup]] -== Manual index lifecycle management +=== Manual index lifecycle management -APM Server can take advantage of the index lifecycle management (ILM) capabilities of Elasticsearch. -ILM enables you to automate how you want to manage your indices over time. -You can base actions on factors such as shard size and performance requirements. - -The guide below will help you set up a custom ILM policy for span indices. -You can repeat the actions for any other indices. +The guide below will help you set up a custom index lifecycle management policy for span indices. +You can repeat these actions for any other indices (transactions, errors, etc.). NOTE: If you're migrating from an existing setup, any indices present before ILM was configured will need to be managed manually. diff --git a/docs/ilm.asciidoc b/docs/ilm.asciidoc new file mode 100644 index 00000000000..bd51490aaf6 --- /dev/null +++ b/docs/ilm.asciidoc @@ -0,0 +1,131 @@ +[[ilm]] +[role="xpack"] +== Index lifecycle management (ILM) + +Use the {ref}/getting-started-index-lifecycle-management.html[index lifecycle management (ILM)] +feature in {es} to manage your APM Server indices as they age. +ILM enables you to automate how you want to manage your indices over time, +by automating rollovers to a new index when the existing index reaches a specified size or age. + +You can view and edit the index lifecycle policies in the *Index lifecycle policies* UI in {kib}. +For more information about working with the UI, +see {kibana-ref}/index-lifecycle-policies.html[Index lifecycle policies]. + +APM Server currently offers two ways to get started with ILM: + +1. <> - Get up and running with a default index lifecycle management policy as quickly as possible. +2. <> - Customize and manage your own ILM policies. + +NOTE: If you're migrating from an existing setup, +any indices present before ILM was configured will need to be managed manually. + +[[ilm-default]] +=== ILM default policy + +Index lifecycle management will manage an index based on its defined policy. +The default ILM configuration applies hot and warm phase policies. +Cold and delete phases are not defined. +Because errors and spans lose information value faster than metrics and transactions do, +there are two different policies defined: +one for `errors` and `spans`, and one for `metrics` and `transactions`. + +[options="header"] +|======================================================================= +|Event type |Hot |Warm +|Errors & Spans +|Rollover: `max_size: 50gb`, `max_age: 1 day`, +`priority: 100` +|`min_age: 7 days`, +readonly, +`priority: 50` +|Transactions & Metrics +|Rollover: `max_size: 50gb`, `max_age: 7 days`, +`priority: 100` +|`min_age: 31 days`, +readonly, +`priority: 50` +|======================================================================= + +IMPORTANT: Changes to the default index lifecycle policy do not take effect until the current index has rolled over. +If you'd like to manage a custom policy, see <>. + +[float] +[[ilm-setup]] +==== ILM default policy setup + +To set up index lifecycle management, set `ilm.enabled` to `true` in apm-server.yml. +ILM can only be enabled for `output.elasticsearch`. +When enabled, configurations defined for `output.elasticsearch.index` and `output.elasticsearch.indices` will be ignored. + +It is recommended to set up index lifecycle management (ILM) before starting APM Server. +This excludes setup from the ingestion process, which allows you to ensure ILM is set up correctly before using APM. + +Run the <> with the ` --index-management` flag to set up the default ILM policy: + +[source,js] +----------------------- +./apm-server setup --index-management +----------------------- +// CONSOLE + +You can confirm the policy was created with the GET lifecycle policy API. +Here's what the transaction response looks like: + +[source,js] +----------------------- +GET _ilm/policy +{ + "apm-7.2.0-transaction": { + "version": 1, + "modified_date": "2019-05-28T15:55:26.791Z", + "policy": { + "phases": { + "warm": { + "min_age": "31d",<1> + "actions": { + "readonly": {}, + "set_priority": { + "priority": 50 + } + } + }, + "hot": { + "min_age": "0ms", + "actions": { + "rollover": { + "max_size": "50gb",<2> + "max_age": "7d"<3> + }, + "set_priority": { + "priority": 100<4> + } + } + } + } + } + } +} +----------------------- +// CONSOLE +<1> Move to warm phase after _31 days_ +<2> Rollover after _50gb_ +<3> Rollover after _7 days_ +<4> Priority for recovering your indices after a node restart. Higher priorities are recovered first. + +Your indices are now configured to use index lifecycle management. Go ahead and <>. + +NOTE: If you switch between ILM enabled/disabled multiple times, +you should set `setup.template.overwrite` to `true` to ensure a complete setup. + +[float] +==== ILM default policy upgrades + +If you decide to customize the default ILM policy, any customizations will be overwritten when you upgrade APM Server. +Default policies are also subject to change in future releases + +// Policies are versioned so they can change. +// Indices are versioned so they can change. +// An upgrade creates new templates, policies, and indices. +// If you customize anything, it will revert back to the default during an upgrade + +include::./ilm-setup.asciidoc[] \ No newline at end of file diff --git a/docs/redirects.asciidoc b/docs/redirects.asciidoc index 237e5ce5d86..510de79a272 100644 --- a/docs/redirects.asciidoc +++ b/docs/redirects.asciidoc @@ -256,4 +256,7 @@ Loading dashboards from APM Server is no longer supported. Please see the {kiban [role="exclude",id="load-kibana-dashboards"] === Dashboards -Loading dashboards from APM Server is no longer supported. Please see the {kibana-ref}/xpack-apm.html[Kibana APM UI] documentation. \ No newline at end of file +Loading Kibana dashboards from APM Server is no longer supported. +Please use the {kibana-ref}/xpack-apm.html[Kibana APM UI] instead. +As an alternative, a small number of dashboards and visualizations are available in the +https://github.com/elastic/apm-contrib/tree/master/apm-ui[apm-contrib] repository.