More generic docs for use in APM Server (elastic#6184)

* Use specific beat name's instead of 'the Beat'
* Dont show Logstash info unless it's supported.
* Make the index pattern decoupled from the beat name
* Make it possible to skip the pipeline docs in output.elasticsearch
* Introduce beat_default_index_prefix.
* Use name of beat in shared-kibana-config.asciidoc
* Add .\ for PS instruction and make it possible to remove logstash mention.
* Introduce 'has_ml_jobs'
* Added `html_docs` to .gitignore
* Better outputconfig for Elasticsearch only beats.
* Less blamy wording
* Only talk about Filebeat for filebeat docs
* Update the rest of the max_retries sections to only talk about Filebeat for filebeat
* Special case for apm-server as it was only introduced in 6.0
* expand beat_default_index_prefix use (#2)

.gitignore

@@ -7,6 +7,7 @@
 /*/logs
 /*/fields.yml
 /*/*.template*.json
+**/html_docs
 
 # Files
 .DS_Store

auditbeat/docs/index.asciidoc

@@ -10,6 +10,8 @@ include::{asciidoc-dir}/../../shared/attributes.asciidoc[]
 :beatname_pkg: {beatname_lc}
 :github_repo_name: beats
 :discuss_forum: beats/{beatname_lc}
+:beat_default_index_prefix: {beatname_lc}
+:has_ml_jobs: yes
 
 include::../../libbeat/docs/shared-beats-attributes.asciidoc[]
 

filebeat/docs/index.asciidoc

@@ -10,6 +10,8 @@ include::{asciidoc-dir}/../../shared/attributes.asciidoc[]
 :beatname_pkg: {beatname_lc}
 :github_repo_name: beats
 :discuss_forum: beats/{beatname_lc}
+:beat_default_index_prefix: {beatname_lc}
+:has_ml_jobs: yes
 
 include::../../libbeat/docs/shared-beats-attributes.asciidoc[]
 

heartbeat/docs/index.asciidoc

@@ -10,6 +10,8 @@ include::{asciidoc-dir}/../../shared/attributes.asciidoc[]
 :beatname_pkg: heartbeat-elastic
 :github_repo_name: beats
 :discuss_forum: beats/{beatname_lc}
+:beat_default_index_prefix: {beatname_lc}
+:has_ml_jobs: yes
 
 include::../../libbeat/docs/shared-beats-attributes.asciidoc[]
 

libbeat/docs/command-reference.asciidoc

@@ -19,7 +19,15 @@
 :keystore-command-short-desc: Manages the <<keystore,secrets keystore>>
 :modules-command-short-desc: Manages configured modules
 :run-command-short-desc: Runs {beatname_uc}. This command is used by default if you start {beatname_uc} without specifying a command
+
+ifeval::["{has_ml_jobs}"=="yes"]
 :setup-command-short-desc: Sets up the initial environment, including the index template, Kibana dashboards (when available), and machine learning jobs (when available)
+endif::[]
+
+ifeval::["{has_ml_jobs}"!="yes"]
+:setup-command-short-desc: Sets up the initial environment, including the index template, Kibana dashboards (when available)
+endif::[]
+
 :test-command-short-desc: Tests the configuration
 :version-command-short-desc: Shows information about the current version
 
@@ -31,7 +39,7 @@
 <titleabbrev>Command reference</titleabbrev>
 ++++
 
-{beatname_uc} provides a command-line interface for running the Beat and
+{beatname_uc} provides a command-line interface for starting {beatname_uc} and
 performing common tasks, like testing configuration files and loading
 dashboards. The command-line also supports <<global-flags,global flags>>
 for controlling global behaviors.

libbeat/docs/dashboards.asciidoc

@@ -9,20 +9,16 @@
 //// include::../../libbeat/docs/dashboards.asciidoc[]
 //////////////////////////////////////////////////////////////////////////
 
-
 {beatname_uc} comes packaged with example Kibana dashboards, visualizations,
 and searches for visualizing {beatname_uc} data in Kibana. Before you can use
-the dashboards, you need to create the index  pattern, +{beatname_lc}-*+, and
+the dashboards, you need to create the index pattern, +{beat_default_index_prefix}-*+, and
 load the dashboards into Kibana. To do this, you can either run the `setup`
 command (as described here) or
 <<configuration-dashboards,configure dashboard loading>> in the
 +{beatname_lc}.yml+ config file.
 
-NOTE: Starting with Beats 6.0.0, the dashboards are loaded via the Kibana API.
-This requires a Kibana endpoint configuration. You should have configured the
-endpoint earlier when you
-<<{beatname_lc}-configuration,configured {beatname_uc}>>. If you didn't,
-configure it now.
+This requires a Kibana endpoint configuration. If you didn't already configure
+a Kibana endpoint, see <<{beatname_lc}-configuration,configured {beatname_uc}>> 
 
 Make sure Kibana is running before you perform this step. If you are accessing a
 secured Kibana instance, make sure you've configured credentials as described in

libbeat/docs/dashboardsconfig.asciidoc

@@ -20,7 +20,7 @@ To load the dashboards, you can either enable dashboard loading in the
 run the `setup` command. Dashboard loading is disabled by default.
 
 When dashboard loading is enabled, {beatname_uc} uses the Kibana API to load the
-sample dashboards. Dashboard loading is only attempted at Beat startup.
+sample dashboards. Dashboard loading is only attempted when {beatname_uc} starts up.
 If Kibana is not available at startup, {beatname_uc} will stop with an error.
 
 To enable dashboard loading, add the following setting to the config file:

libbeat/docs/index.asciidoc

@@ -10,7 +10,8 @@ include::{asciidoc-dir}/../../shared/attributes.asciidoc[]
 :beatname_pkg: {beatname_lc}
 :github_repo_name: beats
 :discuss_forum: beats/{beatname_lc}
-
+:beat_default_index_prefix: {beatname_lc}
+:has_ml_jobs: yes
 
 include::../../libbeat/docs/shared-beats-attributes.asciidoc[]
 

libbeat/docs/loggingconfig.asciidoc

@@ -14,7 +14,7 @@
 == Configure logging
 
 The `logging` section of the +{beatname_lc}.yml+ config file contains options
-for configuring the Beats logging output. The logging system can write logs to
+for configuring the logging output. The logging system can write logs to
 the syslog or rotate log files. If logging is not explicitly configured the file
 output is used.
 
@@ -67,7 +67,7 @@ Minimum log level. One of `debug`, `info`, `warning`, or `error`. The default
 log level is `info`.
 
 `debug`:: Logs debug messages, including a detailed printout of all events
-flushed by the Beat. Also logs informational messages, warnings, errors, and
+flushed. Also logs informational messages, warnings, errors, and
 critical errors. When the log level is `debug`, you can specify a list of
 <<selectors,`selectors`>> to display debug messages for specific components. If
 no selectors are specified, the `*` selector is used to display debug messages
@@ -84,9 +84,9 @@ published. Also logs any warnings, errors, or critical errors.
 [[selectors]]
 ==== `logging.selectors`
 
-The list of debugging-only selector tags used by different Beats components. Use `*`
-to enable debug output for all components. For example add `publish` to display
-all the debug messages related to event publishing. When starting the Beat,
+The list of debugging-only selector tags used by different {beatname_uc} components.
+Use `*` to enable debug output for all components. For example add `publish` to display
+all the debug messages related to event publishing. When starting {beatname_lc},
 selectors can be overwritten using the `-d` command line option (`-d` also sets
 the debug log level).
 
@@ -123,8 +123,7 @@ the <<directory-layout>> section for details.
 [float]
 ==== `logging.files.name`
 
-The name of the file that logs are written to. By default, the name of the Beat
-is used.
+The name of the file that logs are written to. The default is '{beatname_lc}'.
 
 [float]
 ==== `logging.files.rotateeverybytes`

libbeat/docs/outputconfig.asciidoc

@@ -13,6 +13,12 @@
 [[configuring-output]]
 == Configure the output
 
+ifdef::only-elasticsearch[]
+You configure {beatname_uc} to write to Elasticsearch by setting options 
+in the `output.elasticsearch` section of the +{beatname_lc}.yml+ config file
+endif::[]
+
+ifndef::only-elasticsearch[]
 You configure {beatname_uc} to write to a specific output by setting options
 in the `output` section of the +{beatname_lc}.yml+ config file. Only a single
 output may be defined.
@@ -26,14 +32,16 @@ The following topics describe how to configure each supported output:
 * <<file-output>>
 * <<console-output>>
 
+endif::[]
+
 [[elasticsearch-output]]
 === Configure the Elasticsearch output
 
 ++++
 <titleabbrev>Elasticsearch</titleabbrev>
 ++++
 
-When you specify Elasticsearch for the output, the Beat sends the transactions directly to Elasticsearch by using the Elasticsearch HTTP API.
+When you specify Elasticsearch for the output, {beatname_uc} sends the transactions directly to Elasticsearch by using the Elasticsearch HTTP API.
 
 Example configuration:
 
@@ -222,6 +230,7 @@ output.elasticsearch:
         message: "ERR"
 ------------------------------------------------------------------------------
 
+ifndef::no-pipeline[]
 ===== `pipeline`
 
 A format string value that specifies the ingest node pipeline to write events to.
@@ -266,25 +275,29 @@ output.elasticsearch:
       when.equals:
         fields.type: "normal"
 ------------------------------------------------------------------------------
+endif::[]
 
 ===== `max_retries`
 
 The number of times to retry publishing an event after a publishing failure.
 After the specified number of retries, the events are typically dropped.
-Some Beats, such as Filebeat, ignore the `max_retries` setting and retry until all
-events are published.
 
+ifeval::["{beatname_lc}" == "filebeat"]
+Filebeat will ignore the `max_retries` setting and retry until all
+events are published.
+endif::[]
+ifeval::["{beatname_lc}" != "filebeat"]
 Set `max_retries` to a value less than 0 to retry until all events are published.
+endif::[]
 
 The default is 3.
 
 ===== `bulk_max_size`
 
 The maximum number of events to bulk in a single Elasticsearch bulk API index request. The default is 50.
 
-If the Beat sends single events, the events are collected into batches. If the Beat publishes
-a large batch of events (larger than the value specified by `bulk_max_size`), the batch is
-split.
+Events can be collected into batches. {beatname_uc} will split batches larger than `bulk_max_size`
+into multiple batches.
 
 Specifying a larger batch size can improve performance by lowering the overhead of sending events.
 However big batch sizes can also increase processing times, which might result in
@@ -307,6 +320,7 @@ Elasticsearch.
 
 See <<configuration-ssl>> for more information.
 
+ifndef::only-elasticsearch[]
 
 [[logstash-output]]
 === Configure the Logstash output
@@ -515,10 +529,14 @@ The number of seconds to wait for responses from the Logstash server before timi
 
 The number of times to retry publishing an event after a publishing failure.
 After the specified number of retries, the events are typically dropped.
-Some Beats, such as Filebeat, ignore the `max_retries` setting and retry until all
-events are published.
 
+ifeval::["{beatname_lc}" == "filebeat"]
+Filebeat will ignore the `max_retries` setting and retry until all
+events are published.
+endif::[]
+ifeval::["{beatname_lc}" != "filebeat"]
 Set `max_retries` to a value less than 0 to retry until all events are published.
+endif::[]
 
 The default is 3.
 
@@ -706,10 +724,15 @@ brokers, topics, partition, and active leaders to use for publishing.
 
 The number of times to retry publishing an event after a publishing failure.
 After the specified number of retries, the events are typically dropped.
-Some Beats, such as Filebeat, ignore the `max_retries` setting and retry until all
+
+ifeval::["{beatname_lc}" == "filebeat"]
+Filebeat will ignore the `max_retries` setting and retry until all
 events are published.
+endif::[]
 
+ifeval::["{beatname_lc}" != "filebeat"]
 Set `max_retries` to a value less than 0 to retry until all events are published.
+endif::[]
 
 The default is 3.
 
@@ -928,13 +951,18 @@ The Redis connection timeout in seconds. The default is 5 seconds.
 
 The number of times to retry publishing an event after a publishing failure.
 After the specified number of retries, the events are typically dropped.
-Some Beats, such as Filebeat, ignore the `max_retries` setting and retry until all
-events are published.
 
+ifeval::["{beatname_lc}" == "filebeat"]
+Filebeat will ignore the `max_retries` setting and retry until all
+events are published.
+endif::[]
+ifeval::["{beatname_lc}" != "filebeat"]
 Set `max_retries` to a value less than 0 to retry until all events are published.
+endif::[]
 
 The default is 3.
 
+
 ===== `bulk_max_size`
 
 The maximum number of events to bulk in a single Redis request or pipeline. The default is 2048.
@@ -1162,3 +1190,5 @@ When specified, the `cloud.auth` overwrites the `output.elasticsearch.username`
 `output.elasticsearch.password` settings. Because the Kibana settings inherit
 the username and password from the Elasticsearch output, this can also be used
 to set the `setup.kibana.username` and `setup.kibana.password` options.
+
+endif::[]

libbeat/docs/processors-using.asciidoc

@@ -543,7 +543,7 @@ default. For example, FileBeat enables the `container` indexer, which indexes
 pod metadata based on all container IDs, and a `logs_path` matcher, which takes
 the `source` field, extracts the container ID, and uses it to retrieve metadata.
 
-The configuration below enables the processor when the Beat is run as a pod in
+The configuration below enables the processor when {beatname_lc} is run as a pod in
 Kubernetes.
 
 [source,yaml]

libbeat/docs/repositories.asciidoc

@@ -88,7 +88,7 @@ install {beatname_uc} by running:
 sudo apt-get update && sudo apt-get install {beatname_pkg}
 --------------------------------------------------
 
-. To configure the Beat to start automatically during boot, run:
+. To configure {beatname_uc} to start automatically during boot, run:
 +
 ["source","sh",subs="attributes"]
 --------------------------------------------------

libbeat/docs/security/basic-auth.asciidoc

@@ -12,39 +12,39 @@ To configure authentication credentials for {beatname_uc}:
 `monitor` cluster privileges, and `read`, `write`, and `create_index`
 privileges for the indices that {beatname_uc} creates. You can create roles from the
 **Management / Roles** UI in {kib} or through the `role` API.
-For example, the following request creates a ++{beatname_lc}_writer++ role:
+For example, the following request creates a ++{beat_default_index_prefix}_writer++ role:
 +
 ["source","sh",subs="attributes,callouts"]
 ---------------------------------------------------------------
-POST _xpack/security/role/{beatname_lc}_writer
+POST _xpack/security/role/{beat_default_index_prefix}_writer
 {
   "cluster": ["manage_index_templates", "monitor"],
   "indices": [
     {
-      "names": [ "{beatname_lc}-*" ], <1>
+      "names": [ "{beat_default_index_prefix}-*" ], <1>
       "privileges": ["write","create_index"]
     }
   ]
 }
 ---------------------------------------------------------------
 <1> If you use a custom {beatname_uc} index pattern, specify that pattern
-instead of the default ++{beatname_lc}-*++ pattern.
+instead of the default ++{beat_default_index_prefix}-*++ pattern.
 
 . Assign the writer role to the user that {beatname_uc} will use to connect to
 {es}:
 
 .. To authenticate as a native user, create a user for the {beatname_uc} to use
 internally and assign it the writer role. You can create users from the
 **Management / Users** UI in {kib} or through the `user` API. For example, the
-following request creates a ++{beatname_lc}_internal++ user that has the
-++{beatname_lc}_writer++ role:
+following request creates a ++{beat_default_index_prefix}_internal++ user that has the
+++{beat_default_index_prefix}_writer++ role:
 +
 ["source","sh",subs="attributes,callouts"]
 ---------------------------------------------------------------
-POST /_xpack/security/user/{beatname_lc}_internal
+POST /_xpack/security/user/{beat_default_index_prefix}_internal
 {
   "password" : "x-pack-test-password",
-  "roles" : [ "{beatname_lc}_writer"],
+  "roles" : [ "{beat_default_index_prefix}_writer"],
   "full_name" : "Internal {beatname_uc} User"
 }
 ---------------------------------------------------------------
@@ -56,7 +56,7 @@ the user by the distinguished name that appears in its certificate.
 --
 ["source","yaml",subs="attributes,callouts"]
 ---------------------------------------------------------------
-{beatname_lc}_writer:
+{beat_default_index_prefix}_writer:
   - "cn=Internal {beatname_uc} User,ou=example,o=com"
 ---------------------------------------------------------------
 For more information, see
@@ -68,14 +68,14 @@ in the {beatname_uc} configuration file:
 
 .. To use basic authentication, configure the `username` and `password`
 settings. For example, the following {beatname_uc} output configuration
-uses the native ++{beatname_lc}_internal++ user to connect to {es}:
+uses the native ++{beat_default_index_prefix}_internal++ user to connect to {es}:
 +
 ["source","js",subs="attributes,callouts"]
 --------------------------------------------------
 output.elasticsearch:
     hosts: ["localhost:9200"]
-    index: "{beatname_lc}"
-    username: "{beatname_lc}_internal"
+    index: "{beat_default_index_prefix}"
+    username: "{beat_default_index_prefix}_internal"
     password: "x-pack-test-password"
 --------------------------------------------------
 
@@ -86,7 +86,7 @@ output.elasticsearch:
 --------------------------------------------------
 output.elasticsearch:
     hosts: ["localhost:9200"]
-    index: "{beatname_lc}"
+    index: "{beat_default_index_prefix}"
     ssl.certificate: "/etc/pki/client/cert.pem" <1>
     ssl.key: "/etc/pki/client/cert.key"
 --------------------------------------------------