Backport to 6.x: Docs for new beats (elastic#8811, elastic#8864, elas…

…tic#8744) (elastic#9024)

* Add conditions for each platform (elastic#8811)

* Add conditions for each platform

* Add conditions to directory layout topic

* Combine conditions for deb and rpm

* Update Journalbeat docs (elastic#8864)

* More updates for journalbeat

* Remove unwanted comments and extra files

* Add changes from review

* Changes from second round of reviews

* Remove backoff options

* Remove questions for reviewers

* Change file to journal

* Change default  max_backoff in docs to 20s

* More fixes from review

* Revert "Remove backoff options"

This reverts commit d57e4b3.

* Add global options back

* Initialize docs for functionbeat (elastic#8744)

* Initialize docs

* Run make update

* In progress fixes from review

* More changes from the review and testing

* Add fixes from the review

* Fixed download path

* Remove unused dashboard image placeholders

* Add missing options and remove boiler plate text

* Run make update

* Remove broken link

docs/config-options.asciidoc

@@ -5,70 +5,68 @@
 <titleabbrev>Configure inputs</titleabbrev>
 ++++
 
-IMPORTANT: This documentation is placeholder content. It has not yet been reviewed.
-
 By default, {beatname_uc} reads log events from the default systemd journals. To
 specify other journal files, set the <<{beatname_lc}-paths,`paths`>> option in
-the +{beatname_lc}.inputs+ section of the +{beatname_lc}.yml+ file. 
-
-The list of paths is a YAML array, so each path begins with a dash (-). Each
-path can be a directory path (to collect events from all journals in a
-directory), or a file path. For example:
+the +{beatname_lc}.inputs+ section of the +{beatname_lc}.yml+ file. Each path
+can be a directory path (to collect events from all journals in a directory), or
+a file path. For example:
 
 ["source","sh",subs="attributes"]
 ----
 {beatname_lc}.inputs:
 - paths:
   - "/dev/log"
-  - "/var/log/messages/my-journal-file"
+  - "/var/log/messages/my-journal-file.journal"
 ----
 
-Within the +{beatname_lc}.inputs+ section, you can also specify options that
-control the position where {beatname_uc} starts reading the journal file, and
-set filters to reduce the fields that {beatname_uc} needs to process. See
-<<{beatname_lc}-options>> for a list of available options.
-
-[float]
-=== Configuration examples
-
-The following example shows how to monitor multiple journals under the
-same directory. {beatname_uc} merges all journals under the directory into a
-single journal and reads them. With `seek` set to `cursor`, {beatname_uc}
-starts reading at the beginning of the journal, but will continue reading where
-it left off after a reload or restart. 
-
+Within the configuration file, you can also specify options that control how
+{beatname_uc} reads the journal files and which fields are sent to the
+configured output. See <<{beatname_lc}-options>> for a list of available
+options.
+
+The following examples show how to configure {beatname_uc} for some common use
+cases.
+
+[[monitor-multiple-journals]]
+.Example 1: Monitor multiple journals under the same directory
+This example configures {beatname_uc} to read from multiple journals that are
+stored under the same directory. {beatname_uc} merges all journals under the
+directory into a single event stream and reads the events. With `seek` set to
+`cursor`, {beatname_uc} starts reading at the beginning of the journal, but will
+continue reading at the last known position after a reload or restart.
 ["source","sh",subs="attributes"]
 ----
 {beatname_lc}.inputs:
 - paths: ["/path/to/journal/directory"]
   seek: cursor
 ----
 
-The following examples show how to get Redis events from a Docker container that
-is tagged as `redis`.
-
-//TODO: Add a better explanation of the options.
-
-This example uses the translated fields by Journald:
-
+[[filter-using-field-names]]
+.Example 2: Fetch log events for Redis running on Docker (uses field names from systemd)
+This example configures {beatname_uc} to fetch log events for Redis running in a
+Docker container. The fields are matched using field names from the systemd
+journal.
 ["source","sh",subs="attributes"]
 ----
 {beatname_lc}.inputs:
 - paths: []
   include_matches:
-    - "container.image.tag=redis"
-    - "process.name=redis"
+    - "CONTAINER_TAG=redis"
+    - "_COMM=redis"
 ----
 
-This example uses the field names from the systemd journal:
-
+[[filter-using-translated-names]]
+.Example 3: Fetch log events for Redis running on Docker (uses translated field names)
+This example also configures {beatname_uc} to fetch log events for Redis running
+in a Docker container. However, in this example the fields are matched using the
+<<translated-fields,translated field names>> provided by {beatname_uc}.
 ["source","sh",subs="attributes"]
 ----
 {beatname_lc}.inputs:
 - paths: []
   include_matches:
-    - "CONTAINER_TAG=redis"
-    - "_COMM=redis"
+    - "container.image.tag=redis"
+    - "process.name=redis"
 ----
 
 [id="{beatname_lc}-options"]
@@ -86,17 +84,134 @@ path (to collect events from all journals in a directory), or a file path. If
 you specify a directory, {beatname_uc} merges all journals under the directory
 into a single journal and reads them.
 
-//QUESTION: Are globs supported? If so, I need to add more detail here. 
+If no paths are specified, {beatname_uc} reads from the default journal.
+
+[float]
+[id="{beatname_lc}-backoff"]
+==== `backoff`
+
+The number of seconds to wait before trying to read again from journals. The
+default is 1s.
+
+[float]
+[id="{beatname_lc}-max-backoff"]
+==== `max_backoff`
+
+The maximum number of seconds to wait before attempting to read again from
+journals. The default is 60s.
 
 [float]
 [id="{beatname_lc}-seek"]
 ==== `seek`
 
 The position to start reading the journal from. Valid settings are:
 
-* `head`: Starts reading at the beginning of the file.
-* `tail`: Starts reading at the end of the file.
-* `cursor`: Initially starts reading at the beginning of the file, but continues
-reading where it left off after a reload or restart. 
+* `head`: Starts reading at the beginning of the journal. After a restart,
+{beatname_uc} resends all log messages in the journal.
+* `tail`: Starts reading at the end of the journal. After a restart, 
+{beatname_uc} resends the last message, which might result in duplicates. If
+multiple log messages are written to a journal while {beatname_uc} is down,
+only the last log message is sent on restart. 
+* `cursor`: On first read, starts reading at the beginning of the journal. After a
+reload or restart, continues reading at the last known position.
+
+When specified under `paths`, the `seek` setting applies to all journals under
+the configured paths. When specified directly under the +{beatname_lc}+
+namespace, the setting applies to all journals read by {beatname_uc}.
 
-//TODO: ADD OTHER OPTIONS HERE.     
+If you have old log files and want to skip lines, start {beatname_uc} with
+`seek: tail` specified. Then stop {beatname_uc}, set `seek: cursor`, and restart
+{beatname_uc}.
+
+[float]
+[id="{beatname_lc}-include-matches"]
+==== `include_matches`
+
+A list of filter expressions used to match fields. The format of the expression
+is `field=value`. {beatname_uc} fetches all events that exactly match the
+expressions. Pattern matching is not supported.
+
+To reference fields, use one of the following:
+
+* The field name used by the systemd journal. For example,
+`CONTAINER_TAG=redis` (<<filter-using-field-names,see a full example>>).
+* The <<translated-fields,translated field name>> used by
+{beatname_uc}. For example, `container.image.tag=redis`
+(<<filter-using-translated-names,see a full example>>). {beatname_uc}
+does not translate all fields from the journal. For custom fields, use the name
+specified in the systemd journal.
+
+When specified under `paths`, the `include_matches` filter is applied to all
+journals under the configured paths. When specified directly under the
++{beatname_lc}+ namespace, the setting applies to all journals read by
+{beatname_uc}.
+
+[float]
+[[translated-fields]]
+=== Translated field names
+
+You can use the following translated names in filter expressions to reference
+journald fields:
+
+[horizontal]
+*Journald field name*:: *Translated name*
+`COREDUMP_UNIT`::             `journald.coredump.unit`
+`COREDUMP_USER_UNIT`::        `journald.coredump.user_unit`
+`OBJECT_AUDIT_LOGINUID`::     `journald.object.audit.login_uid`
+`OBJECT_AUDIT_SESSION`::      `journald.object.audit.session`
+`OBJECT_CMDLINE`::            `journald.object.cmd`
+`OBJECT_COMM`::               `journald.object.name`
+`OBJECT_EXE`::                `journald.object.executable`
+`OBJECT_GID`::                `journald.object.gid`
+`OBJECT_PID`::                `journald.object.pid`
+`OBJECT_SYSTEMD_OWNER_UID`::  `journald.object.systemd.owner_uid`
+`OBJECT_SYSTEMD_SESSION`::    `journald.object.systemd.session`
+`OBJECT_SYSTEMD_UNIT`::       `journald.object.systemd.unit`
+`OBJECT_SYSTEMD_USER_UNIT`::  `journald.object.systemd.user_unit`
+`OBJECT_UID`::                `journald.object.uid`
+`_AUDIT_LOGINUID`::            `process.audit.login_uid`
+`_AUDIT_SESSION`::             `process.audit.session`
+`_BOOT_ID`::                   `host.boot_id`
+`_CAP_EFFECTIVE`::             `process.capabilites`
+`_CMDLINE`::                   `process.cmd`
+`_CODE_FILE`::                 `journald.code.file`
+`_CODE_FUNC`::                 `journald.code.func`
+`_CODE_LINE`::                 `journald.code.line`
+`_COMM`::                      `process.name`
+`_EXE`::                       `process.executable`
+`_GID`::                       `process.uid`
+`_HOSTNAME`::                  `host.name`
+`_KERNEL_DEVICE`::            `journald.kernel.device`
+`_KERNEL_SUBSYSTEM`::         `journald.kernel.subsystem`
+`_MACHINE_ID`::                `host.id`
+`_MESSAGE`::                   `message`
+`_PID`::                       `process.pid`
+`_PRIORITY`::                  `syslog.priority`
+`_SYSLOG_FACILITY`::           `syslog.facility`
+`_SYSLOG_IDENTIFIER`::         `syslog.identifier`
+`_SYSLOG_PID`::                `syslog.pid`
+`_SYSTEMD_CGROUP`::            `systemd.cgroup`
+`_SYSTEMD_INVOCATION_ID`::    `systemd.invocation_id`
+`_SYSTEMD_OWNER_UID`::         `systemd.owner_uid`
+`_SYSTEMD_SESSION`::           `systemd.session`
+`_SYSTEMD_SLICE`::             `systemd.slice`
+`_SYSTEMD_UNIT`::              `systemd.unit`
+`_SYSTEMD_USER_SLICE`::       `systemd.user_slice`
+`_SYSTEMD_USER_UNIT`::         `systemd.user_unit`
+`_TRANSPORT`::                 `systemd.transport`
+`_UDEV_DEVLINK`::             `journald.kernel.device_symlinks`
+`_UDEV_DEVNODE`::             `journald.kernel.device_node_path`
+`_UDEV_SYSNAME`::             `journald.kernel.device_name`
+`_UID`::                       `process.uid`
+
+
+The following translated fields for
+https://docs.docker.com/config/containers/logging/journald/[Docker] are also
+available: 
+
+[horizontal]
+`CONTAINER_ID`::              `conatiner.id_truncated`
+`CONTAINER_ID_FULL`::         `container.id`
+`CONTAINER_NAME`::            `container.name`
+`CONTAINER_PARTIAL_MESSAGE`:: `container.partial`
+`CONTAINER_TAG`::             `container.image.tag`

docs/configuring-howto.asciidoc

@@ -4,21 +4,11 @@
 [partintro]
 --
 
-IMPORTANT: This documentation is placeholder content. It has not yet been reviewed.
-
 Before modifying configuration settings, make sure you've completed the
 <<{beatname_lc}-configuration,configuration steps>> in the Getting Started.
 This section describes some common use cases for changing configuration options.
 
-To configure {beatname_uc}, you edit the configuration file. For rpm and deb,
-you’ll find the configuration file at +/etc/{beatname_lc}/{beatname_lc}.yml+.
-There's also a full example configuration file at
-+/etc/{beatname_lc}/{beatname_lc}.reference.yml+ that shows all non-deprecated
-options. For mac and win, look in the archive that you extracted.
-
-The {beatname_uc} configuration file uses http://yaml.org/[YAML] for its syntax.
-See the {libbeat}/config-file-format.html[Config File Format] section of the
-_{libbeat_docs}_ for more about the structure of the config file.
+include::../../libbeat/docs/shared-configuring.asciidoc[]
 
 The following topics describe how to configure {beatname_uc}:
 
@@ -31,25 +21,21 @@ The following topics describe how to configure {beatname_uc}:
 * <<configuring-ingest-node>>
 * <<configuration-path>>
 * <<setup-kibana-endpoint>>
-* <<configuration-dashboards>>
 * <<configuration-template>>
 * <<configuration-logging>>
 * <<using-environ-vars>>
-//* <<configuration-autodiscover>>
 * <<yaml-tips>>
 * <<regexp-support>>
 * <<http-endpoint>>
-//* <<{beatname_lc}-reference-yml>>
+* <<{beatname_lc}-reference-yml>>
 
 --
 
 include::./config-options.asciidoc[]
 
 include::./general-options.asciidoc[]
 
-:allplatforms:
 include::../../libbeat/docs/queueconfig.asciidoc[]
-:allplatforms!:
 
 include::../../libbeat/docs/outputconfig.asciidoc[]
 
@@ -71,20 +57,12 @@ include::../../libbeat/docs/loggingconfig.asciidoc[]
 include::../../libbeat/docs/shared-env-vars.asciidoc[]
 :standalone!:
 
-//OPEN ISSUE: DO WE NEED AUTODISCOVER?
-//include::../../libbeat/docs/shared-autodiscover.asciidoc[]
-
 :standalone:
-:allplatforms:
 include::../../libbeat/docs/yaml.asciidoc[]
 :standalone!:
-:allplatforms!:
 
 include::../../libbeat/docs/regexp.asciidoc[]
 
 include::../../libbeat/docs/http-endpoint.asciidoc[]
 
-// TODO: Uncomment the following include statement when the reference yaml file
-// is available in the repo. Also uncomment the link in the jump list at the top
-// of this file.
-//include::../../libbeat/docs/reference-yml.asciidoc[]
+include::../../libbeat/docs/reference-yml.asciidoc[]

docs/faq.asciidoc

@@ -1,24 +1,10 @@
 [[faq]]
 == Frequently asked questions
 
-IMPORTANT: This documentation is placeholder content. It has not yet been reviewed.
-
 This section contains frequently asked questions about {beatname_uc}. Also check
 out the https://discuss.elastic.co/c/beats/{beatname_lc}[{beatname_uc}
 discussion forum].
 
-[float]
-[id="{beatname_lc}-sometext"]
-=== Question 1?
-
-ADD DESCRIPTION HERE
-
-[float]
-[id="{beatname_lc}-sometext2"]
-=== Question 2?
-
-ADD DESCRIPTION HERE
-
 include::../../libbeat/docs/faq-limit-bandwidth.asciidoc[]
 
 include::../../libbeat/docs/shared-faq.asciidoc[]

docs/filtering.asciidoc

@@ -1,20 +1,15 @@
 [[filtering-and-enhancing-data]]
 == Filter and enhance the exported data
 
-IMPORTANT: This documentation is placeholder content. It has not yet been reviewed.
-
 Your use case might require only a subset of the data exported by {beatname_uc},
 or you might need to enhance the exported data (for example, by adding
 metadata). {beatname_uc} provides a couple of options for filtering and
 enhancing exported data.
 
-You can configure each input to include or exclude specific lines or files. This
-allows you to specify different filtering criteria for each input. To do this,
-you use the `include_lines`, `exclude_lines`, and `exclude_files` options under
-the +{beatname_lc}.inputs+ section of the config file (see
-<<configuration-{beatname_lc}-options>>). The disadvantage of this approach is
-that you need to implement a configuration option for each filtering criteria
-that you need.
+You can configure {beatname_uc} to include events that match specific filtering
+criteria. To do this, use the <<{beatname_lc}-include-matches,`include_matches`>>
+option. The advantage of this approach is that you can reduce the number of
+fields that {beatname_uc} needs to process.
 
 Another approach (the one described here) is to define processors to configure
 global processing across all data exported by {beatname_uc}.
@@ -26,12 +21,6 @@ global processing across all data exported by {beatname_uc}.
 
 include::../../libbeat/docs/processors.asciidoc[]
 
-[float]
-[[specific-example]]
-==== XYZ example
-
-ADD EXAMPLES SPECIFIC TO THE BEAT, OR DELETE THIS SECTION
-
 // You must set the processor-scope attribute to resolve the attribute reference
 // defined in processors-using.asciidoc. The attribute is used to indicate where
 // processors are valid. If processors are valid in more than two locations