From 3f1fec1813ef095d5cdc8e3c6bb2aa3b69feb090 Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Fri, 26 Oct 2018 15:19:35 +0300 Subject: [PATCH] [Docs] audit logfile structured format (#34584) Documents the new structured logfile format for auditing that was introduced by #31931. Most changes herein are for 6.x . In 7.0 the deprecated format is gone and a follow-up PR is in order. --- .../settings/audit-settings.asciidoc | 48 +++++++++++++------ .../security/auditing/output-logfile.asciidoc | 40 ++++++++++++++-- .../en/security/auditing/overview.asciidoc | 11 +++-- 3 files changed, 77 insertions(+), 22 deletions(-) diff --git a/docs/reference/settings/audit-settings.asciidoc b/docs/reference/settings/audit-settings.asciidoc index 524198df58c47..ec661a1f30c9f 100644 --- a/docs/reference/settings/audit-settings.asciidoc +++ b/docs/reference/settings/audit-settings.asciidoc @@ -18,10 +18,18 @@ Set to `true` to enable auditing on the node. The default value is `false`. `xpack.security.audit.outputs`:: Specifies where audit logs are output. For example: `[ index, logfile ]`. The default value is `logfile`, which puts the auditing events in a dedicated -`_access.log` file on the node. You can also specify `index`, which -puts the auditing events in an {es} index that is prefixed with -`.security_audit_log`. The index can reside on the same cluster or a separate -cluster. +file named `_audit.log` on each node. +You can also specify `index`, which puts the auditing events in an {es} index +that is prefixed with `.security_audit_log`. The index can reside on the same +cluster or a separate cluster. + +For backwards compatibility reasons, if you use the logfile output type, a +`_access.log` file is also created. It contains the same +information, but it uses the older (pre-6.5.0) formatting style. +If the backwards compatible format is not required, it should be disabled. +To do that, change its logger level to `off` in the `log4j2.properties` file. +For more information, see <>. + + -- TIP: If the index is unavailable, it is possible for auditing events to @@ -57,17 +65,27 @@ audited in plain text when including the request body in audit events. [[node-audit-settings]] ==== Local Node Info Settings -`xpack.security.audit.logfile.prefix.emit_node_name`:: -Specifies whether to include the node's name in the local node info. The -default value is `true`. - -`xpack.security.audit.logfile.prefix.emit_node_host_address`:: -Specifies whether to include the node's IP address in the local node info. The -default value is `false`. - -`xpack.security.audit.logfile.prefix.emit_node_host_name`:: -Specifies whether to include the node's host name in the local node info. The -default value is `false`. +`xpack.security.audit.logfile.emit_node_name`:: +Specifies whether to include the <> as a field in +each audit event. +The default value is `true`. + +`xpack.security.audit.logfile.emit_node_host_address`:: +Specifies whether to include the node's IP address as a field in each audit event. +The default value is `false`. + +`xpack.security.audit.logfile.emit_node_host_name`:: +Specifies whether to include the node's host name as a field in each audit event. +The default value is `false`. + +`xpack.security.audit.logfile.emit_node_id`:: +Specifies whether to include the node id as a field in each audit event. +This is available for the new format only. That is to say, this information +does not exist in the `_access.log` file. +Unlike <>, whose value might change if the administrator +changes the setting in the config file, the node id will persist across cluster +restarts and the administrator cannot change it. +The default value is `true`. [[index-audit-settings]] ==== Audit Log Indexing Configuration Settings diff --git a/x-pack/docs/en/security/auditing/output-logfile.asciidoc b/x-pack/docs/en/security/auditing/output-logfile.asciidoc index ee33f618f9665..d9e7eb81c3f3a 100644 --- a/x-pack/docs/en/security/auditing/output-logfile.asciidoc +++ b/x-pack/docs/en/security/auditing/output-logfile.asciidoc @@ -3,13 +3,41 @@ === Logfile audit output The `logfile` audit output is the default output for auditing. It writes data to -the `_access.log` file in the logs directory. +the `_audit.log` file in the logs directory. To maintain +compatibility with releases prior to 6.5.0, a `_access.log` file +is also generated. They differ in the output format but the contents +are similar. For systems that are not ingesting the audit file for search or +analytics it is strongly recommended to only keep the newer format. +Turning off the deprecated output format can be achieved by disabling the logger +in the `log4j2.properties` file (hint: there is a config comment +about it). +For more information, see {ref}/logging.html#configuring-logging-levels[configuring-logging]. + [float] [[audit-log-entry-format]] === Log entry format -The format of a log entry is: +The log entries in the `_audit.log` file +have the following format: + +- Each log entry is a one line JSON document and each one is printed on a separate line. +- The fields of a log entry are ordered. However, if a field does not have a value it + will not be printed. The precise line pattern, together with the complete field + order, are specified in the `log4j2.properties` config file. +- The log entry does not contain nested inner JSON objects, i.e. the doc is flat. +- The field names follow a dotted notation to flatten inner objects. +- A field's value can be a string, a number or an array of strings. +- A field's value, a request body as well, will be escaped as per the JSON RFC 4627. + +There is a list of <> specifying the +set of fields for each sog entry type. + +[float] +[[deprecated-audit-log-entry-format]] +=== Deprecated log entry format + +The log entries in the `_access.log` file have the following format: [source,txt] ---------------------------------------------------------------------------- @@ -48,8 +76,14 @@ audited in plain text when including the request body in audit events. [[logging-file]] You can also configure how the logfile is written in the `log4j2.properties` file located in `ES_PATH_CONF`. By default, audit information is appended to the -`_access.log` file located in the standard Elasticsearch `logs` directory +`_audit.log` file located in the standard Elasticsearch `logs` directory (typically located at `$ES_HOME/logs`). The file rolls over on a daily basis. +The deprecated logfile audit format (`_access.log`) can be disabled +from the same `log4j2.properties` file (hint: look for the comment +instructing to set the log level to `off`). The deprecated format is a duplication +of information that is in place to assure backwards compatibility. If you are +not strict about the audit format it is strongly recommended to only use the +`_audit.log` log appender. [float] [[audit-log-ignore-policy]] diff --git a/x-pack/docs/en/security/auditing/overview.asciidoc b/x-pack/docs/en/security/auditing/overview.asciidoc index b874af3d1c43c..6f04e17d83138 100644 --- a/x-pack/docs/en/security/auditing/overview.asciidoc +++ b/x-pack/docs/en/security/auditing/overview.asciidoc @@ -16,11 +16,14 @@ must set `xpack.security.audit.enabled` to `true` in `elasticsearch.yml`. {Security} provides two ways to persist audit logs: * The <> output, which persists events to - a dedicated `_access.log` file on the host's file system. -* The <> output, which persists events to an Elasticsearch index. -The audit index can reside on the same cluster, or a separate cluster. + a dedicated `_audit.log` file on the host's file system. + For backwards compatibility reasons, a file named `_access.log` + is also generated. +* The <> output, which persists events to an Elasticsearch + index. The audit index can reside on the same cluster, or a separate cluster. -By default, only the `logfile` output is used when enabling auditing. +By default, only the `logfile` output is used when enabling auditing, +implicitly outputing to both `_audit.log` and `_access.log`. To facilitate browsing and analyzing the events, you can also enable indexing by setting `xpack.security.audit.outputs` in `elasticsearch.yml`: