Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs for the audit logfile structured format #34584

Merged
merged 10 commits into from
Oct 26, 2018
Merged

Docs for the audit logfile structured format #34584

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
0b1bedf
Finally ....
albertzaharovits Oct 18, 2018
a8d8c71
Addressed Jay's feedback
albertzaharovits Oct 18, 2018
5a6d6c0
Lisa's feedback
albertzaharovits Oct 23, 2018
de3d362
fix links
albertzaharovits Oct 23, 2018
445edd2
More feedback
albertzaharovits Oct 23, 2018
4f5b88b
Update docs/reference/settings/audit-settings.asciidoc
lcawl Oct 23, 2018
15b35a0
Update docs/reference/settings/audit-settings.asciidoc
lcawl Oct 23, 2018
5d1055c
Update x-pack/docs/en/security/auditing/output-logfile.asciidoc
lcawl Oct 23, 2018
4dc2b18
Update x-pack/docs/en/security/auditing/output-logfile.asciidoc
lcawl Oct 23, 2018
000778e
Update x-pack/docs/en/security/auditing/output-logfile.asciidoc
lcawl Oct 23, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 31 additions & 15 deletions docs/reference/settings/audit-settings.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,17 @@ Set to `true` to enable auditing on the node. The default value is `false`.
`xpack.security.audit.outputs`::
Specifies where audit logs are output. For example: `[ index, logfile ]`. The
default value is `logfile`, which puts the auditing events in a dedicated
`<clustername>_access.log` file on the node. You can also specify `index`, which
puts the auditing events in an {es} index that is prefixed with
`.security_audit_log`. The index can reside on the same cluster or a separate
cluster.
file named `<clustername>_audit.log` on each node.
You can also specify `index`, which puts the auditing events in an {es} index
that is prefixed with `.security_audit_log`. The index can reside on the same
cluster or a separate cluster.

For backwards compatibility reasons, the logfile contents are duplicated to
the `<clustername>_access.log` file with the format for versions prior to 6.5.0.
If the backwards compatible format is not required, it should be disabled.
albertzaharovits marked this conversation as resolved.
Show resolved Hide resolved
To do that, change its logger level to `off` in the `log4j2.properties` file
from `ES_PATH_CONF`.
Copy link
Contributor

@lcawl lcawl Oct 23, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest changing this to something like this:
"... in the log4j2.properties file. For more information, see <<configuring-logging-levels>>. "


+
--
TIP: If the index is unavailable, it is possible for auditing events to
Expand Down Expand Up @@ -57,17 +64,26 @@ audited in plain text when including the request body in audit events.
[[node-audit-settings]]
==== Local Node Info Settings

`xpack.security.audit.logfile.prefix.emit_node_name`::
Specifies whether to include the node's name in the local node info. The
default value is `true`.

`xpack.security.audit.logfile.prefix.emit_node_host_address`::
Specifies whether to include the node's IP address in the local node info. The
default value is `false`.

`xpack.security.audit.logfile.prefix.emit_node_host_name`::
Specifies whether to include the node's host name in the local node info. The
default value is `false`.
`xpack.security.audit.logfile.emit_node_name`::
Specifies whether to include the <<node-name, node name>> as a field in each
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This link fails when I build the documentation. Are you trying to link to https://www.elastic.co/guide/en/elasticsearch/reference/master/node.name.html ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

replaced with {ref}/node.name.html[node name]

audit event.
The default value is `true`.

`xpack.security.audit.logfile.emit_node_host_address`::
Specifies whether to include the node's IP address as a field in each audit event.
The default value is `false`.

`xpack.security.audit.logfile.emit_node_host_name`::
Specifies whether to include the node's host name as a field in each audit event.
The default value is `false`.

`xpack.security.audit.logfile.emit_node_id`::
Specifies whether to include the node id as a field in each audit event.
This is available for the new format only.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it worth adding "that is to say, this information does not exist in the <clustername>_access.log file"?

Unlike <<node-name, node name>>, whose value might change if the administrator
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This "node-name" link also fails. If you let me know what you're trying to link to, I can help fix it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

replaced with {ref}/node.name.html[node name]

changes the setting in the config file, the node id will persist across cluster
restarts and the administrator has no handle to change it.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "has no handle to change it" is a bit unclear to me. Can we just say, "cannot change it"?

The default value is `true`.

[[index-audit-settings]]
==== Audit Log Indexing Configuration Settings
Expand Down
38 changes: 35 additions & 3 deletions x-pack/docs/en/security/auditing/output-logfile.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,39 @@
=== Logfile audit output

The `logfile` audit output is the default output for auditing. It writes data to
the `<clustername>_access.log` file in the logs directory.
the `<clustername>_audit.log` file in the logs directory. To maintain
compatibility with releases prior to 6.5.0, another file `<clustername>_access.log`
is also generated next to it. They differ in the output format but the contents
albertzaharovits marked this conversation as resolved.
Show resolved Hide resolved
are similar. For systems that are not ingesting the audit file for search or
analytics it is strongly recommended to only keep the newer format.
Turning off the deprecated output format can be achieved by disabling the logger
in the `log4j2.properties` file in `ES_PATH_CONF` (hint: there is a config comment
about it).
albertzaharovits marked this conversation as resolved.
Show resolved Hide resolved

[float]
[[audit-log-entry-format]]
=== Log entry format

The format of a log entry is:
This is describing the format for log entries as they are appended to the
albertzaharovits marked this conversation as resolved.
Show resolved Hide resolved
new `<clustername>_audit.log` file.
albertzaharovits marked this conversation as resolved.
Show resolved Hide resolved

- Each log entry is a one line JSON document and each one is printed on a separate line.
- The fields of a log entry are ordered. However, if a field does not have a value it
will not be printed. The precise line pattern, together with the complete field
order, are specified in the `log4j2.properties` config file.
- The log entry does not contain nested inner JSON objects, i.e. the doc is flat.
- The field names follow a dotted notation to flatten inner objects.
- A field's value can be a string, a number or an array of strings.
- A field's value, a request body as well, will be escaped as per the JSON RFC 4627.

There is a list of <<audit-event-types, audit event types>> specifying the
set of fields for each sog entry type.

[float]
[[deprecated-audit-log-entry-format]]
=== Deprecated log entry format

This is the format for a log entries as appended to `<clustername>_access.log`:
albertzaharovits marked this conversation as resolved.
Show resolved Hide resolved

[source,txt]
----------------------------------------------------------------------------
Expand Down Expand Up @@ -48,8 +74,14 @@ audited in plain text when including the request body in audit events.
[[logging-file]]
You can also configure how the logfile is written in the `log4j2.properties`
file located in `ES_PATH_CONF`. By default, audit information is appended to the
`<clustername>_access.log` file located in the standard Elasticsearch `logs` directory
`<clustername>_audit.log` file located in the standard Elasticsearch `logs` directory
(typically located at `$ES_HOME/logs`). The file rolls over on a daily basis.
The deprecated logfile audit format (`<clustername>_access.log`) can be disabled
from the same `log4j2.properties` file (hint: look for the comment
instructing to set the log level to `off`). The deprecated format is a duplication
of information that is in place to assure backwards compatibility. If you are
not strict about the audit format it is strongly recommended to only use the
`<clustername>_audit.log` log appender.

[float]
[[audit-log-ignore-policy]]
Expand Down
15 changes: 11 additions & 4 deletions x-pack/docs/en/security/auditing/overview.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,18 @@ must set `xpack.security.audit.enabled` to `true` in `elasticsearch.yml`.
{Security} provides two ways to persist audit logs:

* The <<audit-log-output, `logfile`>> output, which persists events to
a dedicated `<clustername>_access.log` file on the host's file system.
* The <<audit-index, `index`>> output, which persists events to an Elasticsearch index.
The audit index can reside on the same cluster, or a separate cluster.
a dedicated `<clustername>_audit.log` file on the host's file system.
For backwards compatibility reasons, a file named `<clustername>_access.log`
is also generated. This has a format compatible to the one in releases prior
to 6.5.0. Between the two, the contents are identical, therefore it is recommended
to disable the `<clustername>_access.log` output. For this, turn off the
logger in the `log4j2.properties` config file from `ES_PATH_CONF`. (hint: look
out for a comment in the config file).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be too much information for such a high-level introductory page. i.e. since it's covered in the linked logfile page, I think you could just cut off after the "...is also generated" sentence.

* The <<audit-index, `index`>> output, which persists events to an Elasticsearch
index. The audit index can reside on the same cluster, or a separate cluster.

By default, only the `logfile` output is used when enabling auditing.
By default, only the `logfile` output is used when enabling auditing,
implicitly outputing to both `<clustername>_audit.log` and `<clustername>_access.log`.
To facilitate browsing and analyzing the events, you can also enable
indexing by setting `xpack.security.audit.outputs` in `elasticsearch.yml`:

Expand Down