Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Add processor.event info to Logstash output #20721

Merged
merged 5 commits into from
Sep 1, 2020
Merged

docs: Add processor.event info to Logstash output #20721

Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
931352d
docs: APM Server updates
bmorelli25 Aug 20, 2020
f97fdce
docs: us {ls} instead of Logstash
bmorelli25 Aug 20, 2020
037ba9d
docs: other small fixes
bmorelli25 Aug 20, 2020
964ef25
Apply suggestions from code review
bmorelli25 Aug 31, 2020
5481ee5
docs: use attributes instead of product names
bmorelli25 Aug 31, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 32 additions & 21 deletions libbeat/outputs/logstash/docs/logstash.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@
<titleabbrev>Logstash</titleabbrev>
++++

The Logstash output sends events directly to Logstash by using the lumberjack
protocol, which runs over TCP. Logstash allows for additional processing and routing of
The {ls} output sends events directly to {ls} by using the lumberjack
protocol, which runs over TCP. {ls} allows for additional processing and routing of
generated events.

// tag::shared-logstash-config[]
Expand All @@ -26,11 +26,10 @@ If you want to use {ls} to perform additional processing on the data collected b

To do this, edit the {beatname_uc} configuration file to disable the {es}
output by commenting it out and enable the {ls} output by uncommenting the
logstash section:
{ls} section:

[source,yaml]
------------------------------------------------------------------------------
#----------------------------- Logstash output --------------------------------
output.logstash:
hosts: ["127.0.0.1:5044"]
------------------------------------------------------------------------------
Expand Down Expand Up @@ -71,6 +70,9 @@ for more about the `@metadata` field.
<2> The default is {beat_default_index_prefix}. To change this value, set the
<<logstash-index,`index`>> option in the {beatname_uc} config file.
<3> The current version of {beatname_uc}.

You can access this metadata from within the Logstash config file to set values
bmorelli25 marked this conversation as resolved.
Show resolved Hide resolved
dynamically based on the contents of the metadata.
endif::[]

ifdef::apm-server[]
Expand All @@ -85,24 +87,24 @@ ifdef::apm-server[]
}
}
------------------------------------------------------------------------------
<1> {beatname_uc} uses the `@metadata` field to send metadata to Logstash. See the
<1> {beatname_uc} uses the `@metadata` field to send metadata to {ls}. See the
{logstash-ref}/event-dependent-configuration.html#metadata[Logstash documentation]
for more about the `@metadata` field.
<2> The default is {beat_default_index_prefix}. To change this value, set the
<<logstash-index,`index`>> option in the {beatname_uc} config file.
<3> The default pipeline configuration: `apm`. Additional pipelines can be enabled
with a {logstash-ref}/use-ingest-pipelines.html[Logstash pipeline config].
bmorelli25 marked this conversation as resolved.
Show resolved Hide resolved
<4> The current version of {beatname_uc}.
endif::[]

You can access this metadata from within the Logstash config file to set values
dynamically based on the contents of the metadata.
In addition to metadata, {beatname_uc} provides the `processor.event` field, which
can be used to separate {apm-overview-ref-v}/apm-data-model.html[event types] into different indices.
endif::[]

ifndef::apm-server[]
For example, the following Logstash configuration file tells
bmorelli25 marked this conversation as resolved.
Show resolved Hide resolved
Logstash to use the index reported by {beatname_uc} for indexing events
bmorelli25 marked this conversation as resolved.
Show resolved Hide resolved
into Elasticsearch:
bmorelli25 marked this conversation as resolved.
Show resolved Hide resolved

ifndef::apm-server[]
[source,logstash]
------------------------------------------------------------------------------

Expand All @@ -126,6 +128,10 @@ the Beat's version. For example:
endif::[]

ifdef::apm-server[]
For example, the following {ls} configuration file tells
{ls} to use the index and event types reported by {beatname_uc} for indexing events
into Elasticsearch:
bmorelli25 marked this conversation as resolved.
Show resolved Hide resolved

[source,logstash]
------
input {
Expand Down Expand Up @@ -156,11 +162,11 @@ output {
}
------
<1> Creates a new field named `@metadata.index`.
`%{[@metadata][beat]}` sets the first part of the index name to the value of the `beat` metadata field.
`%{[@metadata][beat]}` sets the first part of the index name to the value of the `metadata.beat` field.
`%{[@metadata][version]}` sets the second part to {beatname_uc}'s version.
`%{[processor][event]}` sets the final part based on the APM event type.
For example: +{beat_default_index_prefix}-{version}-sourcemap+.
<2> In addition to the above rules, this pattern appends a date to the `index` name so Logstash creates a new index each day.
<2> In addition to the above rules, this pattern appends a date to the `index` name so {ls} creates a new index each day.
For example: +{beat_default_index_prefix}-{version}-transaction-{sample_date_0}+.
endif::[]

Expand All @@ -172,10 +178,10 @@ NOTE: If ILM is not being used, set `index` to `%{[@metadata][beat]}-%{[@metadat
endif::[]

ifdef::apm-server[]
==== Logstash and ILM
==== {ls} and ILM

When used with {apm-server-ref}/ilm.html[Index lifecycle management], Logstash does not need to create a new index each day.
Here's a sample Logstash configuration file that would accomplish this:
When used with {apm-server-ref}/ilm.html[Index lifecycle management], {ls} does not need to create a new index each day.
Here's a sample {ls} configuration file that would accomplish this:

[source,logstash]
------
Expand All @@ -188,10 +194,15 @@ input {
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[processor][event]}"
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[processor][event]}" <1>
}
}
------
<1> Outputs documents to an index:
`%{[@metadata][beat]}` sets the first part of the index name to the value of the `metadata.beat` field.
`%{[@metadata][version]}` sets the second part to {beatname_uc}'s version.
`%{[processor][event]}` sets the final part based on the APM event type.
For example: +{beat_default_index_prefix}-{version}-sourcemap+.
endif::[]

==== Compatibility
Expand Down Expand Up @@ -224,14 +235,14 @@ The list of known Logstash servers to connect to. If load balancing is disabled,
multiple hosts are configured, one host is selected randomly (there is no precedence).
If one host becomes unreachable, another one is selected randomly.

All entries in this list can contain a port number. The default port number 5044 will be used, if no number is given.
All entries in this list can contain a port number. The default port number 5044 will be used if no number is given.

===== `compression_level`

The gzip compression level. Setting this value to 0 disables compression.
The compression level must be in the range of 1 (best speed) to 9 (best compression).

Increasing the compression level will reduce the network usage but will increase the cpu usage.
Increasing the compression level will reduce the network usage but will increase the CPU usage.

The default value is 3.

Expand Down Expand Up @@ -277,7 +288,7 @@ NOTE: The "ttl" option is not yet supported on an async Logstash client (one wit

===== `pipelining`

Configures number of batches to be sent asynchronously to logstash while waiting
Configures the number of batches to be sent asynchronously to logstash while waiting
bmorelli25 marked this conversation as resolved.
Show resolved Hide resolved
for ACK from logstash. Output only becomes blocking once number of `pipelining`
bmorelli25 marked this conversation as resolved.
Show resolved Hide resolved
batches have been written. Pipelining is disabled if a value of 0 is
configured. The default value is 2.
Expand Down Expand Up @@ -306,7 +317,7 @@ output.logstash:
===== `proxy_use_local_resolver`

The `proxy_use_local_resolver` option determines if Logstash hostnames are
bmorelli25 marked this conversation as resolved.
Show resolved Hide resolved
resolved locally when using a proxy. The default value is false which means
resolved locally when using a proxy. The default value is false, which means
that when a proxy is used the name resolution occurs on the proxy server.

[[logstash-index]]
Expand Down Expand Up @@ -364,9 +375,9 @@ number of events to be contained in a batch.

===== `slow_start`

If enabled only a subset of events in a batch of events is transferred per transaction.
If enabled, only a subset of events in a batch of events is transferred per transaction.
The number of events to be sent increases up to `bulk_max_size` if no error is encountered.
On error the number of events per transaction is reduced again.
On error, the number of events per transaction is reduced again.

The default is `false`.

Expand Down