Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Elastic Log Driver] Create a config shim between libbeat and the user #18605

Merged
merged 21 commits into from
May 29, 2020
Merged

[Elastic Log Driver] Create a config shim between libbeat and the user #18605

Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
a7fa4cb
init commit of config shim
fearful-symmetry May 17, 2020
909c862
update docs
fearful-symmetry May 17, 2020
c361c99
make check
fearful-symmetry May 17, 2020
f430487
add timeout
fearful-symmetry May 19, 2020
376d4f3
Merge remote-tracking branch 'upstream/master' into config-shim
fearful-symmetry May 19, 2020
59a5c74
move config system to use typeconv
fearful-symmetry May 19, 2020
815b1ae
add rest of backoff settings.
fearful-symmetry May 19, 2020
6081adc
make fmt
fearful-symmetry May 19, 2020
87135f5
some cleanup
fearful-symmetry May 19, 2020
29b7cd4
use uint64 hash for structs
fearful-symmetry May 20, 2020
1b73771
make fmt
fearful-symmetry May 20, 2020
5b4f426
create custom index manager, remove ILM support
fearful-symmetry May 21, 2020
111354c
add support for multiple endpoints
fearful-symmetry May 21, 2020
3465bfa
update tests
fearful-symmetry May 21, 2020
10a0536
update docs
fearful-symmetry May 21, 2020
703a132
remove setup options
fearful-symmetry May 26, 2020
26d467c
remove old tests
fearful-symmetry May 26, 2020
3a363a0
try to update asciidocm change 'endpoint' to 'hosts'
fearful-symmetry May 27, 2020
489af14
trying to fix CI
fearful-symmetry May 27, 2020
82f7e8b
update docs
fearful-symmetry May 28, 2020
035e3d8
fix backtics
fearful-symmetry May 28, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
227 changes: 22 additions & 205 deletions x-pack/dockerlogbeat/docs/configuration.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,6 @@ you can set them in the `daemon.json` file for all containers.

* <<cloud-options>>
* <<es-output-options>>
* <<ls-output-options>>
* <<kafka-output-options>>
* <<redis-output-options>>

[float]
=== Usage examples
Expand All @@ -39,11 +36,11 @@ For more examples, see <<log-driver-usage-examples>>.
|=====
|Option | Description

|`cloud.id`
|`cloud_id`
|The Cloud ID found in the Elastic Cloud web console. This ID is
used to resolve the {stack} URLs when connecting to {ess} on {ecloud}.

|`cloud.auth`
|`cloud_auth`
|The username and password combination for connecting to {ess} on {ecloud}. The
format is `"username:password"`.
|=====
Expand All @@ -61,242 +58,62 @@ format is `"username:password"`.
|=====
|Option |Default |Description

|`output.elasticsearch.hosts`
|`hosts`
|`"localhost:9200"`
|The list of {es} nodes to connect to. Specify each node as a `URL` or
`IP:PORT`. For example: `http://192.0.2.0`, `https://myhost:9230` or
`192.0.2.0:9300`. If no port is specified, the default is `9200`.

|`output.elasticsearch.protocol`
|`http`
|The protocol (`http` or `https`) that {es} is reachable on. If you specify a
URL for `hosts`, the value of `protocol` is overridden by whatever scheme you
specify in the URL.

|`output.elasticsearch.username`
|`user`
|
|The basic authentication username for connecting to {es}.

|`output.elasticsearch.password`
|`password`
|
|The basic authentication password for connecting to {es}.

|`output.elasticsearch.index`
|`index`
|
|A {beats-ref}/config-file-format-type.html#_format_string_sprintf[format string]
value that specifies the index to write events to when you're using daily
indices. For example: +"dockerlogs-%{+yyyy.MM.dd}"+.

3+|*Advanced:*

|`output.elasticsearch.backoff.init`
|`backoff_init`
|`1s`
|The number of seconds to wait before trying to reconnect to {es} after
a network error. After waiting `backoff.init` seconds, the {log-driver}
tries to reconnect. If the attempt fails, the backoff timer is increased
exponentially up to `backoff.max`. After a successful connection, the backoff
timer is reset.

|`output.elasticsearch.backoff.max`
|`backoff_max`
|`60s`
|The maximum number of seconds to wait before attempting to connect to
{es} after a network error.

|`output.elasticsearch.bulk_max_size`
|`50`
|The maximum number of events to bulk in a single {es} bulk API index request.
Specify 0 to allow the queue to determine the batch size.

|`output.elasticsearch.compression_level`
|`0`
|The gzip compression level. Valid compression levels range from 1 (best speed)
to 9 (best compression). Specify 0 to disable compression. Higher compression
levels reduce network usage, but increase CPU usage.

|`output.elasticsearch.escape_html`
|`false`
|Whether to escape HTML in strings.

|`output.elasticsearch.headers`
|
|Custom HTTP headers to add to each request created by the {es} output. Specify
multiple header values for the same header name by separating them with a comma.

|`output.elasticsearch.loadbalance`
|`false`
|Whether to load balance when sending events to multiple hosts. The load
balancer also supports multiple workers per host (see
`output.elasticsearch.worker`.)

|`output.elasticsearch.max_retries`
|`3`
|The number of times to retry publishing an event after a publishing failure.
After the specified number of retries, the events are typically dropped. Specify
0 to retry indefinitely.

|`output.elasticsearch.parameters`
|
| A dictionary of HTTP parameters to pass within the URL with index operations.

|`output.elasticsearch.path`
|`api_key`
|
|An HTTP path prefix that is prepended to the HTTP API calls. This is useful for
cases where {es} listens behind an HTTP reverse proxy that exports the API under
a custom prefix.
|Instead of using usernames and passwords,
you can use API keys to secure communication with {es}.

|`output.elasticsearch.pipeline`
|`pipeline`
|
|A {beats-ref}/config-file-format-type.html#_format_string_sprintf[format string]
value that specifies the {ref}/ingest.html[ingest node pipeline] to write events
to.
|A format string value that specifies the ingest node pipeline to write events to.

|`output.elasticsearch.proxy_url`
|
|The URL of the proxy to use when connecting to the {es} servers. Specify a
`URL` or `IP:PORT`.

|`output.elasticsearch.timeout`
|`timeout`
|`90`
|The HTTP request timeout in seconds for the {es} request.

|`output.elasticsearch.worker`
|`1`
|The number of workers per configured host publishing events to {es}. Use with
load balancing mode (`output.elasticsearch.loadbalance`) set to `true`. Example:
If you have 2 hosts and 3 workers, in total 6 workers are started (3 for each
host).

|=====


[float]
[[ls-output-options]]
=== {ls} output options

[options="header"]
|=====
|Option | Default | Description

|`output.logstash.hosts`
|`"localhost:5044"`
|The list of known {ls} servers to connect to. If load balancing is
disabled, but multiple hosts are configured, one host is selected randomly
(there is no precedence). If one host becomes unreachable, another one is
selected randomly. If no port is specified, the default is `5044`.

|`output.logstash.index`
|
|The index root name to write events to. For example +"dockerlogs"+ generates
+"dockerlogs-{version}"+ indices.

3+|*Advanced:*

|`output.logstash.backoff.init`
|`1s`
|The number of seconds to wait before trying to reconnect to {ls} after
a network error. After waiting `backoff.init` seconds, the {log-driver}
tries to reconnect. If the attempt fails, the backoff timer is increased
exponentially up to `backoff.max`. After a successful connection, the backoff
timer is reset.

|`output.logstash.backoff.max`
|`60s`
|The maximum number of seconds to wait before attempting to connect to
{ls} after a network error.

|`output.logstash.bulk_max_size`
|`2048`
|The maximum number of events to bulk in a single {ls} request. Specify 0 to
allow the queue to determine the batch size.

|`output.logstash.compression_level`
|`0`
|The gzip compression level. Valid compression levels range from 1 (best speed)
to 9 (best compression). Specify 0 to disable compression. Higher compression
levels reduce network usage, but increase CPU usage.

|`output.logstash.escape_html`
|`false`
|Whether to escape HTML in strings.
|The http request timeout in seconds for the Elasticsearch request.

|`output.logstash.loadbalance`
|`false`
|Whether to load balance when sending events to multiple {ls} hosts. If set to
`false`, the driver sends all events to only one host (determined at random) and
switches to another host if the selected one becomes unresponsive.

|`output.logstash.pipelining`
|`2`
|The number of batches to send asynchronously to {ls} while waiting for an ACK
from {ls}. Specify 0 to disable pipelining.

|`output.logstash.proxy_url`
|`proxy_url`
|
|The URL of the SOCKS5 proxy to use when connecting to the {ls} servers. The
value must be a URL with a scheme of `socks5://`. You can embed a
username and password in the URL (for example,
`socks5://user:password@socks5-proxy:2233`).

|`output.logstash.proxy_use_local_resolver`
|`false`
|Whether to resolve {ls} hostnames locally when using a proxy. If `false`,
name resolution occurs on the proxy server.
|The URL of the proxy to use when connecting to the Elasticsearch servers. The
value may be either a complete URL or a `host[:port]``, in which case the `http`
fearful-symmetry marked this conversation as resolved.
Show resolved Hide resolved
scheme is assumed. If a value is not specified through the configuration file
then proxy environment variables are used. See the
https://golang.org/pkg/net/http/#ProxyFromEnvironment[Go documentation]
for more information about the environment variables.

|`output.logstash.slow_start`
|`false`
|When enabled, only a subset of events in a batch are transferred per
transaction. If there are no errors, the number of events per transaction
is increased up to the bulk max size (see `output.logstash.bulk_max_size`).
On error, the number of events per transaction is reduced again.

|`output.logstash.timeout`
|`30`
|The number of seconds to wait for responses from the {ls} server before
timing out.

|`output.logstash.ttl`
|`0`
|Time to live for a connection to {ls} after which the connection will be
re-established. Useful when {ls} hosts represent load balancers. Because
connections to {ls} hosts are sticky, operating behind load balancers can lead
to uneven load distribution across instances. Specify a TTL on the connection
to distribute connections across instances. Specify 0 to disable this feature.
This option is not supported if `output.logstash.pipelining` is set.

|`output.logstash.worker`
|`1`
|The number of workers per configured host publishing events to {ls}. Use with
load balancing mode (`output.logstash.loadbalance`) set to `true`. Example:
If you have 2 hosts and 3 workers, in total 6 workers are started (3 for each
host).

|=====

[float]
[[kafka-output-options]]
=== Kafka output options

// TODO: Add kafka output options here.

// NOTE: The following annotation renders as: "Coming in a future update. This
// documentation is a work in progress."

coming[a future update. This documentation is a work in progress]

Need the docs now? See the
{filebeat-ref}/kafka-output.html[Kafka output docs] for {filebeat}.
The {log-driver} supports most of the same options, just make sure you use
the fully qualified setting names.

[float]
[[redis-output-options]]
=== Redis output options

// TODO: Add Redis output options here.

coming[a future update. This documentation is a work in progress]

Need the docs now? See the
{filebeat-ref}/redis-output.html[Redis output docs] for {filebeat}.
The {log-driver} supports most of the same options, just make sure you use
the fully qualified setting names.
14 changes: 6 additions & 8 deletions x-pack/dockerlogbeat/docs/install.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,10 +81,9 @@ example:
["source","sh",subs="attributes"]
----
docker run --log-driver=elastic/{log-driver-alias}:{version} \
--log-opt output.elasticsearch.hosts="https://myhost:9200" \
--log-opt output.elasticsearch.username="myusername" \
--log-opt output.elasticsearch.password="mypassword" \
--log-opt output.elasticsearch.index="elastic-log-driver-%{+yyyy.MM.dd}" \
--log-opt endpoint="https://myhost:9200" \
--log-opt user="myusername" \
--log-opt password="mypassword" \
-it debian:jessie /bin/bash
----
// end::log-driver-run[]
Expand All @@ -100,10 +99,9 @@ example:
{
"log-driver" : "elastic/{log-driver-alias}:{version}",
"log-opts" : {
"output.elasticsearch.hosts" : "https://myhost:9200",
"output.elasticsearch.username" : "myusername",
"output.elasticsearch.password" : "mypassword",
"output.elasticsearch.index" : "elastic-log-driver-%{+yyyy.MM.dd}"
"endpoint" : "https://myhost:9200",
"user" : "myusername",
"password" : "mypassword"
}
}
----
Expand Down
1 change: 0 additions & 1 deletion x-pack/dockerlogbeat/docs/limitations.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,5 @@ This release of the {log-driver} has the following known problems and
limitations:

* Spool to disk (beta) is not supported.
* Complex config options can't be easily represented via `--log-opts`.
* Mapping templates and other assets that are normally installed by the
{beats} setup are not available.
Loading