Skip to content

Commit

Permalink
[Elastic Log Driver] Create a config shim between libbeat and the user (
Browse files Browse the repository at this point in the history 
#18605)

* init commit of config shim

* update docs

* make check

* add timeout

* move config system to use typeconv

* add rest of backoff settings.

* make fmt

* some cleanup

* use uint64 hash for structs

* make fmt

* create custom index manager, remove ILM support

* add support for multiple endpoints

* update tests

* update docs

* remove setup options

* remove old tests

* try to update asciidocm change 'endpoint' to 'hosts'

* trying to fix CI

* update docs

* fix backtics
  • Loading branch information
@fearful-symmetry
fearful-symmetry authored May 29, 2020
1 parent 600998b commit 73150c2
Show file tree
Hide file tree
Showing 12 changed files with 264 additions and 269 deletions.
227 changes: 22 additions & 205 deletions x-pack/dockerlogbeat/docs/configuration.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,6 @@ you can set them in the `daemon.json` file for all containers.

* <<cloud-options>>
* <<es-output-options>>
* <<ls-output-options>>
* <<kafka-output-options>>
* <<redis-output-options>>

[float]
=== Usage examples
Expand All @@ -39,11 +36,11 @@ For more examples, see <<log-driver-usage-examples>>.
|=====
|Option | Description

|`cloud.id`
|`cloud_id`
|The Cloud ID found in the Elastic Cloud web console. This ID is
used to resolve the {stack} URLs when connecting to {ess} on {ecloud}.

|`cloud.auth`
|`cloud_auth`
|The username and password combination for connecting to {ess} on {ecloud}. The
format is `"username:password"`.
|=====
Expand All @@ -61,242 +58,62 @@ format is `"username:password"`.
|=====
|Option |Default |Description

|`output.elasticsearch.hosts`
|`hosts`
|`"localhost:9200"`
|The list of {es} nodes to connect to. Specify each node as a `URL` or
`IP:PORT`. For example: `http://192.0.2.0`, `https://myhost:9230` or
`192.0.2.0:9300`. If no port is specified, the default is `9200`.

|`output.elasticsearch.protocol`
|`http`
|The protocol (`http` or `https`) that {es} is reachable on. If you specify a
URL for `hosts`, the value of `protocol` is overridden by whatever scheme you
specify in the URL.

|`output.elasticsearch.username`
|`user`
|
|The basic authentication username for connecting to {es}.

|`output.elasticsearch.password`
|`password`
|
|The basic authentication password for connecting to {es}.

|`output.elasticsearch.index`
|`index`
|
|A {beats-ref}/config-file-format-type.html#_format_string_sprintf[format string]
value that specifies the index to write events to when you're using daily
indices. For example: +"dockerlogs-%{+yyyy.MM.dd}"+.

3+|*Advanced:*

|`output.elasticsearch.backoff.init`
|`backoff_init`
|`1s`
|The number of seconds to wait before trying to reconnect to {es} after
a network error. After waiting `backoff.init` seconds, the {log-driver}
tries to reconnect. If the attempt fails, the backoff timer is increased
exponentially up to `backoff.max`. After a successful connection, the backoff
timer is reset.

|`output.elasticsearch.backoff.max`
|`backoff_max`
|`60s`
|The maximum number of seconds to wait before attempting to connect to
{es} after a network error.

|`output.elasticsearch.bulk_max_size`
|`50`
|The maximum number of events to bulk in a single {es} bulk API index request.
Specify 0 to allow the queue to determine the batch size.

|`output.elasticsearch.compression_level`
|`0`
|The gzip compression level. Valid compression levels range from 1 (best speed)
to 9 (best compression). Specify 0 to disable compression. Higher compression
levels reduce network usage, but increase CPU usage.

|`output.elasticsearch.escape_html`
|`false`
|Whether to escape HTML in strings.

|`output.elasticsearch.headers`
|
|Custom HTTP headers to add to each request created by the {es} output. Specify
multiple header values for the same header name by separating them with a comma.

|`output.elasticsearch.loadbalance`
|`false`
|Whether to load balance when sending events to multiple hosts. The load
balancer also supports multiple workers per host (see
`output.elasticsearch.worker`.)

|`output.elasticsearch.max_retries`
|`3`
|The number of times to retry publishing an event after a publishing failure.
After the specified number of retries, the events are typically dropped. Specify
0 to retry indefinitely.

|`output.elasticsearch.parameters`
|
| A dictionary of HTTP parameters to pass within the URL with index operations.

|`output.elasticsearch.path`
|`api_key`
|
|An HTTP path prefix that is prepended to the HTTP API calls. This is useful for
cases where {es} listens behind an HTTP reverse proxy that exports the API under
a custom prefix.
|Instead of using usernames and passwords,
you can use API keys to secure communication with {es}.

|`output.elasticsearch.pipeline`
|`pipeline`
|
|A {beats-ref}/config-file-format-type.html#_format_string_sprintf[format string]
value that specifies the {ref}/ingest.html[ingest node pipeline] to write events
to.
|A format string value that specifies the ingest node pipeline to write events to.

|`output.elasticsearch.proxy_url`
|
|The URL of the proxy to use when connecting to the {es} servers. Specify a
`URL` or `IP:PORT`.

|`output.elasticsearch.timeout`
|`timeout`
|`90`
|The HTTP request timeout in seconds for the {es} request.

|`output.elasticsearch.worker`
|`1`
|The number of workers per configured host publishing events to {es}. Use with
load balancing mode (`output.elasticsearch.loadbalance`) set to `true`. Example:
If you have 2 hosts and 3 workers, in total 6 workers are started (3 for each
host).

|=====


[float]
[[ls-output-options]]
=== {ls} output options

[options="header"]
|=====
|Option | Default | Description

|`output.logstash.hosts`
|`"localhost:5044"`
|The list of known {ls} servers to connect to. If load balancing is
disabled, but multiple hosts are configured, one host is selected randomly
(there is no precedence). If one host becomes unreachable, another one is
selected randomly. If no port is specified, the default is `5044`.

|`output.logstash.index`
|
|The index root name to write events to. For example +"dockerlogs"+ generates
+"dockerlogs-{version}"+ indices.

3+|*Advanced:*

|`output.logstash.backoff.init`
|`1s`
|The number of seconds to wait before trying to reconnect to {ls} after
a network error. After waiting `backoff.init` seconds, the {log-driver}
tries to reconnect. If the attempt fails, the backoff timer is increased
exponentially up to `backoff.max`. After a successful connection, the backoff
timer is reset.

|`output.logstash.backoff.max`
|`60s`
|The maximum number of seconds to wait before attempting to connect to
{ls} after a network error.

|`output.logstash.bulk_max_size`
|`2048`
|The maximum number of events to bulk in a single {ls} request. Specify 0 to
allow the queue to determine the batch size.

|`output.logstash.compression_level`
|`0`
|The gzip compression level. Valid compression levels range from 1 (best speed)
to 9 (best compression). Specify 0 to disable compression. Higher compression
levels reduce network usage, but increase CPU usage.

|`output.logstash.escape_html`
|`false`
|Whether to escape HTML in strings.
|The http request timeout in seconds for the Elasticsearch request.

|`output.logstash.loadbalance`
|`false`
|Whether to load balance when sending events to multiple {ls} hosts. If set to
`false`, the driver sends all events to only one host (determined at random) and
switches to another host if the selected one becomes unresponsive.

|`output.logstash.pipelining`
|`2`
|The number of batches to send asynchronously to {ls} while waiting for an ACK
from {ls}. Specify 0 to disable pipelining.

|`output.logstash.proxy_url`
|`proxy_url`
|
|The URL of the SOCKS5 proxy to use when connecting to the {ls} servers. The
value must be a URL with a scheme of `socks5://`. You can embed a
username and password in the URL (for example,
`socks5://user:password@socks5-proxy:2233`).

|`output.logstash.proxy_use_local_resolver`
|`false`
|Whether to resolve {ls} hostnames locally when using a proxy. If `false`,
name resolution occurs on the proxy server.
|The URL of the proxy to use when connecting to the Elasticsearch servers. The
value may be either a complete URL or a `host[:port]`, in which case the `http`
scheme is assumed. If a value is not specified through the configuration file
then proxy environment variables are used. See the
https://golang.org/pkg/net/http/#ProxyFromEnvironment[Go documentation]
for more information about the environment variables.

|`output.logstash.slow_start`
|`false`
|When enabled, only a subset of events in a batch are transferred per
transaction. If there are no errors, the number of events per transaction
is increased up to the bulk max size (see `output.logstash.bulk_max_size`).
On error, the number of events per transaction is reduced again.

|`output.logstash.timeout`
|`30`
|The number of seconds to wait for responses from the {ls} server before
timing out.

|`output.logstash.ttl`
|`0`
|Time to live for a connection to {ls} after which the connection will be
re-established. Useful when {ls} hosts represent load balancers. Because
connections to {ls} hosts are sticky, operating behind load balancers can lead
to uneven load distribution across instances. Specify a TTL on the connection
to distribute connections across instances. Specify 0 to disable this feature.
This option is not supported if `output.logstash.pipelining` is set.

|`output.logstash.worker`
|`1`
|The number of workers per configured host publishing events to {ls}. Use with
load balancing mode (`output.logstash.loadbalance`) set to `true`. Example:
If you have 2 hosts and 3 workers, in total 6 workers are started (3 for each
host).

|=====

[float]
[[kafka-output-options]]
=== Kafka output options

// TODO: Add kafka output options here.

// NOTE: The following annotation renders as: "Coming in a future update. This
// documentation is a work in progress."

coming[a future update. This documentation is a work in progress]

Need the docs now? See the
{filebeat-ref}/kafka-output.html[Kafka output docs] for {filebeat}.
The {log-driver} supports most of the same options, just make sure you use
the fully qualified setting names.

[float]
[[redis-output-options]]
=== Redis output options

// TODO: Add Redis output options here.

coming[a future update. This documentation is a work in progress]

Need the docs now? See the
{filebeat-ref}/redis-output.html[Redis output docs] for {filebeat}.
The {log-driver} supports most of the same options, just make sure you use
the fully qualified setting names.
14 changes: 6 additions & 8 deletions x-pack/dockerlogbeat/docs/install.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -81,10 +81,9 @@ example:
["source","sh",subs="attributes"]
----
docker run --log-driver=elastic/{log-driver-alias}:{version} \
--log-opt output.elasticsearch.hosts="https://myhost:9200" \
--log-opt output.elasticsearch.username="myusername" \
--log-opt output.elasticsearch.password="mypassword" \
--log-opt output.elasticsearch.index="elastic-log-driver-%{+yyyy.MM.dd}" \
--log-opt endpoint="https://myhost:9200" \
--log-opt user="myusername" \
--log-opt password="mypassword" \
-it debian:jessie /bin/bash
----
// end::log-driver-run[]
Expand All @@ -100,10 +99,9 @@ example:
{
"log-driver" : "elastic/{log-driver-alias}:{version}",
"log-opts" : {
"output.elasticsearch.hosts" : "https://myhost:9200",
"output.elasticsearch.username" : "myusername",
"output.elasticsearch.password" : "mypassword",
"output.elasticsearch.index" : "elastic-log-driver-%{+yyyy.MM.dd}"
"endpoint" : "https://myhost:9200",
"user" : "myusername",
"password" : "mypassword"
}
}
----
Expand Down
1 change: 0 additions & 1 deletion x-pack/dockerlogbeat/docs/limitations.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,5 @@ This release of the {log-driver} has the following known problems and
limitations:

* Spool to disk (beta) is not supported.
* Complex config options can't be easily represented via `--log-opts`.
* Mapping templates and other assets that are normally installed by the
{beats} setup are not available.
Loading

0 comments on commit 73150c2

Please sign in to comment.