Skip to content

Commit

Permalink
Allow users to select the cloud metadata providers (elastic#13812)
Browse files Browse the repository at this point in the history
* Allow users to select the cloud metadata providers

We introduce a new setting 'providers' to the add_cloud_metadata
processor.

By now all the implementation for metadata providers requires developers
to mark a provider as 'local'. The alibaba and tencent providers are not
marked as local by now.

If the 'providers' setting is not used, then no all providers marked as
'local' are applied. This is a breaking change, because alibaba and
tencent providers will not be enabled anymore by default.

If the providers setting is used, only the selected providers will be
used.
  • Loading branch information
Steffen Siering authored Sep 26, 2019
1 parent dddd34d commit fa6d344
Show file tree
Hide file tree
Showing 16 changed files with 786 additions and 474 deletions.
2 changes: 2 additions & 0 deletions CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Fixed a crash under Windows when fetching processes information. {pull}12833[12833]
- Update to Golang 1.12.7. {pull}12931[12931]
- Remove `in_cluster` configuration parameter for Kuberentes, now in-cluster configuration is used only if no other kubeconfig is specified {pull}13051[13051]
- Disable Alibaba Cloud and Tencent Cloud metadata providers by default. {pull}13812[12812]

*Auditbeat*

Expand Down Expand Up @@ -257,6 +258,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Add autodetection mode for add_docker_metadata and enable it by default in included configuration files{pull}13374[13374]
- Added `monitoring.cluster_uuid` setting to associate Beat data with specified ES cluster in Stack Monitoring UI. {pull}13182[13182]
- Add autodetection mode for add_kubernetes_metadata and enable it by default in included configuration files. {pull}13473[13473]
- Add `providers` setting to `add_cloud_metadata` processor. {pull}13812[13812]

*Auditbeat*

Expand Down
25 changes: 21 additions & 4 deletions libbeat/docs/processors-using.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -505,8 +505,8 @@ not:
=== Add cloud metadata

The `add_cloud_metadata` processor enriches each event with instance metadata
from the machine's hosting provider. At startup it will detect the hosting
provider and cache the instance metadata.
from the machine's hosting provider. At startup it will query a list of hosting
providers and cache the instance metadata.

The following cloud providers are supported:

Expand All @@ -518,6 +518,10 @@ The following cloud providers are supported:
- Azure Virtual Machine
- Openstack Nova

The Alibaba Cloud and Tencent cloud providers are disabled by default, because
they require to access a remote host. The `providers` setting allows users to
select a list of default providers to query.

The simple configuration below enables the processor.

[source,yaml]
Expand All @@ -526,7 +530,7 @@ processors:
- add_cloud_metadata: ~
-------------------------------------------------------------------------------

The `add_cloud_metadata` processor has two optional configuration settings.
The `add_cloud_metadata` processor has three optional configuration settings.
The first one is `timeout` which specifies the maximum amount of time to wait
for a successful response when detecting the hosting provider. The default
timeout value is `3s`.
Expand All @@ -535,7 +539,20 @@ If a timeout occurs then no instance metadata will be added to the events. This
makes it possible to enable this processor for all your deployments (in the
cloud or on-premise).

The second optional configuration setting is `overwrite`. When `overwrite` is
The second optional setting is `providers`. The `providers` settings accepts a
list of cloud provider names to be used. If `providers` is not configured, then
all providers that do not access a remote endpoint are enabled by default.

List of names the `providers` setting supports:
- "alibaba", or "ecs" for the Alibaba Cloud provider (disabled by default).
- "azure" for Azure Virtual Machine (enabled by default).
- "digitalocean" for Digital Ocean (enabled by default).
- "aws", or "ec2" for Amazon Web Services (enabled by default).
- "gcp" for Google Copmute Enging (enabled by default).
- "openstack", or "nova" for Openstack Nova (enabled by default).
- "tencent", or "qcloud" for Tencent Cloud (disabled by default).

The third optional configuration setting is `overwrite`. When `overwrite` is
`true`, `add_cloud_metadata` overwrites existing `cloud.*` fields (`false` by
default).

Expand Down
Loading

0 comments on commit fa6d344

Please sign in to comment.