Skip to content

Commit

Permalink
Added YAML format exp. Deleted individual examples
Browse files Browse the repository at this point in the history
  • Loading branch information
@karenzone
karenzone committed Jul 18, 2018
1 parent 31a77d2 commit 83488ff
Showing 1 changed file with 15 additions and 123 deletions.
138 changes: 15 additions & 123 deletions docs/static/azure-module.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ suite of {kib} dashboards to help you start exploring your data immediately.
[[azure-dashboards]]
==== Dashboards

These {kib} dashboards are available and ready for you to use. You can use the dashboards they are, or tailor them to meet your needs.
These {kib} dashboards are available and ready for you to use. You can use them they are, or tailor them to meet your needs.

===== Infrastructure activity monitoring

Expand All @@ -50,17 +50,13 @@ These {kib} dashboards are available and ready for you to use. You can use the d

* *SQL DB Queries*. Info about SQL database queries, including DTU Utilization, errors, and query duration and wait time.




==== Azure_event_hubs plugin

The Azure module uses the `azure_event_hubs` plugin. Basic understanding of the
plugin and options is helpful when you set up the Azure module. See
{logstash-ref}/plugins-inputs-azure_event_hubs.html[azure_event_hubs plugin
documentation] for more information about configurations and options.


[[azure-module-prereqs]]
==== Prerequisites

Expand Down Expand Up @@ -192,6 +188,20 @@ modules:
- ["insights-operational-logs", "Endpoint=sb://example4...", "DefaultEndpointsProtocol=https;AccountName=example4....", "HEAD", "true"]
-----

===== Map input options to the logstash.yml file

The configurations in the `logstash.yml` file must be in YAML format.

Example:

["source","shell",subs="attributes"]
-----
var.input.azure_event_hubs.event_hubs:
- ["name", "event_hub_connection", "storage_connection", "initial_position", "decorate_events"]
- ["insights-operational-logs", "Endpoint=sb://example1...", "DefaultEndpointsProtocol=https;AccountName=example1....", "HEAD", "true"]
- ["insights-metrics-pt1m", "Endpoint=sb://example2...", "DefaultEndpointsProtocol=https;AccountName=example2....", "TAIL", "true"]
-----

[[azure_config_options]]
===== Configuration options

Expand All @@ -210,23 +220,6 @@ configurations, with the following exceptions. The basic configuration uses
Defines the Event Hubs to be read. An array of hashes where each entry is a
hash of the Event Hub name and its configuration options.

[source,ruby]
----
azure_event_hubs {
config_mode => "advanced"
event_hubs => [
{ "event_hub_name1" => {
event_hub_connection => "Endpoint=sb://example1..."
}},
{ "event_hub_name2" => {
event_hub_connection => "Endpoint=sb://example2..."
storage_connection => "DefaultEndpointsProtocol=https;AccountName=example...."
storage_container => "my_container"
}}
]
consumer_group => "logstash" # shared across all Event Hubs
}
----

[id="plugins-{type}s-{plugin}-event_hub_connections"]
===== `event_hub_connections`
Expand All @@ -240,12 +233,6 @@ strings include the EntityPath for the Event Hub.
The `event_hub_connections` option is defined
per Event Hub. All other configuration options are shared among Event Hubs.

[source,ruby]
----
azure_event_hubs {
event_hub_connections => ["Endpoint=sb://example1...;EntityPath=event_hub_name1" , "Endpoint=sb://example2...;EntityPath=event_hub_name2" ]
}
----

[id="plugins-{type}s-{plugin}-event_hub_connection"]
===== `event_hub_connection`
Expand All @@ -258,17 +245,6 @@ configuration options can be set per Event Hub. This option modifies
`event_hub_name`, and should be nested under it. (See sample.) This option
accepts only one connection string.

[source,ruby]
----
azure_event_hubs {
config_mode => "advanced"
event_hubs => [
{ "event_hub_name1" => {
event_hub_connection => "Endpoint=sb://example1...;EntityPath=event_hub_name1"
}}
]
}
----

[id="plugins-{type}s-{plugin}-checkpoint_interval"]
===== `checkpoint_interval`
Expand All @@ -282,13 +258,6 @@ automatically written at the end of each batch, regardless of this setting.

Writing checkpoints too frequently can slow down processing unnecessarily.

[source,ruby]
----
azure_event_hubs {
event_hub_connections => ["Endpoint=sb://example1...;EntityPath=event_hub_name1"]
checkpoint_interval => 5
}
----

[id="plugins-{type}s-{plugin}-consumer_group"]
===== `consumer_group`
Expand All @@ -299,13 +268,6 @@ Consumer group used to read the Event Hub(s). Create a consumer group
specifically for Logstash. Then ensure that all instances of Logstash use that
consumer group so that they can work together properly.

[source,ruby]
----
azure_event_hubs {
event_hub_connections => ["Endpoint=sb://example1...;EntityPath=event_hub_name1"]
consumer_group => "logstash"
}
----

[id="plugins-{type}s-{plugin}-decorate_events"]
===== `decorate_events`
Expand All @@ -316,13 +278,6 @@ azure_event_hubs {
Adds metadata about the Event Hub, including Event Hub name, consumer_group,
processor_host, partition, offset, sequence, timestamp, and event_size.

[source,ruby]
----
azure_event_hubs {
event_hub_connections => ["Endpoint=sb://example1...;EntityPath=event_hub_name1"]
decorate_events => true
}
----

[id="plugins-{type}s-{plugin}-initial_position"]
===== `initial_position`
Expand All @@ -340,13 +295,6 @@ You control the number of seconds using the `initial_position_look_back` option.
Note: If `storage_connection` is set, the `initial_position` value is used only
the first time Logstash reads from the Event Hub.

[source,ruby]
----
azure_event_hubs {
event_hub_connections => ["Endpoint=sb://example1...;EntityPath=event_hub_name1"]
initial_position => "beginning"
}
----

[id="plugins-{type}s-{plugin}-initial_position_look_back"]
===== `initial_position_look_back`
Expand All @@ -359,14 +307,6 @@ events. This option is used only if `initial_position` is set to `look_back`. If
`storage_connection` is set, this configuration applies only the first time {ls}
reads from the Event Hub.

[source,ruby]
----
azure_event_hubs {
event_hub_connections => ["Endpoint=sb://example1...;EntityPath=event_hub_name1"]
initial_position => "look_back"
initial_position_look_back => 86400
}
----

[id="plugins-{type}s-{plugin}-max_batch_size"]
===== `max_batch_size`
Expand All @@ -378,13 +318,6 @@ Maximum number of events retrieved and processed together. A checkpoint is
created after each batch. Increasing this value may help with performance, but
requires more memory.

[source,ruby]
----
azure_event_hubs {
event_hub_connections => ["Endpoint=sb://example1...;EntityPath=event_hub_name1"]
max_batch_size => 125
}
----

[id="plugins-{type}s-{plugin}-storage_connection"]
===== `storage_connection`
Expand All @@ -400,14 +333,6 @@ When this value is not set, the `initial_position` value is used on every restar
We strongly recommend that you define this value for production environments.


[source,ruby]
----
azure_event_hubs {
event_hub_connections => ["Endpoint=sb://example1...;EntityPath=event_hub_name1"]
storage_connection => "DefaultEndpointsProtocol=https;AccountName=example...."
}
----

[id="plugins-{type}s-{plugin}-storage_container"]
===== `storage_container`
* Value type is <<string,string>>
Expand All @@ -416,38 +341,11 @@ azure_event_hubs {
Name of the storage container used to persist offsets and allow multiple instances of {ls}
to work together.

[source,ruby]
----
azure_event_hubs {
event_hub_connections => ["Endpoint=sb://example1...;EntityPath=event_hub_name1"]
storage_connection => "DefaultEndpointsProtocol=https;AccountName=example...."
storage_container => "my_container"
}
----

To avoid overwriting offsets, you can use different storage containers. This is
particularly important if you are monitoring two Event Hubs with the same name.
You can use the advanced configuration model to configure different storage
containers.

[source,ruby]
----
azure_event_hubs {
config_mode => "advanced"
consumer_group => "logstash"
storage_connection => "DefaultEndpointsProtocol=https;AccountName=example...."
event_hubs => [
{"insights-operational-logs" => {
event_hub_connection => "Endpoint=sb://example1..."
storage_container => "insights-operational-logs-1"
}},
{"insights-operational-logs" => {
event_hub_connection => "Endpoint=sb://example2..."
storage_container => "insights-operational-logs-2"
}}
]
}
----

[id="plugins-{type}s-{plugin}-threads"]
===== `threads`
Expand All @@ -459,12 +357,6 @@ Total number of threads used to process events. The value you set here applies
to all Event Hubs. Even with advanced configuration, this value is a global
setting, and can't be set per event hub.

[source,ruby]
----
azure_event_hubs {
threads => 4
}
----

include::shared-module-options.asciidoc[]

Expand Down

0 comments on commit 83488ff

Please sign in to comment.