diff --git a/docs/index-shared3.asciidoc b/docs/index-shared3.asciidoc index 9454487961b..137abf6f43e 100644 --- a/docs/index-shared3.asciidoc +++ b/docs/index-shared3.asciidoc @@ -44,6 +44,9 @@ include::static/arcsight-module.asciidoc[] :edit_url: https://github.com/elastic/logstash/edit/{branch}/docs/static/netflow-module.asciidoc include::static/netflow-module.asciidoc[] +:edit_url: https://github.com/elastic/logstash/edit/{branch}/docs/static/azure-module.asciidoc +include::static/azure-module.asciidoc[] + // Working with Filebeat Modules :edit_url: https://github.com/elastic/logstash/edit/{branch}/docs/static/filebeat-modules.asciidoc diff --git a/docs/static/azure-module.asciidoc b/docs/static/azure-module.asciidoc new file mode 100644 index 00000000000..54ae6a574ae --- /dev/null +++ b/docs/static/azure-module.asciidoc @@ -0,0 +1,285 @@ +[role="xpack"] +[[azure-module]] +=== Azure Module + +++++ +Azure Module +++++ + +:username: username +:hostname: hostname +:event_hub_name: event_hub_name +:event_hub_key: event_hub_key +:event_hub_username: event_hub_username +:event_hub_namespace: event_hub_namespace +:partitions: partitions + +The Microsoft Azure module in Logstash helps you easily integrate your Azure +activity logs and SQL diagnostic logs with the Elastic Stack. The module taps directly into the Azure dashboard, +parses and indexes events into Elasticsearch, and installs a suite of Kibana +dashboards to help you start exploring your data immediately. + +TBD: Add links to MS def of activity logs and SQL diagnostics logs + +NOTE: The Logstash Azure module is an +https://www.elastic.co/products/x-pack[{xpack}] feature under the Basic License +and is therefore free to use. Please contact +mailto:monitor-azure@elastic.co[monitor-azure@elastic.co] for questions or more +information. + +These instructions are designed to help you set up and demo the Azure module in your environment. For a production environment, additional security steps are recommended. + +[[azure-dashboards]] +==== Dashboards + +These dashboards are available and ready for you to use. + +* *Overview*. A top-level view into your Azure operations, including info about users, resource groups, service health, access, activities, and alerts. + +* *Alerts*. Alert info, including activity, alert status (activated, resolved, succeeded), and alerts heatmap + +* *SQL DB Overview*. A top-level view into your SQL databases, including counts for databases, servers, resource groups, and subscriptions. + +* *SQL DB Database View*. Detailed info about each SQL database, including wait time, errors, DTU and storage utilization, size, and read and write input/output. + +* *SQL DB Queries*. Info about SQL database queries, including DTU Utilization, errors, and query duration and wait time. + +* *User Activity*. Info about system users, their activity, and requests. + +You can use the dashboards they are, or tailor them to meet your needs. + +[[azure-prereqs]] +==== Prerequisites + +These instructions assume that Logstash, Elasticsearch, and Kibana are already +installed and running. The products are +https://www.elastic.co/downloads[available to download] and easy to install. + +The Elastic Stack 6.3 (or later) is required for this module. + +NOTE: Logstash, Elasticsearch, and Kibana must run locally. You can also run +Elasticsearch, Kibana and Logstash on separate hosts to consume data from Azure. + +[[azure-setup]] +==== Installation and setup + +To get started with the Azure module: + + . Install the `azureeventhub` plugin. + . Set up the Azure module. + +[[azure-plugin-setup]] +===== Install the plugin + +TBD: From the LS directory? +To install the Azure plugin, run this command: + +["source","shell",subs="attributes"] +----- +bin/logstash-plugin install logstash-input-azureeventhub +----- + +[[azure-module-setup]] +===== Set up the module + +TBD: Check variables, formatting, etc. +TBD: From the LS directory? + +To set up the module, run this command: + +["source","shell",subs="attributes"] +----- +bin/logstash --modules azure --setup \ + -M "azure.var.elasticsearch.username={username}" \ + -M "azure.var.elasticsearch.password={pwd}" \ + -M "azure.var.kibana.username={username}" \ + -M "azure.var.kibana.password={pwd}" \ + -M "azure.var.elasticsearch.hosts={hostname}" \ + -M "azure.var.kibana.host={hostname}" +----- + +TBD: I added the backslashes. Is that correct? + +The `--modules azure` option starts a Logstash pipeline for ingestion into +Azure. The `--setup` option creates an `azure-*` index pattern in Elasticsearch +and imports Kibana dashboards and visualizations. + +NOTE: The `--setup` option is intended only for first-time setup. If you include `setup` on subsequent runs, your existing Kibana dashboards will be overwritten. + + +[[azure-settings]] +==== Add settings + +Add these settings to the logstash.yml file. + +["source","shell",subs="attributes"] +----- +modules: + - name: azure + var.input.azureeventhub.eventhub: "event_hub_name" + var.input.azureeventhub.key: "event_hub_key" + var.input.azureeventhub.username: "event_hub_username" + var.input.azureeventhub.namespace: "event_hub_namespace" + var.input.azureeventhub.partitions: "partitions" + var.elasticsearch.hosts: "localhost:9200" + var.elasticsearch.username: elastic" + var.elasticsearch.password: "{pwd}" + var.kibana.host: “localhost:5601” + var.kibana.username: "elastic" + var.kibana.password: "{pwd}" +----- + +If you want to specify additional options to control the behavior of the Azure +module, see <>. + +[[run-azure]] +==== Start the module + +Run this command from the Logstash install directory: + +["source","shell",subs="attributes"] +----- +bin/logstash +----- + +[[exploring-data-azure]] +==== Explore your data +When the Logstash Azure module starts receiving events, you can begin using the +packaged Kibana dashboards to explore and visualize your data. + +To explore your data with Kibana: + +. Open a browser to http://localhost:5601[http://localhost:5601] (username: + "elastic"; password: "{pwd}") +. Click *Dashboard*. +. Select the dashboard you want to see. + + +[[configuring-azure]] +==== Configure the Module + +You can specify additional options for the Logstash Azure module in the +`logstash.yml` configuration file or with overrides through the command line. For more information about configuring modules, see +<>. + +Configure these values in the `logstash.yml` file. + +["source","yaml",subs="attributes"] +----- +modules: + - name: azure + var.input.azureeventhub.eventhub: "event_hub_name" + var.input.azureeventhub.key: "event_hub_key" + var.input.azureeventhub.username: "event_hub_username" + var.input.azureeventhub.namespace: "event_hub_namespace" + var.input.azureeventhub.partitions: "partitions" + var.elasticsearch.hosts: "localhost:9200" + var.elasticsearch.username: "elastic" + var.elasticsearch.password: "{pwd}" + var.kibana.host: “localhost:5601” + var.kibana.username: "elastic" + var.kibana.password: "{pwd}" +----- +TBD: Verify values and formatting for variables in previous. + +[[azure-config-options]] +===== Configuration options + +The Azure module provides settings for configuring its behavior. These settings +include Azure-specific options and common options that are supported by all +Logstash modules. + +If you override a setting at the command line, remember to prefix the +setting with the module name. For example, use `azure.var.inputs` instead of +`var.inputs`. + +If you don't specify configuration settings, Logstash uses the defaults. + +TBD-check singular or plural on var.inputs. Apply globally as appropriate. +TBD-Discuss. Should we repeat options here or only reference https://github.com/Azure/azure-diagnostics-tools/tree/master/Logstash/logstash-input-azureeventhub + +*Azure Module Options* + +All `var.input.azureeventhub.*` options are documented in the https://github.com/Azure/azure-diagnostics-tools/tree/master/Logstash/logstash-input-azureeventhub[Event Hub plugin]. + +*`var.inputs`*:: ++ +-- +* Should prev be `var.input`* or is plural correct? Verify the default. I guessed. +* Value type is <> +* Default value is "azureeventhub" +-- ++ +-- +Set the input(s) to expose for the Logstash Azure module. Valid settings are +"TBD". +-- + +*`var.input.azureeventhub.eventhub`*:: ++ +-- +* Value type is <> +* Default value is "localhost:39092" +-- ++ +-- +Event hub name. +-- + +*`var.input.azureeventhub.key`*:: ++ +-- +* Value type is +* Default value is +-- ++ +-- +TBD: Add description +-- +*`var.input.azureeventhub.username`*:: ++ +-- +* Value type is +* Default value is +-- ++ +Name of the shared access policy. + +*`var.input.azureeventhub.namespace`*:: ++ +-- +* Value type is +* Default value is +-- ++ +TBD: Add description + +*`var.input.azureeventhub.partitions`*:: ++ +-- +* Value type is +* Default value is +-- ++ +Partition count of the target hub. + +TBD: Look at list of shared module options. Doc implies that all are available +for every module. Is that true? + +include::shared-module-options.asciidoc[] + +[[azure-production]] +==== Deploying the module in production + +TBD: Can we break demo and deployment out in this way? + +Use SSL security. + + +:username!: +:hostname!: +:event_hub_name!: +:event_hub_key!: +:event_hub_username!: +:event_hub_namespace!: +:partitions!: \ No newline at end of file