Skip to content

Commit

Permalink
feat: Add Dynatrace destination
Browse files Browse the repository at this point in the history
  • Loading branch information
fdmsantos committed Oct 17, 2022
1 parent 227d95e commit 378ba91
Show file tree
Hide file tree
Showing 11 changed files with 311 additions and 41 deletions.
61 changes: 44 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ Supports all destinations and all Kinesis Firehose Features.
* [Datadog](#datadog)
* [New Relic](#new-relic)
* [Coralogix](#coralogix)
* [Dynatrace](#dynatrace)
* [Server Side Encryption](#server-side-encryption)
* [Data Transformation with Lambda](#data-transformation-with-lambda)
* [Data Format Conversion](#data-format-conversion)
Expand Down Expand Up @@ -58,6 +59,7 @@ Supports all destinations and all Kinesis Firehose Features.
- DataDog
- Coralogix
- New Relic
- Dynatrace
- Data Transformation With Lambda
- Original Data Backup in S3
- Logging and Encryption
Expand Down Expand Up @@ -270,12 +272,32 @@ module "firehose" {

```hcl
module "firehose" {
source = "fdmsantos/kinesis-firehose/aws"
version = "x.x.x"
name = "firehose-delivery-stream"
destination = "coralogix"
coralogix_endpoint_type = "coralogix_ireland"
http_endpoint_access_key = "<coralogix_private_key>"
source = "fdmsantos/kinesis-firehose/aws"
version = "x.x.x"
name = "firehose-delivery-stream"
destination = "coralogix"
coralogix_endpoint_location = "ireland"
http_endpoint_access_key = "<coralogix_private_key>"
}
```

#### Dynatrace

**To Enabled It:** `destination = "dynatrace"`

**Variables Prefix:** `http_endpoint_`, `dynatrace_endpoint_location` and `dynatrace_api_url`

**Check [HTTP Endpoint](#http-endpoint) to more details and [Destinations Mapping](#destinations-mapping) to see the difference between http_endpoint and dynatrace destinations**

```hcl
module "firehose" {
source = "fdmsantos/kinesis-firehose/aws"
version = "x.x.x"
name = "firehose-delivery-stream"
destination = "dynatrace"
dynatrace_endpoint_location = "eu"
dynatrace_api_url = "https://xyazb123456.live.dynatrace.com"
http_endpoint_access_key = "<dynatrace_api_token>"
}
```

Expand Down Expand Up @@ -576,16 +598,17 @@ module "firehose" {

The destination variable configured in module is mapped to firehose valid destination.

| Module Destination | Firehose Destination | Differences |
|------------------------------|----------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| s3 and extended_s3 | extended_s3 | There is no difference between s3 or extended_s3 destinations |
| redshift | redshift | |
| splunk | splunk | |
| opensearch and elasticsearch | elasticsearch | There is no difference between opensearch or elasticsearch destinations |
| http_endpoint | http_endpoint | |
| datadog | http_endpoint | The difference regarding http_endpoint is the http_endpoint_url and http_endpoint_name variables aren't support, and it's necessary configure datadog_endpoint_type variable |
| newrelic | http_endpoint | The difference regarding http_endpoint is the http_endpoint_url and http_endpoint_name variables aren't support, and it's necessary configure newrelic_endpoint_type variable |
| coralogix | http_endpoint | The difference regarding http_endpoint is the http_endpoint_url and http_endpoint_name variables aren't support, and it's necessary configure coralogix_endpoint_type variable |
| Module Destination | Firehose Destination | Differences |
|------------------------------|----------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| s3 and extended_s3 | extended_s3 | There is no difference between s3 or extended_s3 destinations |
| redshift | redshift | |
| splunk | splunk | |
| opensearch and elasticsearch | elasticsearch | There is no difference between opensearch or elasticsearch destinations |
| http_endpoint | http_endpoint | |
| datadog | http_endpoint | The difference regarding http_endpoint is the http_endpoint_url and http_endpoint_name variables aren't support, and it's necessary configure datadog_endpoint_type variable |
| newrelic | http_endpoint | The difference regarding http_endpoint is the http_endpoint_url and http_endpoint_name variables aren't support, and it's necessary configure newrelic_endpoint_type variable |
| coralogix | http_endpoint | The difference regarding http_endpoint is the http_endpoint_url and http_endpoint_name variables aren't support, and it's necessary configure coralogix_endpoint_location variable |
| dynatrace | http_endpoint | The difference regarding http_endpoint is the http_endpoint_url and http_endpoint_name variables aren't support, and it's necessary configure dynatrace_endpoint_location and dynatrace_api_url variable |

## Examples

Expand All @@ -602,6 +625,7 @@ The destination variable configured in module is mapped to firehose valid destin
- [Datadog](https://github.com/fdmsantos/terraform-aws-kinesis-firehose/tree/main/examples/http-endpoint/datadog) - Creates a Kinesis Firehose Stream with datadog europe metrics as destination.
- [New Relic](https://github.com/fdmsantos/terraform-aws-kinesis-firehose/tree/main/examples/http-endpoint/newrelic) - Creates a Kinesis Firehose Stream with New Relic europe metrics as destination.
- [Coralogix](https://github.com/fdmsantos/terraform-aws-kinesis-firehose/tree/main/examples/http-endpoint/coralogix) - Creates a Kinesis Firehose Stream with coralogix ireland as destination.
- [Dynatrace](https://github.com/fdmsantos/terraform-aws-kinesis-firehose/tree/main/examples/http-endpoint/dynatrace) - Creates a Kinesis Firehose Stream with dynatrace europe as destination.

<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
## Requirements
Expand Down Expand Up @@ -687,7 +711,7 @@ No modules.
| <a name="input_buffer_interval"></a> [buffer\_interval](#input\_buffer\_interval) | Buffer incoming data for the specified period of time, in seconds, before delivering it to the destination | `number` | `300` | no |
| <a name="input_buffer_size"></a> [buffer\_size](#input\_buffer\_size) | Buffer incoming data to the specified size, in MBs, before delivering it to the destination. | `number` | `5` | no |
| <a name="input_configure_existing_application_role"></a> [configure\_existing\_application\_role](#input\_configure\_existing\_application\_role) | Set it to True if want use existing application role to add the firehose Policy | `bool` | `false` | no |
| <a name="input_coralogix_endpoint_type"></a> [coralogix\_endpoint\_type](#input\_coralogix\_endpoint\_type) | Endpoint type to coralogix destination | `string` | `"coralogix_ireland"` | no |
| <a name="input_coralogix_endpoint_location"></a> [coralogix\_endpoint\_location](#input\_coralogix\_endpoint\_location) | Endpoint Location to coralogix destination | `string` | `"ireland"` | no |
| <a name="input_coralogix_parameter_application_name"></a> [coralogix\_parameter\_application\_name](#input\_coralogix\_parameter\_application\_name) | By default, your delivery stream arn will be used as applicationName | `string` | `null` | no |
| <a name="input_coralogix_parameter_subsystem_name"></a> [coralogix\_parameter\_subsystem\_name](#input\_coralogix\_parameter\_subsystem\_name) | By default, your delivery stream name will be used as subsystemName | `string` | `null` | no |
| <a name="input_coralogix_parameter_use_dynamic_values"></a> [coralogix\_parameter\_use\_dynamic\_values](#input\_coralogix\_parameter\_use\_dynamic\_values) | To use dynamic values for applicationName and subsystemName | `bool` | `false` | no |
Expand Down Expand Up @@ -736,6 +760,8 @@ No modules.
| <a name="input_dynamic_partition_record_deaggregation_delimiter"></a> [dynamic\_partition\_record\_deaggregation\_delimiter](#input\_dynamic\_partition\_record\_deaggregation\_delimiter) | Specifies the delimiter to be used for parsing through the records in the delivery stream and deaggregating them | `string` | `null` | no |
| <a name="input_dynamic_partition_record_deaggregation_type"></a> [dynamic\_partition\_record\_deaggregation\_type](#input\_dynamic\_partition\_record\_deaggregation\_type) | Data deaggregation is the process of parsing through the records in a delivery stream and separating the records based either on valid JSON or on the specified delimiter | `string` | `"JSON"` | no |
| <a name="input_dynamic_partitioning_retry_duration"></a> [dynamic\_partitioning\_retry\_duration](#input\_dynamic\_partitioning\_retry\_duration) | Total amount of seconds Firehose spends on retries | `number` | `300` | no |
| <a name="input_dynatrace_api_url"></a> [dynatrace\_api\_url](#input\_dynatrace\_api\_url) | API URL to Dynatrace destination | `string` | `null` | no |
| <a name="input_dynatrace_endpoint_location"></a> [dynatrace\_endpoint\_location](#input\_dynatrace\_endpoint\_location) | Endpoint Location to Dynatrace destination | `string` | `"eu"` | no |
| <a name="input_elasticsearch_domain_arn"></a> [elasticsearch\_domain\_arn](#input\_elasticsearch\_domain\_arn) | The ARN of the Amazon ES domain. The pattern needs to be arn:.* | `string` | `null` | no |
| <a name="input_elasticsearch_enable_vpc"></a> [elasticsearch\_enable\_vpc](#input\_elasticsearch\_enable\_vpc) | Indicates if destination is configured in VPC. Supported only to Elasticsearch destinations | `bool` | `false` | no |
| <a name="input_elasticsearch_index_name"></a> [elasticsearch\_index\_name](#input\_elasticsearch\_index\_name) | The Elasticsearch index name | `string` | `null` | no |
Expand Down Expand Up @@ -769,6 +795,7 @@ No modules.
| <a name="input_kinesis_source_stream_arn"></a> [kinesis\_source\_stream\_arn](#input\_kinesis\_source\_stream\_arn) | The kinesis stream used as the source of the firehose delivery stream | `string` | `null` | no |
| <a name="input_kinesis_source_use_existing_role"></a> [kinesis\_source\_use\_existing\_role](#input\_kinesis\_source\_use\_existing\_role) | Indicates if want use the kinesis firehose role to kinesis data stream access. | `bool` | `true` | no |
| <a name="input_name"></a> [name](#input\_name) | A name to identify the stream. This is unique to the AWS account and region the Stream is created in | `string` | n/a | yes |
| <a name="input_newrelic_endpoint_type"></a> [newrelic\_endpoint\_type](#input\_newrelic\_endpoint\_type) | Endpoint type to New Relic destination | `string` | `"logs_eu"` | no |
| <a name="input_policy_path"></a> [policy\_path](#input\_policy\_path) | Path of policies to that should be added to IAM role for Kinesis Firehose Stream | `string` | `null` | no |
| <a name="input_redshift_cluster_endpoint"></a> [redshift\_cluster\_endpoint](#input\_redshift\_cluster\_endpoint) | The redshift endpoint | `string` | `null` | no |
| <a name="input_redshift_cluster_identifier"></a> [redshift\_cluster\_identifier](#input\_redshift\_cluster\_identifier) | Redshift Cluster identifier. Necessary to associate the iam role to cluster | `string` | `null` | no |
Expand Down
2 changes: 1 addition & 1 deletion examples/http-endpoint/coralogix/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ module "firehose" {
name = "${var.name_prefix}-delivery-stream"
destination = "coralogix"
buffer_interval = 60
coralogix_endpoint_type = "coralogix_ireland"
coralogix_endpoint_location = "ireland"
http_endpoint_access_key = var.coralogix_private_key
http_endpoint_retry_duration = 30
http_endpoint_enable_request_configuration = true
Expand Down
70 changes: 70 additions & 0 deletions examples/http-endpoint/dynatrace/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# Datadog

Configuration in this directory creates kinesis firehose stream with Direct Put as source and dynatrace as destination to Europe.

This example can be tested with Demo Data in Kinesis Firehose Console.

## Usage

To run this example you need to execute:

```bash
$ terraform init
$ terraform plan
$ terraform apply
```

It's necessary configure the following variables:

```hcl
dynatrace_api_token = "<dynatrace_api_token>"
dynatrace_api_url = "<dynatrace_api_url>"
```

<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
## Requirements

| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.13.1 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 4.4 |
| <a name="requirement_random"></a> [random](#requirement\_random) | >= 2.0 |

## Providers

| Name | Version |
|------|---------|
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 4.4 |
| <a name="provider_random"></a> [random](#provider\_random) | >= 2.0 |

## Modules

| Name | Source | Version |
|------|--------|---------|
| <a name="module_firehose"></a> [firehose](#module\_firehose) | ../../../ | n/a |

## Resources

| Name | Type |
|------|------|
| [aws_kms_key.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kms_key) | resource |
| [aws_s3_bucket.s3](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket) | resource |
| [random_pet.this](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/pet) | resource |

## Inputs

| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| <a name="input_dynatrace_api_token"></a> [dynatrace\_api\_token](#input\_dynatrace\_api\_token) | Dynatrace Api Token | `string` | n/a | yes |
| <a name="input_dynatrace_api_url"></a> [dynatrace\_api\_url](#input\_dynatrace\_api\_url) | Dynatrace Api url | `string` | n/a | yes |
| <a name="input_name_prefix"></a> [name\_prefix](#input\_name\_prefix) | Name prefix to use in resources | `string` | `"firehose-to-dynatrace"` | no |

## Outputs

| Name | Description |
|------|-------------|
| <a name="output_firehose_role"></a> [firehose\_role](#output\_firehose\_role) | Firehose Role |
| <a name="output_kinesis_firehose_arn"></a> [kinesis\_firehose\_arn](#output\_kinesis\_firehose\_arn) | The ARN of the Kinesis Firehose Stream |
| <a name="output_kinesis_firehose_destination_id"></a> [kinesis\_firehose\_destination\_id](#output\_kinesis\_firehose\_destination\_id) | The Destination id of the Kinesis Firehose Stream |
| <a name="output_kinesis_firehose_version_id"></a> [kinesis\_firehose\_version\_id](#output\_kinesis\_firehose\_version\_id) | The Version id of the Kinesis Firehose Stream |
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
44 changes: 44 additions & 0 deletions examples/http-endpoint/dynatrace/main.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
resource "random_pet" "this" {
length = 2
}

resource "aws_s3_bucket" "s3" {
bucket = "${var.name_prefix}-destination-bucket-${random_pet.this.id}"
force_destroy = true
}

resource "aws_kms_key" "this" {
description = "${var.name_prefix}-kms-key"
deletion_window_in_days = 7
}

module "firehose" {
source = "../../../"
name = "${var.name_prefix}-delivery-stream"
destination = "dynatrace"
buffer_interval = 60
dynatrace_endpoint_location = "eu"
dynatrace_api_url = var.dynatrace_api_url
http_endpoint_access_key = var.dynatrace_api_token
http_endpoint_retry_duration = 60
http_endpoint_enable_request_configuration = true
http_endpoint_request_configuration_content_encoding = "GZIP"
http_endpoint_request_configuration_common_attributes = [
{
name = "testname"
value = "testvalue"
},
{
name = "testname2"
value = "testvalue2"
}
]
s3_backup_mode = "All"
s3_backup_prefix = "backup/"
s3_backup_bucket_arn = aws_s3_bucket.s3.arn
s3_backup_buffer_interval = 100
s3_backup_buffer_size = 100
s3_backup_compression = "GZIP"
s3_backup_enable_encryption = true
s3_backup_kms_key_arn = aws_kms_key.this.arn
}
19 changes: 19 additions & 0 deletions examples/http-endpoint/dynatrace/outputs.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
output "firehose_role" {
description = "Firehose Role"
value = module.firehose.kinesis_firehose_role_arn
}

output "kinesis_firehose_arn" {
description = "The ARN of the Kinesis Firehose Stream"
value = module.firehose.kinesis_firehose_arn
}

output "kinesis_firehose_destination_id" {
description = "The Destination id of the Kinesis Firehose Stream"
value = module.firehose.kinesis_firehose_destination_id
}

output "kinesis_firehose_version_id" {
description = "The Version id of the Kinesis Firehose Stream"
value = module.firehose.kinesis_firehose_version_id
}
16 changes: 16 additions & 0 deletions examples/http-endpoint/dynatrace/variables.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
variable "name_prefix" {
description = "Name prefix to use in resources"
type = string
default = "firehose-to-dynatrace"
}

variable "dynatrace_api_token" {
description = "Dynatrace Api Token"
type = string
sensitive = true
}

variable "dynatrace_api_url" {
description = "Dynatrace Api url"
type = string
}
14 changes: 14 additions & 0 deletions examples/http-endpoint/dynatrace/versions.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
terraform {
required_version = ">= 0.13.1"

required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 4.4"
}
random = {
source = "hashicorp/random"
version = ">= 2.0"
}
}
}
Loading

0 comments on commit 378ba91

Please sign in to comment.