From 30586d3a9d8bb79b48675af2bec5d973387e25d0 Mon Sep 17 00:00:00 2001 From: carolxob Date: Wed, 16 Nov 2022 15:38:21 -0500 Subject: [PATCH 01/49] Add Peer forwarder to doc site repo. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 156 ++++++++++++++++++++++++++++++++ 1 file changed, 156 insertions(+) create mode 100644 _data-prepper/peer-forwarder.md diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md new file mode 100644 index 0000000000..13e3421e0e --- /dev/null +++ b/_data-prepper/peer-forwarder.md @@ -0,0 +1,156 @@ +--- +layout: default +title: Configuration reference +nav_order: 30 +--- + +## Peer Forwarder +An HTTP service which performs peer forwarding of `Event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. + +Peer Forwarder groups events based on the identification keys provided the processors. +For `service_map_stateful` and `otel_trace_raw` it's `traceId` by default and can not be configured. +It's configurable for `aggregate` processor using `identification_keys` configuration option. You can find more information about identification keys [here](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). + +--- + +Presently peer discovery is provided by either a static list or by a DNS record lookup or AWS Cloudmap. + +### Static discovery mode +Static discover mode allows Data Prepper node to discover nodes using a list of IP addresses or domain names. +```yaml +peer_forwarder: + discovery_mode: static + static_endpoints: ["data-prepper1", "data-prepper2"] +``` + +### DNS lookup discovery mode +We recommend using DNS discovery over static discovery when scaling out a Data Prepper cluster. The core concept is to configure a DNS provider to return a list of Data Prepper hosts when given a single domain name. +This is a [DNS A Record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) which indicates a list of IP addresses of a given domain. +```yaml +peer_forwarder: + discovery_mode: dns + domain_name: "data-prepper-cluster.my-domain.net" +``` + +### AWS Cloud Map discovery mode + +[AWS Cloud Map](https://docs.aws.amazon.com/cloud-map/latest/dg/what-is-cloud-map.html) provides API-based service discovery as well as DNS-based service discovery. + +Peer forwarder can use the API-based service discovery. To support this you must have an existing +namespace configured for API instance discovery. You can create a new one following the instructions +provided by the [Cloud Map documentation](https://docs.aws.amazon.com/cloud-map/latest/dg/working-with-namespaces.html). + +Your Data Prepper configuration needs to include: +* `aws_cloud_map_namespace_name` - Set to your Cloud Map Namespace name +* `aws_cloud_map_service_name` - Set to the service name within your specified Namespace +* `aws_region` - The AWS region where your namespace exists. +* `discovery_mode` - Set to `aws_cloud_map` + +Your Data Prepper configuration can optionally include: +* `aws_cloud_map_query_parameters` - Key/value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. + +Example configuration: + +```yaml +peer_forwarder: + discovery_mode: aws_cloud_map + aws_cloud_map_namespace_name: "my-namespace" + aws_cloud_map_service_name: "data-prepper-cluster" + aws_cloud_map_query_parameters: + instance_type: "r5.xlarge" + aws_region: "us-east-1" +``` + +The Data Prepper must also be running with the necessary permissions. The following +IAM policy shows the necessary permissions. + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "CloudMapPeerForwarder", + "Effect": "Allow", + "Action": "servicediscovery:DiscoverInstances", + "Resource": "*" + } + ] +} +``` +--- +## Configuration + +* `port`(Optional): An `int` between 0 and 65535 represents the port peer forwarder server is running on. Default value is `4994`. +* `request_timeout`(Optional): Duration - An `int` representing the request timeout in milliseconds for Peer Forwarder HTTP server. Default value is `10000`. +* `server_thread_count`(Optional): An `int` representing number of threads used by Peer Forwarder server. Defaults to `200`. +* `client_thread_count`(Optional): An `int` representing number of threads used by Peer Forwarder client. Defaults to `200`. +* `maxConnectionCount`(Optional): An `int` representing maximum number of open connections for Peer Forwarder server. Default value is `500`. +* `discovery_mode`(Optional): A `String` representing the peer discovery mode to be used. Allowable values are `local_node`, `static`, `dns`, and `aws_cloud_map`. Defaults to `local_node` which processes events locally. +* `static_endpoints`(Optional): A `list` containing endpoints of all Data Prepper instances. Required if `discovery_mode` is set to `static`. +* `domain_name`(Optional): A `String` representing single domain name to query DNS against. Typically, used by creating multiple [DNS A Records](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) for the same domain. Required if `discovery_mode` is set to `dns`. +* `aws_cloud_map_namespace_name`(Optional) - A `String` representing the Cloud Map namespace when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. +* `aws_cloud_map_service_name`(Optional) - A `String` representing the Cloud Map service when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. +* `aws_cloud_map_query_parameters`(Optional): A `Map` of Key/value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. +* `buffer_size`(Optional): An `int` representing max number of unchecked records the buffer accepts (num of unchecked records = num of records written into the buffer + num of in-flight records not yet checked by the Checkpointing API). Default is `512`. +* `batch_size`(Optional): An `int` representing max number of records the buffer returns on read. Default is `48`. +* `aws_region`(Optional) : A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. +* `drain_timeout`(Optional) : A `Duration` representing the wait time for the peer forwarder to complete processing data before shutdown. + +### SSL +The SSL configuration for setting up trust manager for peer forwarding client to connect to other Data Prepper instances. + +* `ssl`(Optional) : A `boolean` that enables TLS/SSL. Default value is `true`. +* `ssl_certificate_file`(Optional) : A `String` representing the SSL certificate chain file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_certificate.pem` which is default certificate file. Read more about how the certificate file is generated [here](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates). +* `ssl_key_file`(Optional) : A `String` represents the SSL key file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_private_key.pem` which is default private key file. Read more about how the private key file is generated [here](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates). +* `ssl_insecure_disable_verification`(Optional) : A `boolean` that disables the verification of server's TLS certificate chain. Default value is `false`. +* `ssl_fingerprint_verification_only`(Optional) : A `boolean` that disables the verification of server's TLS certificate chain and instead verifies only the certificate fingerprint. Default value is `false`. +* `use_acm_certificate_for_ssl`(Optional) : A `boolean` that enables TLS/SSL using certificate and private key from AWS Certificate Manager (ACM). Default is `false`. +* `acm_certificate_arn`(Optional) : A `String` represents the ACM certificate ARN. ACM certificate take preference over S3 or local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. +* `acm_private_key_password`(Optional) : A `String` that represents the ACM private key password which that will be used to decrypt the private key. If it's not provided, a random password will be generated. +* `acm_certificate_timeout_millis`(Optional) : An `int` representing the timeout in milliseconds for ACM to get certificates. Default value is `120000`. +* `aws_region`(Optional) : A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. + +```yaml +peer_forwarder: + ssl: true + ssl_certificate_file: "" + ssl_key_file: "" +``` + +### Authentication + +* `authentication`(Optional) : A `Map` that enables mTLS. It can either be `mutual_tls` or `unauthenticated`. Default value is `unauthenticated`. +```yaml +peer_forwarder: + authentication: + mutual_tls: +``` + +## Metrics + +Core Peer Forwarder introduces the following custom metrics and all the metrics are prefixed by `core.peerForwarder` + +### Timer + +- `requestForwardingLatency`: measures latency of forwarding requests by peer forwarder client. +- `requestProcessingLatency`: measures latency of processing requests by peer forwarder server. + +### Counter + +- `requests`: measures total number of forwarded requests. +- `requestsFailed`: measures total number of failed requests. Requests with HTTP response code other than `200`. +- `requestsSuccessful`: measures total number of successful requests. Requests with HTTP response code `200`. +- `requestsTooLarge`: measures total number of requests which are too large to be written to peer forwarder buffer. Requests with HTTP response code `413`. +- `requestTimeouts`: measures total number of requests which timed out while writing content to peer forwarder buffer. Requests with HTTP response code `408`. +- `requestsUnprocessable`: measures total number of requests which failed due to unprocessable entity. Requests with HTTP response code `422`. +- `badRequests`: measures total number of requests with bad request format. Requests with HTTP response code `400`. +- `recordsSuccessfullyForwarded`: measures total number of forwarded records successfully. +- `recordsFailedForwarding`: measures total number of records failed to be forwarded. +- `recordsToBeForwarded`: measures total number of records to be forwarded. +- `recordsToBeProcessedLocally`: measures total number of records to be processed locally. +- `recordsActuallyProcessedLocally`: measures total number of records actually processed locally. Sum of `recordsToBeProcessedLocally` and `recordsFailedForwarding`. +- `recordsReceivedFromPeers`: measures total number of records received from remote peers. + +### Gauge + +- `peerEndpoints`: measures number of dynamically discovered peer data-prepper endpoints. For `static` mode, the size is fixed. From 73fab9fe7c3391de21e008cf1d062004d1ef48b9 Mon Sep 17 00:00:00 2001 From: carolxob Date: Wed, 16 Nov 2022 15:46:16 -0500 Subject: [PATCH 02/49] Minor updates. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 13e3421e0e..1d6c0a7d8f 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -1,7 +1,7 @@ --- layout: default title: Configuration reference -nav_order: 30 +nav_order: 12 --- ## Peer Forwarder From f8feef6743a6edca4a0c68f03a63739a050dee1a Mon Sep 17 00:00:00 2001 From: carolxob Date: Wed, 16 Nov 2022 15:47:37 -0500 Subject: [PATCH 03/49] Minor updates to header section. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 1d6c0a7d8f..17501b976b 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -1,6 +1,6 @@ --- layout: default -title: Configuration reference +title: Peer forwarder nav_order: 12 --- From 24d4a5e3980e37251c149f81fecf794e4d65aa68 Mon Sep 17 00:00:00 2001 From: carolxob Date: Mon, 21 Nov 2022 15:16:46 -0500 Subject: [PATCH 04/49] Minor copyedits and heading adjustements. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 41 +++++++++++++++------------------ 1 file changed, 19 insertions(+), 22 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 17501b976b..c182ab9329 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -4,18 +4,17 @@ title: Peer forwarder nav_order: 12 --- -## Peer Forwarder -An HTTP service which performs peer forwarding of `Event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. +Peer Forwarder is an HTTP service which performs peer forwarding of `Event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. -Peer Forwarder groups events based on the identification keys provided the processors. -For `service_map_stateful` and `otel_trace_raw` it's `traceId` by default and can not be configured. -It's configurable for `aggregate` processor using `identification_keys` configuration option. You can find more information about identification keys [here](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). +Peer Forwarder groups events based on the identification keys provided the processors. For `service_map_stateful` and `otel_trace_raw` it's `traceId` by default and can not be configured. It's configurable for `aggregate` processor using `identification_keys` configuration option. You can find more information about identification keys [here](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). --- -Presently peer discovery is provided by either a static list or by a DNS record lookup or AWS Cloudmap. +Right now, peer discovery is provided by either a static list or by a DNS record lookup or AWS Cloudmap. -### Static discovery mode +## Discovery modes + +### Static discovery Static discover mode allows Data Prepper node to discover nodes using a list of IP addresses or domain names. ```yaml peer_forwarder: @@ -23,22 +22,20 @@ peer_forwarder: static_endpoints: ["data-prepper1", "data-prepper2"] ``` -### DNS lookup discovery mode -We recommend using DNS discovery over static discovery when scaling out a Data Prepper cluster. The core concept is to configure a DNS provider to return a list of Data Prepper hosts when given a single domain name. -This is a [DNS A Record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) which indicates a list of IP addresses of a given domain. +### DNS lookup discovery +We recommend using DNS discovery over static discovery when scaling out a Data Prepper cluster. The core concept is to configure a DNS provider to return a list of Data Prepper hosts when given a single domain name. This is a [DNS A Record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) which indicates a list of IP addresses of a given domain. + ```yaml peer_forwarder: discovery_mode: dns domain_name: "data-prepper-cluster.my-domain.net" ``` -### AWS Cloud Map discovery mode +### AWS Cloud Map discovery [AWS Cloud Map](https://docs.aws.amazon.com/cloud-map/latest/dg/what-is-cloud-map.html) provides API-based service discovery as well as DNS-based service discovery. -Peer forwarder can use the API-based service discovery. To support this you must have an existing -namespace configured for API instance discovery. You can create a new one following the instructions -provided by the [Cloud Map documentation](https://docs.aws.amazon.com/cloud-map/latest/dg/working-with-namespaces.html). +Peer forwarder can use the API-based service discovery. To support this you must have an existing namespace configured for API instance discovery. You can create a new one following the instructions provided by the [Cloud Map documentation](https://docs.aws.amazon.com/cloud-map/latest/dg/working-with-namespaces.html). Your Data Prepper configuration needs to include: * `aws_cloud_map_namespace_name` - Set to your Cloud Map Namespace name @@ -49,7 +46,7 @@ Your Data Prepper configuration needs to include: Your Data Prepper configuration can optionally include: * `aws_cloud_map_query_parameters` - Key/value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. -Example configuration: +#### Example configuration ```yaml peer_forwarder: @@ -78,7 +75,7 @@ IAM policy shows the necessary permissions. } ``` --- -## Configuration +# Configuration * `port`(Optional): An `int` between 0 and 65535 represents the port peer forwarder server is running on. Default value is `4994`. * `request_timeout`(Optional): Duration - An `int` representing the request timeout in milliseconds for Peer Forwarder HTTP server. Default value is `10000`. @@ -96,7 +93,7 @@ IAM policy shows the necessary permissions. * `aws_region`(Optional) : A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. * `drain_timeout`(Optional) : A `Duration` representing the wait time for the peer forwarder to complete processing data before shutdown. -### SSL +## SSL The SSL configuration for setting up trust manager for peer forwarding client to connect to other Data Prepper instances. * `ssl`(Optional) : A `boolean` that enables TLS/SSL. Default value is `true`. @@ -117,7 +114,7 @@ peer_forwarder: ssl_key_file: "" ``` -### Authentication +## Authentication * `authentication`(Optional) : A `Map` that enables mTLS. It can either be `mutual_tls` or `unauthenticated`. Default value is `unauthenticated`. ```yaml @@ -126,16 +123,16 @@ peer_forwarder: mutual_tls: ``` -## Metrics +# Metrics Core Peer Forwarder introduces the following custom metrics and all the metrics are prefixed by `core.peerForwarder` -### Timer +## Timer - `requestForwardingLatency`: measures latency of forwarding requests by peer forwarder client. - `requestProcessingLatency`: measures latency of processing requests by peer forwarder server. -### Counter +## Counter - `requests`: measures total number of forwarded requests. - `requestsFailed`: measures total number of failed requests. Requests with HTTP response code other than `200`. @@ -151,6 +148,6 @@ Core Peer Forwarder introduces the following custom metrics and all the metrics - `recordsActuallyProcessedLocally`: measures total number of records actually processed locally. Sum of `recordsToBeProcessedLocally` and `recordsFailedForwarding`. - `recordsReceivedFromPeers`: measures total number of records received from remote peers. -### Gauge +## Gauge - `peerEndpoints`: measures number of dynamically discovered peer data-prepper endpoints. For `static` mode, the size is fixed. From 7fb9b33985d87a1a1e2256f9ad64b2fe29f25183 Mon Sep 17 00:00:00 2001 From: carolxob Date: Mon, 21 Nov 2022 15:27:30 -0500 Subject: [PATCH 05/49] Minor copyedits and heading updates. Additional of Optional section to simplify wording. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 54 ++++++++++++++++++--------------- 1 file changed, 29 insertions(+), 25 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index c182ab9329..8d78d48b7d 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -77,35 +77,39 @@ IAM policy shows the necessary permissions. --- # Configuration -* `port`(Optional): An `int` between 0 and 65535 represents the port peer forwarder server is running on. Default value is `4994`. -* `request_timeout`(Optional): Duration - An `int` representing the request timeout in milliseconds for Peer Forwarder HTTP server. Default value is `10000`. -* `server_thread_count`(Optional): An `int` representing number of threads used by Peer Forwarder server. Defaults to `200`. -* `client_thread_count`(Optional): An `int` representing number of threads used by Peer Forwarder client. Defaults to `200`. -* `maxConnectionCount`(Optional): An `int` representing maximum number of open connections for Peer Forwarder server. Default value is `500`. -* `discovery_mode`(Optional): A `String` representing the peer discovery mode to be used. Allowable values are `local_node`, `static`, `dns`, and `aws_cloud_map`. Defaults to `local_node` which processes events locally. -* `static_endpoints`(Optional): A `list` containing endpoints of all Data Prepper instances. Required if `discovery_mode` is set to `static`. -* `domain_name`(Optional): A `String` representing single domain name to query DNS against. Typically, used by creating multiple [DNS A Records](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) for the same domain. Required if `discovery_mode` is set to `dns`. -* `aws_cloud_map_namespace_name`(Optional) - A `String` representing the Cloud Map namespace when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. -* `aws_cloud_map_service_name`(Optional) - A `String` representing the Cloud Map service when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. -* `aws_cloud_map_query_parameters`(Optional): A `Map` of Key/value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. -* `buffer_size`(Optional): An `int` representing max number of unchecked records the buffer accepts (num of unchecked records = num of records written into the buffer + num of in-flight records not yet checked by the Checkpointing API). Default is `512`. -* `batch_size`(Optional): An `int` representing max number of records the buffer returns on read. Default is `48`. -* `aws_region`(Optional) : A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. -* `drain_timeout`(Optional) : A `Duration` representing the wait time for the peer forwarder to complete processing data before shutdown. +## Optional + +* `port`: An `int` between 0 and 65535 represents the port peer forwarder server is running on. Default value is `4994`. +* `request_timeout`: Duration - An `int` representing the request timeout in milliseconds for Peer Forwarder HTTP server. Default value is `10000`. +* `server_thread_count`: An `int` representing number of threads used by Peer Forwarder server. Defaults to `200`. +* `client_thread_count`: An `int` representing number of threads used by Peer Forwarder client. Defaults to `200`. +* `maxConnectionCount`: An `int` representing maximum number of open connections for Peer Forwarder server. Default value is `500`. +* `discovery_mode`: A `String` representing the peer discovery mode to be used. Allowable values are `local_node`, `static`, `dns`, and `aws_cloud_map`. Defaults to `local_node` which processes events locally. +* `static_endpoints`: A `list` containing endpoints of all Data Prepper instances. Required if `discovery_mode` is set to `static`. +* `domain_name`: A `String` representing single domain name to query DNS against. Typically, used by creating multiple [DNS A Records](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) for the same domain. Required if `discovery_mode` is set to `dns`. +* `aws_cloud_map_namespace_name`: A `String` representing the Cloud Map namespace when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. +* `aws_cloud_map_service_name`: A `String` representing the Cloud Map service when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. +* `aws_cloud_map_query_parameters`: A `Map` of Key/value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. +* `buffer_size`: An `int` representing max number of unchecked records the buffer accepts (num of unchecked records = num of records written into the buffer + num of in-flight records not yet checked by the Checkpointing API). Default is `512`. +* `batch_size`: An `int` representing max number of records the buffer returns on read. Default is `48`. +* `aws_region`: A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. +* `drain_timeout`: A `Duration` representing the wait time for the peer forwarder to complete processing data before shutdown. ## SSL The SSL configuration for setting up trust manager for peer forwarding client to connect to other Data Prepper instances. -* `ssl`(Optional) : A `boolean` that enables TLS/SSL. Default value is `true`. -* `ssl_certificate_file`(Optional) : A `String` representing the SSL certificate chain file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_certificate.pem` which is default certificate file. Read more about how the certificate file is generated [here](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates). -* `ssl_key_file`(Optional) : A `String` represents the SSL key file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_private_key.pem` which is default private key file. Read more about how the private key file is generated [here](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates). -* `ssl_insecure_disable_verification`(Optional) : A `boolean` that disables the verification of server's TLS certificate chain. Default value is `false`. -* `ssl_fingerprint_verification_only`(Optional) : A `boolean` that disables the verification of server's TLS certificate chain and instead verifies only the certificate fingerprint. Default value is `false`. -* `use_acm_certificate_for_ssl`(Optional) : A `boolean` that enables TLS/SSL using certificate and private key from AWS Certificate Manager (ACM). Default is `false`. -* `acm_certificate_arn`(Optional) : A `String` represents the ACM certificate ARN. ACM certificate take preference over S3 or local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. -* `acm_private_key_password`(Optional) : A `String` that represents the ACM private key password which that will be used to decrypt the private key. If it's not provided, a random password will be generated. -* `acm_certificate_timeout_millis`(Optional) : An `int` representing the timeout in milliseconds for ACM to get certificates. Default value is `120000`. -* `aws_region`(Optional) : A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. +### Optional + +* `ssl`: A `boolean` that enables TLS/SSL. Default value is `true`. +* `ssl_certificate_file`: A `String` representing the SSL certificate chain file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_certificate.pem` which is default certificate file. Read more about how the certificate file is generated [here](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates). +* `ssl_key_file`: A `String` represents the SSL key file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_private_key.pem` which is default private key file. Read more about how the private key file is generated [here](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates). +* `ssl_insecure_disable_verification`: A `boolean` that disables the verification of server's TLS certificate chain. Default value is `false`. +* `ssl_fingerprint_verification_only`: A `boolean` that disables the verification of server's TLS certificate chain and instead verifies only the certificate fingerprint. Default value is `false`. +* `use_acm_certificate_for_ssl`: A `boolean` that enables TLS/SSL using certificate and private key from AWS Certificate Manager (ACM). Default is `false`. +* `acm_certificate_arn`: A `String` represents the ACM certificate ARN. ACM certificate take preference over S3 or local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. +* `acm_private_key_password`: A `String` that represents the ACM private key password which that will be used to decrypt the private key. If it's not provided, a random password will be generated. +* `acm_certificate_timeout_millis`: An `int` representing the timeout in milliseconds for ACM to get certificates. Default value is `120000`. +* `aws_region`: A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. ```yaml peer_forwarder: From e709ce91c7747e4f3ef9118cd9aadb18e51b362d Mon Sep 17 00:00:00 2001 From: carolxob Date: Tue, 22 Nov 2022 13:12:03 -0500 Subject: [PATCH 06/49] Minor copyedits and heading adjustments. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 8d78d48b7d..27dde1e048 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -12,9 +12,9 @@ Peer Forwarder groups events based on the identification keys provided the proce Right now, peer discovery is provided by either a static list or by a DNS record lookup or AWS Cloudmap. -## Discovery modes +# Discovery modes -### Static discovery +## Static Static discover mode allows Data Prepper node to discover nodes using a list of IP addresses or domain names. ```yaml peer_forwarder: @@ -22,7 +22,7 @@ peer_forwarder: static_endpoints: ["data-prepper1", "data-prepper2"] ``` -### DNS lookup discovery +## DNS lookup We recommend using DNS discovery over static discovery when scaling out a Data Prepper cluster. The core concept is to configure a DNS provider to return a list of Data Prepper hosts when given a single domain name. This is a [DNS A Record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) which indicates a list of IP addresses of a given domain. ```yaml @@ -31,7 +31,7 @@ peer_forwarder: domain_name: "data-prepper-cluster.my-domain.net" ``` -### AWS Cloud Map discovery +## AWS Cloud Map [AWS Cloud Map](https://docs.aws.amazon.com/cloud-map/latest/dg/what-is-cloud-map.html) provides API-based service discovery as well as DNS-based service discovery. @@ -46,7 +46,7 @@ Your Data Prepper configuration needs to include: Your Data Prepper configuration can optionally include: * `aws_cloud_map_query_parameters` - Key/value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. -#### Example configuration +### Example configuration ```yaml peer_forwarder: @@ -58,8 +58,9 @@ peer_forwarder: aws_region: "us-east-1" ``` -The Data Prepper must also be running with the necessary permissions. The following -IAM policy shows the necessary permissions. +### IAM policy with necessary permissions + +The Data Prepper must also be running with the necessary permissions. The following IAM policy shows the necessary permissions. ```json { @@ -77,7 +78,7 @@ IAM policy shows the necessary permissions. --- # Configuration -## Optional +## Optional configuration * `port`: An `int` between 0 and 65535 represents the port peer forwarder server is running on. Default value is `4994`. * `request_timeout`: Duration - An `int` representing the request timeout in milliseconds for Peer Forwarder HTTP server. Default value is `10000`. @@ -95,10 +96,10 @@ IAM policy shows the necessary permissions. * `aws_region`: A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. * `drain_timeout`: A `Duration` representing the wait time for the peer forwarder to complete processing data before shutdown. -## SSL +# SSL configuration The SSL configuration for setting up trust manager for peer forwarding client to connect to other Data Prepper instances. -### Optional +## Optional SSL configuration * `ssl`: A `boolean` that enables TLS/SSL. Default value is `true`. * `ssl_certificate_file`: A `String` representing the SSL certificate chain file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_certificate.pem` which is default certificate file. Read more about how the certificate file is generated [here](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates). @@ -118,7 +119,7 @@ peer_forwarder: ssl_key_file: "" ``` -## Authentication +# Authentication * `authentication`(Optional) : A `Map` that enables mTLS. It can either be `mutual_tls` or `unauthenticated`. Default value is `unauthenticated`. ```yaml @@ -129,14 +130,14 @@ peer_forwarder: # Metrics -Core Peer Forwarder introduces the following custom metrics and all the metrics are prefixed by `core.peerForwarder` +Core Peer Forwarder introduces the following custom metrics and all the metrics are prefixed by `core.peerForwarder`. -## Timer +# Timer - `requestForwardingLatency`: measures latency of forwarding requests by peer forwarder client. - `requestProcessingLatency`: measures latency of processing requests by peer forwarder server. -## Counter +# Counter - `requests`: measures total number of forwarded requests. - `requestsFailed`: measures total number of failed requests. Requests with HTTP response code other than `200`. From 74d1ed3495e2b0ba315aef11341e7efb95f6ae66 Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 9 Dec 2022 09:19:54 -0700 Subject: [PATCH 07/49] Changed capitalization in title. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 27dde1e048..8b3f2214b2 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -1,6 +1,6 @@ --- layout: default -title: Peer forwarder +title: Peer Forwarder nav_order: 12 --- From 5f3fb82c74fc4d890cac2c6ac5863e5a517428a9 Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 9 Dec 2022 14:46:53 -0700 Subject: [PATCH 08/49] Minor changes from doc review feedback. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 8b3f2214b2..2af16ffaba 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -4,7 +4,7 @@ title: Peer Forwarder nav_order: 12 --- -Peer Forwarder is an HTTP service which performs peer forwarding of `Event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. +Peer Forwarder is an HTTP service which performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. Peer Forwarder groups events based on the identification keys provided the processors. For `service_map_stateful` and `otel_trace_raw` it's `traceId` by default and can not be configured. It's configurable for `aggregate` processor using `identification_keys` configuration option. You can find more information about identification keys [here](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). From f32c31b9f1b3288542ba74e55a9618259e4bceba Mon Sep 17 00:00:00 2001 From: Caroline <113052567+carolxob@users.noreply.github.com> Date: Fri, 9 Dec 2022 14:50:14 -0700 Subject: [PATCH 09/49] Update _data-prepper/peer-forwarder.md Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com> --- _data-prepper/peer-forwarder.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 2af16ffaba..2f066f7b58 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -6,7 +6,7 @@ nav_order: 12 Peer Forwarder is an HTTP service which performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. -Peer Forwarder groups events based on the identification keys provided the processors. For `service_map_stateful` and `otel_trace_raw` it's `traceId` by default and can not be configured. It's configurable for `aggregate` processor using `identification_keys` configuration option. You can find more information about identification keys [here](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). +Peer Forwarder groups events based on the identification keys provided the processors. For `service_map_stateful` and `otel_trace_raw`, it's `traceId` by default, and can not be configured. It's configurable for `aggregate` processor using `identification_keys` configuration option. You can find more information about identification keys [here](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). --- From 01c4fb05bd26b39aa043f4f6118550534a4f1be7 Mon Sep 17 00:00:00 2001 From: Caroline <113052567+carolxob@users.noreply.github.com> Date: Fri, 9 Dec 2022 14:50:26 -0700 Subject: [PATCH 10/49] Update _data-prepper/peer-forwarder.md Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com> --- _data-prepper/peer-forwarder.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 2f066f7b58..22b9cbc99a 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -10,7 +10,7 @@ Peer Forwarder groups events based on the identification keys provided the proce --- -Right now, peer discovery is provided by either a static list or by a DNS record lookup or AWS Cloudmap. +Peer discovery is currently provided by either a static list or by a DNS record lookup or AWS Cloudmap. # Discovery modes From d4b1faef0f0d32820328b19de8e95b58a4e53645 Mon Sep 17 00:00:00 2001 From: Caroline <113052567+carolxob@users.noreply.github.com> Date: Fri, 9 Dec 2022 14:50:35 -0700 Subject: [PATCH 11/49] Update _data-prepper/peer-forwarder.md Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com> --- _data-prepper/peer-forwarder.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 22b9cbc99a..68491bdb6a 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -15,7 +15,7 @@ Peer discovery is currently provided by either a static list or by a DNS record # Discovery modes ## Static -Static discover mode allows Data Prepper node to discover nodes using a list of IP addresses or domain names. +Static discover mode allows a Data Prepper node to discover nodes using a list of IP addresses or domain names. ```yaml peer_forwarder: discovery_mode: static From 73cc07e63d982f4f5bf0a95ee959c97cd992ca30 Mon Sep 17 00:00:00 2001 From: Caroline <113052567+carolxob@users.noreply.github.com> Date: Fri, 9 Dec 2022 14:50:45 -0700 Subject: [PATCH 12/49] Update _data-prepper/peer-forwarder.md Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com> --- _data-prepper/peer-forwarder.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 68491bdb6a..f7026e5b82 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -60,7 +60,7 @@ peer_forwarder: ### IAM policy with necessary permissions -The Data Prepper must also be running with the necessary permissions. The following IAM policy shows the necessary permissions. +The Data Prepper must also be running with the necessary permissions. The following IAM policy shows the necessary permissions: ```json { From c9c3727ac63a29fb4af03be1454184e59dab7ea5 Mon Sep 17 00:00:00 2001 From: Caroline <113052567+carolxob@users.noreply.github.com> Date: Fri, 9 Dec 2022 14:50:54 -0700 Subject: [PATCH 13/49] Update _data-prepper/peer-forwarder.md Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com> --- _data-prepper/peer-forwarder.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index f7026e5b82..6370daba5b 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -97,7 +97,7 @@ The Data Prepper must also be running with the necessary permissions. The follow * `drain_timeout`: A `Duration` representing the wait time for the peer forwarder to complete processing data before shutdown. # SSL configuration -The SSL configuration for setting up trust manager for peer forwarding client to connect to other Data Prepper instances. +The SSL configuration for setting up trust manager for the peer forwarding client to connect to other Data Prepper instances. ## Optional SSL configuration From db822eeb1d9c9afdc1cc3b50ff6d17d59cc79d8b Mon Sep 17 00:00:00 2001 From: Caroline <113052567+carolxob@users.noreply.github.com> Date: Fri, 9 Dec 2022 14:51:50 -0700 Subject: [PATCH 14/49] Update _data-prepper/peer-forwarder.md Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com> --- _data-prepper/peer-forwarder.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 6370daba5b..3adf7bfa4e 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -143,7 +143,7 @@ Core Peer Forwarder introduces the following custom metrics and all the metrics - `requestsFailed`: measures total number of failed requests. Requests with HTTP response code other than `200`. - `requestsSuccessful`: measures total number of successful requests. Requests with HTTP response code `200`. - `requestsTooLarge`: measures total number of requests which are too large to be written to peer forwarder buffer. Requests with HTTP response code `413`. -- `requestTimeouts`: measures total number of requests which timed out while writing content to peer forwarder buffer. Requests with HTTP response code `408`. +- `requestTimeouts`: measures the total number of requests which timed out while writing content to peer forwarder buffer. Requests with HTTP response code `408`. - `requestsUnprocessable`: measures total number of requests which failed due to unprocessable entity. Requests with HTTP response code `422`. - `badRequests`: measures total number of requests with bad request format. Requests with HTTP response code `400`. - `recordsSuccessfullyForwarded`: measures total number of forwarded records successfully. From 491db107cf2046b376e4a1a02c62da4f1da937f6 Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 9 Dec 2022 15:07:59 -0700 Subject: [PATCH 15/49] Trying to commit file. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 3adf7bfa4e..45968a8c38 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -6,7 +6,11 @@ nav_order: 12 Peer Forwarder is an HTTP service which performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. +<<<<<<< HEAD Peer Forwarder groups events based on the identification keys provided the processors. For `service_map_stateful` and `otel_trace_raw`, it's `traceId` by default, and can not be configured. It's configurable for `aggregate` processor using `identification_keys` configuration option. You can find more information about identification keys [here](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). +======= +Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The identification key is configurable for the `aggregate` processor using the `identification_keys` configuration option. You can find more information about [identification keys](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) +>>>>>>> c1b85e08 (Minor changes.) --- From a87761079160a818d779786a9d979cbc77f77ba4 Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 9 Dec 2022 15:17:08 -0700 Subject: [PATCH 16/49] Trying to push file. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 45968a8c38..7b7a0139b4 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -6,11 +6,7 @@ nav_order: 12 Peer Forwarder is an HTTP service which performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. -<<<<<<< HEAD -Peer Forwarder groups events based on the identification keys provided the processors. For `service_map_stateful` and `otel_trace_raw`, it's `traceId` by default, and can not be configured. It's configurable for `aggregate` processor using `identification_keys` configuration option. You can find more information about identification keys [here](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). -======= -Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The identification key is configurable for the `aggregate` processor using the `identification_keys` configuration option. You can find more information about [identification keys](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) ->>>>>>> c1b85e08 (Minor changes.) +Peer Forwarder groups events based on the identification keys provided the processors. For `service_map_stateful` and `otel_trace_raw`, the value is `traceId` by default, and can not be configured. It's configurable for `aggregate` processor using `identification_keys` configuration option. You can find more information about identification keys [here](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). --- From ab8edae279d585d2a6cc738f90695f38970ddc11 Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 9 Dec 2022 15:21:25 -0700 Subject: [PATCH 17/49] Trying to push file again. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 7b7a0139b4..bc43871282 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -6,7 +6,11 @@ nav_order: 12 Peer Forwarder is an HTTP service which performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. +<<<<<<< HEAD Peer Forwarder groups events based on the identification keys provided the processors. For `service_map_stateful` and `otel_trace_raw`, the value is `traceId` by default, and can not be configured. It's configurable for `aggregate` processor using `identification_keys` configuration option. You can find more information about identification keys [here](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). +======= +Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The identification key is configurable for the `aggregate` processor using the `identification_keys` configuration option. You can find more information about [identification keys](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). +>>>>>>> 0d83461f (Trying to push file.) --- From f83dffd7fa951ec117441f42c8a37817bbf0d377 Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 9 Dec 2022 15:23:09 -0700 Subject: [PATCH 18/49] Trying to push file one more time after rebasing. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index bc43871282..83480e4a72 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -6,15 +6,11 @@ nav_order: 12 Peer Forwarder is an HTTP service which performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. -<<<<<<< HEAD -Peer Forwarder groups events based on the identification keys provided the processors. For `service_map_stateful` and `otel_trace_raw`, the value is `traceId` by default, and can not be configured. It's configurable for `aggregate` processor using `identification_keys` configuration option. You can find more information about identification keys [here](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). -======= Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The identification key is configurable for the `aggregate` processor using the `identification_keys` configuration option. You can find more information about [identification keys](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). ->>>>>>> 0d83461f (Trying to push file.) --- -Peer discovery is currently provided by either a static list or by a DNS record lookup or AWS Cloudmap. +Peer discovery is currently provided by either a static list or by a DNS record lookup or AWS Cloudmap. # Discovery modes From 0ee78500cf5a5ac40b86979d313770e440dede2c Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 9 Dec 2022 15:32:01 -0700 Subject: [PATCH 19/49] Minor change. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 83480e4a72..6cc3a906b9 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -6,7 +6,7 @@ nav_order: 12 Peer Forwarder is an HTTP service which performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. -Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The identification key is configurable for the `aggregate` processor using the `identification_keys` configuration option. You can find more information about [identification keys](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). +Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The value is configurable for the `aggregate` processor using the `identification_keys` configuration option. You can find more information about [identification keys](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). --- @@ -17,7 +17,7 @@ Peer discovery is currently provided by either a static list or by a DNS record ## Static Static discover mode allows a Data Prepper node to discover nodes using a list of IP addresses or domain names. ```yaml -peer_forwarder: +peer_forwarder:4 discovery_mode: static static_endpoints: ["data-prepper1", "data-prepper2"] ``` From 2cb064d6b94fd0e69e1f9ac71e8abe125ad70487 Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 9 Dec 2022 15:45:17 -0700 Subject: [PATCH 20/49] Minor edits. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 6cc3a906b9..a75831ea2d 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -14,6 +14,8 @@ Peer discovery is currently provided by either a static list or by a DNS record # Discovery modes +See the following information about discovery modes below. + ## Static Static discover mode allows a Data Prepper node to discover nodes using a list of IP addresses or domain names. ```yaml From 0609ee4ad9c203d9fdb1c695f8064e1c7ae9cceb Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 9 Dec 2022 16:00:55 -0700 Subject: [PATCH 21/49] Converted optional configuration section to table. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 43 ++++++++++++++++++--------------- 1 file changed, 24 insertions(+), 19 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index a75831ea2d..1f02954118 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -17,6 +17,7 @@ Peer discovery is currently provided by either a static list or by a DNS record See the following information about discovery modes below. ## Static + Static discover mode allows a Data Prepper node to discover nodes using a list of IP addresses or domain names. ```yaml peer_forwarder:4 @@ -78,25 +79,29 @@ The Data Prepper must also be running with the necessary permissions. The follow } ``` --- -# Configuration - -## Optional configuration - -* `port`: An `int` between 0 and 65535 represents the port peer forwarder server is running on. Default value is `4994`. -* `request_timeout`: Duration - An `int` representing the request timeout in milliseconds for Peer Forwarder HTTP server. Default value is `10000`. -* `server_thread_count`: An `int` representing number of threads used by Peer Forwarder server. Defaults to `200`. -* `client_thread_count`: An `int` representing number of threads used by Peer Forwarder client. Defaults to `200`. -* `maxConnectionCount`: An `int` representing maximum number of open connections for Peer Forwarder server. Default value is `500`. -* `discovery_mode`: A `String` representing the peer discovery mode to be used. Allowable values are `local_node`, `static`, `dns`, and `aws_cloud_map`. Defaults to `local_node` which processes events locally. -* `static_endpoints`: A `list` containing endpoints of all Data Prepper instances. Required if `discovery_mode` is set to `static`. -* `domain_name`: A `String` representing single domain name to query DNS against. Typically, used by creating multiple [DNS A Records](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) for the same domain. Required if `discovery_mode` is set to `dns`. -* `aws_cloud_map_namespace_name`: A `String` representing the Cloud Map namespace when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. -* `aws_cloud_map_service_name`: A `String` representing the Cloud Map service when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. -* `aws_cloud_map_query_parameters`: A `Map` of Key/value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. -* `buffer_size`: An `int` representing max number of unchecked records the buffer accepts (num of unchecked records = num of records written into the buffer + num of in-flight records not yet checked by the Checkpointing API). Default is `512`. -* `batch_size`: An `int` representing max number of records the buffer returns on read. Default is `48`. -* `aws_region`: A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. -* `drain_timeout`: A `Duration` representing the wait time for the peer forwarder to complete processing data before shutdown. +## Configuration + +See the following for optional configuration values. + +### Optional configuration + +| Value | Description | +| ---- | ----------- | +| `port` | An `int` between 0 and 65535 represents the port peer forwarder server is running on. Default value is `4994`.| +| `request_timeout` | Duration - An `int` representing the request timeout in milliseconds for Peer Forwarder HTTP server. Default value is `10000`. | +| `server_thread_count` | An `int` representing number of threads used by Peer Forwarder server. Defaults to `200`.| +| `client_thread_count` | An `int` representing number of threads used by Peer Forwarder client. Defaults to `200`.| +| `maxConnectionCount` | An `int` representing maximum number of open connections for Peer Forwarder server. Default value is `500`. | +| `discovery_mode` | A `String` representing the peer discovery mode to be used. Allowable values are `local_node`, `static`, `dns`, and `aws_cloud_map`. Defaults to `local_node` which processes events locally. | +| `static_endpoints` | A `list` containing endpoints of all Data Prepper instances. Required if `discovery_mode` is set to `static`. | +| `domain_name` | A `String` representing single domain name to query DNS against. Typically, used by creating multiple [DNS A Records](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) for the same domain. Required if `discovery_mode` is set to `dns`. | +| `aws_cloud_map_namespace_name` | A `String` representing the Cloud Map namespace when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | +| `aws_cloud_map_service_name` | A `String` representing the Cloud Map service when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | +| `aws_cloud_map_query_parameters` | A `Map` of Key/value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. | +| `buffer_size` | An `int` representing max number of unchecked records the buffer accepts (num of unchecked records = num of records written into the buffer + num of in-flight records not yet checked by the Checkpointing API). Default is `512`. | +| `batch_size` | An `int` representing max number of records the buffer returns on read. Default is `48`. | +| `aws_region` | A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. | +| `drain_timeout` | A `Duration` representing the wait time for the peer forwarder to complete processing data before shutdown. | # SSL configuration The SSL configuration for setting up trust manager for the peer forwarding client to connect to other Data Prepper instances. From 2d1cf02a5b521f1e3f58d7aff1e9e5c620c456ca Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 9 Dec 2022 16:03:44 -0700 Subject: [PATCH 22/49] Minor adjustmenets. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 1f02954118..95de481620 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -4,11 +4,12 @@ title: Peer Forwarder nav_order: 12 --- +#Peer Forwarder + Peer Forwarder is an HTTP service which performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. -Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The value is configurable for the `aggregate` processor using the `identification_keys` configuration option. You can find more information about [identification keys](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). +Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The value is configurable for the `aggregate` processor using the `identification_keys` configuration option. You can find more information here about [identification keys](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). ---- Peer discovery is currently provided by either a static list or by a DNS record lookup or AWS Cloudmap. @@ -103,10 +104,10 @@ See the following for optional configuration values. | `aws_region` | A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. | | `drain_timeout` | A `Duration` representing the wait time for the peer forwarder to complete processing data before shutdown. | -# SSL configuration +## SSL configuration The SSL configuration for setting up trust manager for the peer forwarding client to connect to other Data Prepper instances. -## Optional SSL configuration +### Optional SSL configuration * `ssl`: A `boolean` that enables TLS/SSL. Default value is `true`. * `ssl_certificate_file`: A `String` representing the SSL certificate chain file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_certificate.pem` which is default certificate file. Read more about how the certificate file is generated [here](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates). From cb5134af34a4fb64d3690c478a426012ecb75749 Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 9 Dec 2022 16:05:19 -0700 Subject: [PATCH 23/49] Minor adjustments again. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 95de481620..fe5155f14a 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -4,7 +4,7 @@ title: Peer Forwarder nav_order: 12 --- -#Peer Forwarder +# Peer Forwarder Peer Forwarder is an HTTP service which performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. @@ -13,11 +13,11 @@ Peer Forwarder groups events based on the identification keys provided by the pr Peer discovery is currently provided by either a static list or by a DNS record lookup or AWS Cloudmap. -# Discovery modes +## Discovery modes See the following information about discovery modes below. -## Static +### Static Static discover mode allows a Data Prepper node to discover nodes using a list of IP addresses or domain names. ```yaml @@ -26,7 +26,7 @@ peer_forwarder:4 static_endpoints: ["data-prepper1", "data-prepper2"] ``` -## DNS lookup +### DNS lookup We recommend using DNS discovery over static discovery when scaling out a Data Prepper cluster. The core concept is to configure a DNS provider to return a list of Data Prepper hosts when given a single domain name. This is a [DNS A Record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) which indicates a list of IP addresses of a given domain. ```yaml @@ -35,7 +35,7 @@ peer_forwarder: domain_name: "data-prepper-cluster.my-domain.net" ``` -## AWS Cloud Map +### AWS Cloud Map [AWS Cloud Map](https://docs.aws.amazon.com/cloud-map/latest/dg/what-is-cloud-map.html) provides API-based service discovery as well as DNS-based service discovery. @@ -127,7 +127,7 @@ peer_forwarder: ssl_key_file: "" ``` -# Authentication +## Authentication * `authentication`(Optional) : A `Map` that enables mTLS. It can either be `mutual_tls` or `unauthenticated`. Default value is `unauthenticated`. ```yaml @@ -136,16 +136,16 @@ peer_forwarder: mutual_tls: ``` -# Metrics +## Metrics Core Peer Forwarder introduces the following custom metrics and all the metrics are prefixed by `core.peerForwarder`. -# Timer +### Timer - `requestForwardingLatency`: measures latency of forwarding requests by peer forwarder client. - `requestProcessingLatency`: measures latency of processing requests by peer forwarder server. -# Counter +### Counter - `requests`: measures total number of forwarded requests. - `requestsFailed`: measures total number of failed requests. Requests with HTTP response code other than `200`. @@ -161,6 +161,6 @@ Core Peer Forwarder introduces the following custom metrics and all the metrics - `recordsActuallyProcessedLocally`: measures total number of records actually processed locally. Sum of `recordsToBeProcessedLocally` and `recordsFailedForwarding`. - `recordsReceivedFromPeers`: measures total number of records received from remote peers. -## Gauge +### Gauge - `peerEndpoints`: measures number of dynamically discovered peer data-prepper endpoints. For `static` mode, the size is fixed. From 0b4a055e0a7894726726818b3c1504254bbcc39d Mon Sep 17 00:00:00 2001 From: carolxob Date: Mon, 12 Dec 2022 11:32:14 -0700 Subject: [PATCH 24/49] Updates from doc review feedback. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 73 ++++++++++++++++++++------------- 1 file changed, 45 insertions(+), 28 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index fe5155f14a..ff5a090e23 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -6,9 +6,9 @@ nav_order: 12 # Peer Forwarder -Peer Forwarder is an HTTP service which performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. +Peer Forwarder is an HTTP service which performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, Peer Forwarder is supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. -Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The value is configurable for the `aggregate` processor using the `identification_keys` configuration option. You can find more information here about [identification keys](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). +Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The value is configurable for the `aggregate` processor using the `identification_keys` configuration option. You can find more information about [identification keys](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) on the aggregate processor page. Peer discovery is currently provided by either a static list or by a DNS record lookup or AWS Cloudmap. @@ -64,7 +64,7 @@ peer_forwarder: ### IAM policy with necessary permissions -The Data Prepper must also be running with the necessary permissions. The following IAM policy shows the necessary permissions: +The Data Prepper must also be running with the necessary permissions. The following IAM policy shows the necessary permissions. ```json { @@ -109,16 +109,20 @@ The SSL configuration for setting up trust manager for the peer forwarding clien ### Optional SSL configuration -* `ssl`: A `boolean` that enables TLS/SSL. Default value is `true`. -* `ssl_certificate_file`: A `String` representing the SSL certificate chain file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_certificate.pem` which is default certificate file. Read more about how the certificate file is generated [here](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates). -* `ssl_key_file`: A `String` represents the SSL key file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_private_key.pem` which is default private key file. Read more about how the private key file is generated [here](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates). -* `ssl_insecure_disable_verification`: A `boolean` that disables the verification of server's TLS certificate chain. Default value is `false`. -* `ssl_fingerprint_verification_only`: A `boolean` that disables the verification of server's TLS certificate chain and instead verifies only the certificate fingerprint. Default value is `false`. -* `use_acm_certificate_for_ssl`: A `boolean` that enables TLS/SSL using certificate and private key from AWS Certificate Manager (ACM). Default is `false`. -* `acm_certificate_arn`: A `String` represents the ACM certificate ARN. ACM certificate take preference over S3 or local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. -* `acm_private_key_password`: A `String` that represents the ACM private key password which that will be used to decrypt the private key. If it's not provided, a random password will be generated. -* `acm_certificate_timeout_millis`: An `int` representing the timeout in milliseconds for ACM to get certificates. Default value is `120000`. -* `aws_region`: A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. +See the table below for optional SSL configuration descriptions. + +| Value | Description | +| ----- | ----------- | +| `ssl` | A `boolean` that enables TLS/SSL. Default value is `true`. | +| `ssl_certificate_file`| A `String` representing the SSL certificate chain file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_certificate.pem` which is default certificate file. Read more about how the certificate file is generated [here](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates). | +| `ssl_key_file`| A `String` represents the SSL key file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_private_key.pem` which is default private key file. Read more about how the private key file is generated [here](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates). | +| `ssl_insecure_disable_verification` | A `boolean` that disables the verification of server's TLS certificate chain. Default value is `false`. | +| `ssl_fingerprint_verification_only` | A `boolean` that disables the verification of server's TLS certificate chain and instead verifies only the certificate fingerprint. Default value is `false`. | +| `use_acm_certificate_for_ssl` | A `boolean` that enables TLS/SSL using certificate and private key from AWS Certificate Manager (ACM). Default is `false`. | +| `acm_certificate_arn`| A `String` represents the ACM certificate ARN. ACM certificate take preference over S3 or local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. | +| `acm_private_key_password` | A `String` that represents the ACM private key password which that will be used to decrypt the private key. If it's not provided, a random password will be generated. | +| `acm_certificate_timeout_millis` | An `int` representing the timeout in milliseconds for ACM to get certificates. Default value is `120000`. | +| `aws_region` | A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. | ```yaml peer_forwarder: @@ -129,7 +133,10 @@ peer_forwarder: ## Authentication -* `authentication`(Optional) : A `Map` that enables mTLS. It can either be `mutual_tls` or `unauthenticated`. Default value is `unauthenticated`. +This section describes optional authentication. + +`authentication`(optional) : A `Map` that enables mTLS. It can either be `mutual_tls` or `unauthenticated`. The default value is `unauthenticated`. + ```yaml peer_forwarder: authentication: @@ -138,29 +145,39 @@ peer_forwarder: ## Metrics +This section describes custom metrics capabilities, including timer, counter, and gauge. + Core Peer Forwarder introduces the following custom metrics and all the metrics are prefixed by `core.peerForwarder`. ### Timer +This section describes the metrics feature, timer. + - `requestForwardingLatency`: measures latency of forwarding requests by peer forwarder client. - `requestProcessingLatency`: measures latency of processing requests by peer forwarder server. ### Counter -- `requests`: measures total number of forwarded requests. -- `requestsFailed`: measures total number of failed requests. Requests with HTTP response code other than `200`. -- `requestsSuccessful`: measures total number of successful requests. Requests with HTTP response code `200`. -- `requestsTooLarge`: measures total number of requests which are too large to be written to peer forwarder buffer. Requests with HTTP response code `413`. -- `requestTimeouts`: measures the total number of requests which timed out while writing content to peer forwarder buffer. Requests with HTTP response code `408`. -- `requestsUnprocessable`: measures total number of requests which failed due to unprocessable entity. Requests with HTTP response code `422`. -- `badRequests`: measures total number of requests with bad request format. Requests with HTTP response code `400`. -- `recordsSuccessfullyForwarded`: measures total number of forwarded records successfully. -- `recordsFailedForwarding`: measures total number of records failed to be forwarded. -- `recordsToBeForwarded`: measures total number of records to be forwarded. -- `recordsToBeProcessedLocally`: measures total number of records to be processed locally. -- `recordsActuallyProcessedLocally`: measures total number of records actually processed locally. Sum of `recordsToBeProcessedLocally` and `recordsFailedForwarding`. -- `recordsReceivedFromPeers`: measures total number of records received from remote peers. +See the table below for counter metric options. + +| Value | Description | +| ----- | ----------- | +| `requests`| measures total number of forwarded requests. | +| `requestsFailed`| measures total number of failed requests. Requests with HTTP response code other than `200`. | +| `requestsSuccessful`| measures total number of successful requests. Requests with HTTP response code `200`. | +| `requestsTooLarge`| measures total number of requests which are too large to be written to peer forwarder buffer. Requests with HTTP response code `413`. | +| `requestTimeouts`| measures the total number of requests which timed out while writing content to peer forwarder buffer. Requests with HTTP response code `408`. | +| `requestsUnprocessable`| measures total number of requests which failed due to unprocessable entity. Requests with HTTP response code `422`. | +| `badRequests`| measures total number of requests with bad request format. Requests with HTTP response code `400`. | +| `recordsSuccessfullyForwarded`| measures total number of forwarded records successfully. | +| `recordsFailedForwarding`| measures total number of records failed to be forwarded. | +| `recordsToBeForwarded` | measures total number of records to be forwarded. | +| `recordsToBeProcessedLocally` | measures total number of records to be processed locally. | +| `recordsActuallyProcessedLocally`| measures total number of records actually processed locally. Sum of recordsToBeProcessedLocally` and `recordsFailedForwarding`. | +| `recordsReceivedFromPeers`| measures total number of records received from remote peers. | ### Gauge -- `peerEndpoints`: measures number of dynamically discovered peer data-prepper endpoints. For `static` mode, the size is fixed. +This section describes gauge metric options. + +`peerEndpoints` measures number of dynamically discovered peer data-prepper endpoints. For `static` mode, the size is fixed. From 850625f01f50d87399da174267f29c3719d2c3fb Mon Sep 17 00:00:00 2001 From: carolxob Date: Mon, 12 Dec 2022 15:04:17 -0700 Subject: [PATCH 25/49] Made changes based on doc review feedback. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index ff5a090e23..2a5f4b6eca 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -6,12 +6,12 @@ nav_order: 12 # Peer Forwarder -Peer Forwarder is an HTTP service which performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, Peer Forwarder is supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. +Peer Forwarder is an HTTP service that performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, Peer Forwarder is supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. -Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The value is configurable for the `aggregate` processor using the `identification_keys` configuration option. You can find more information about [identification keys](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) on the aggregate processor page. +Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The `aggregate` processor is configured using the `identification_keys` configuration option. From here, you can specify which keys to use for Peer Forwarder. You can find more information about [identification keys](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) on the aggregate processor page. -Peer discovery is currently provided by either a static list or by a DNS record lookup or AWS Cloudmap. +Peer discovery is currently provided by either a static list or by a DNS record lookup or AWS Cloud Map. ## Discovery modes @@ -19,7 +19,7 @@ See the following information about discovery modes below. ### Static -Static discover mode allows a Data Prepper node to discover nodes using a list of IP addresses or domain names. +Static discovery mode allows a Data Prepper node to discover nodes using a list of IP addresses or domain names. ```yaml peer_forwarder:4 discovery_mode: static @@ -27,7 +27,7 @@ peer_forwarder:4 ``` ### DNS lookup -We recommend using DNS discovery over static discovery when scaling out a Data Prepper cluster. The core concept is to configure a DNS provider to return a list of Data Prepper hosts when given a single domain name. This is a [DNS A Record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) which indicates a list of IP addresses of a given domain. +DNS discovery is preferred over static discovery when scaling out a Data Prepper cluster. The core concept is to configure a DNS provider to return a list of Data Prepper hosts when given a single domain name. This is a [DNS A Record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/), which indicates a list of IP addresses of a given domain. ```yaml peer_forwarder: @@ -39,7 +39,7 @@ peer_forwarder: [AWS Cloud Map](https://docs.aws.amazon.com/cloud-map/latest/dg/what-is-cloud-map.html) provides API-based service discovery as well as DNS-based service discovery. -Peer forwarder can use the API-based service discovery. To support this you must have an existing namespace configured for API instance discovery. You can create a new one following the instructions provided by the [Cloud Map documentation](https://docs.aws.amazon.com/cloud-map/latest/dg/working-with-namespaces.html). +Peer Forwarder can use the API-based service discovery. To support this you must have an existing namespace configured for API instance discovery. You can create a new one following the instructions provided by the [Cloud Map documentation](https://docs.aws.amazon.com/cloud-map/latest/dg/working-with-namespaces.html). Your Data Prepper configuration needs to include: * `aws_cloud_map_namespace_name` - Set to your Cloud Map Namespace name @@ -145,7 +145,7 @@ peer_forwarder: ## Metrics -This section describes custom metrics capabilities, including timer, counter, and gauge. +This section describes Peer Forwarder's metrics capabilities, including timer, counter, and gauge. Core Peer Forwarder introduces the following custom metrics and all the metrics are prefixed by `core.peerForwarder`. @@ -165,19 +165,19 @@ See the table below for counter metric options. | `requests`| measures total number of forwarded requests. | | `requestsFailed`| measures total number of failed requests. Requests with HTTP response code other than `200`. | | `requestsSuccessful`| measures total number of successful requests. Requests with HTTP response code `200`. | -| `requestsTooLarge`| measures total number of requests which are too large to be written to peer forwarder buffer. Requests with HTTP response code `413`. | -| `requestTimeouts`| measures the total number of requests which timed out while writing content to peer forwarder buffer. Requests with HTTP response code `408`. | -| `requestsUnprocessable`| measures total number of requests which failed due to unprocessable entity. Requests with HTTP response code `422`. | +| `requestsTooLarge`| measures total number of requests that are too large to be written to Peer Forwarder buffer. Requests with HTTP response code `413`. | +| `requestTimeouts`| measures the total number of requests that timed out while writing content to peer forwarder buffer. Requests with HTTP response code `408`. | +| `requestsUnprocessable`| measures total number of requests that failed due to unprocessable entity. Requests with HTTP response code `422`. | | `badRequests`| measures total number of requests with bad request format. Requests with HTTP response code `400`. | | `recordsSuccessfullyForwarded`| measures total number of forwarded records successfully. | | `recordsFailedForwarding`| measures total number of records failed to be forwarded. | | `recordsToBeForwarded` | measures total number of records to be forwarded. | | `recordsToBeProcessedLocally` | measures total number of records to be processed locally. | -| `recordsActuallyProcessedLocally`| measures total number of records actually processed locally. Sum of recordsToBeProcessedLocally` and `recordsFailedForwarding`. | +| `recordsActuallyProcessedLocally`| measures total number of records actually processed locally. Sum of `recordsToBeProcessedLocally` and `recordsFailedForwarding`. | | `recordsReceivedFromPeers`| measures total number of records received from remote peers. | ### Gauge This section describes gauge metric options. -`peerEndpoints` measures number of dynamically discovered peer data-prepper endpoints. For `static` mode, the size is fixed. +`peerEndpoints` measures number of dynamically discovered peer Data Prepper endpoints. For `static` mode, the size is fixed. From 588da3f4ac65a229471de9fdf5968dcb6cdeabe9 Mon Sep 17 00:00:00 2001 From: carolxob Date: Tue, 13 Dec 2022 10:49:24 -0700 Subject: [PATCH 26/49] Made minor heading adjustements. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 2a5f4b6eca..fb4d3493c7 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -20,6 +20,7 @@ See the following information about discovery modes below. ### Static Static discovery mode allows a Data Prepper node to discover nodes using a list of IP addresses or domain names. + ```yaml peer_forwarder:4 discovery_mode: static @@ -50,7 +51,7 @@ Your Data Prepper configuration needs to include: Your Data Prepper configuration can optionally include: * `aws_cloud_map_query_parameters` - Key/value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. -### Example configuration +#### Example configuration ```yaml peer_forwarder: @@ -124,6 +125,8 @@ See the table below for optional SSL configuration descriptions. | `acm_certificate_timeout_millis` | An `int` representing the timeout in milliseconds for ACM to get certificates. Default value is `120000`. | | `aws_region` | A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. | +#### Example config + ```yaml peer_forwarder: ssl: true From 81e08ed7eea52b4339a6e08f2b7fe807382c98bf Mon Sep 17 00:00:00 2001 From: carolxob Date: Tue, 13 Dec 2022 11:54:34 -0700 Subject: [PATCH 27/49] Made edits based on doc review feedback. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 64 ++++++++++++++++----------------- 1 file changed, 32 insertions(+), 32 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index fb4d3493c7..e9be7c2f0e 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -40,16 +40,16 @@ peer_forwarder: [AWS Cloud Map](https://docs.aws.amazon.com/cloud-map/latest/dg/what-is-cloud-map.html) provides API-based service discovery as well as DNS-based service discovery. -Peer Forwarder can use the API-based service discovery. To support this you must have an existing namespace configured for API instance discovery. You can create a new one following the instructions provided by the [Cloud Map documentation](https://docs.aws.amazon.com/cloud-map/latest/dg/working-with-namespaces.html). +Peer Forwarder can use the API-based service discovery. To support this, you must have an existing Namespace configured for API instance discovery. You can create a new one following the instructions provided by the [Cloud Map documentation](https://docs.aws.amazon.com/cloud-map/latest/dg/working-with-namespaces.html). Your Data Prepper configuration needs to include: * `aws_cloud_map_namespace_name` - Set to your Cloud Map Namespace name * `aws_cloud_map_service_name` - Set to the service name within your specified Namespace -* `aws_region` - The AWS region where your namespace exists. +* `aws_region` - The AWS Region where your Namespace exists. * `discovery_mode` - Set to `aws_cloud_map` Your Data Prepper configuration can optionally include: -* `aws_cloud_map_query_parameters` - Key/value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. +* `aws_cloud_map_query_parameters` - Key-value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. #### Example configuration @@ -65,7 +65,7 @@ peer_forwarder: ### IAM policy with necessary permissions -The Data Prepper must also be running with the necessary permissions. The following IAM policy shows the necessary permissions. +The Data Prepper must also be running with the necessary permissions. The following IAM policy shows the necessary permissions: ```json { @@ -89,21 +89,21 @@ See the following for optional configuration values. | Value | Description | | ---- | ----------- | -| `port` | An `int` between 0 and 65535 represents the port peer forwarder server is running on. Default value is `4994`.| +| `port` | An `int` between 0 and 65535 represents the port Peer Forwarder server is running on. Default value is `4994`.| | `request_timeout` | Duration - An `int` representing the request timeout in milliseconds for Peer Forwarder HTTP server. Default value is `10000`. | -| `server_thread_count` | An `int` representing number of threads used by Peer Forwarder server. Defaults to `200`.| -| `client_thread_count` | An `int` representing number of threads used by Peer Forwarder client. Defaults to `200`.| +| `server_thread_count` | An `int` representing number of threads used by the Peer Forwarder server. Defaults to `200`.| +| `client_thread_count` | An `int` representing number of threads used by the Peer Forwarder client. Defaults to `200`.| | `maxConnectionCount` | An `int` representing maximum number of open connections for Peer Forwarder server. Default value is `500`. | | `discovery_mode` | A `String` representing the peer discovery mode to be used. Allowable values are `local_node`, `static`, `dns`, and `aws_cloud_map`. Defaults to `local_node` which processes events locally. | | `static_endpoints` | A `list` containing endpoints of all Data Prepper instances. Required if `discovery_mode` is set to `static`. | | `domain_name` | A `String` representing single domain name to query DNS against. Typically, used by creating multiple [DNS A Records](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) for the same domain. Required if `discovery_mode` is set to `dns`. | -| `aws_cloud_map_namespace_name` | A `String` representing the Cloud Map namespace when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | +| `aws_cloud_map_namespace_name` | A `String` representing the Cloud Map Namespace when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | | `aws_cloud_map_service_name` | A `String` representing the Cloud Map service when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | | `aws_cloud_map_query_parameters` | A `Map` of Key/value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. | -| `buffer_size` | An `int` representing max number of unchecked records the buffer accepts (num of unchecked records = num of records written into the buffer + num of in-flight records not yet checked by the Checkpointing API). Default is `512`. | +| `buffer_size` | An `int` representing max number of unchecked records the buffer accepts (num of unchecked records equals the number of records written into the buffer + the number of in-flight records not yet checked by the Checkpointing API). Default is `512`. | | `batch_size` | An `int` representing max number of records the buffer returns on read. Default is `48`. | -| `aws_region` | A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. | -| `drain_timeout` | A `Duration` representing the wait time for the peer forwarder to complete processing data before shutdown. | +| `aws_region` | A `String` represents the AWS region to use `ACM`, `S3`, or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path, or if `discovery_mode` is set to `aws_cloud_map`. | +| `drain_timeout` | A `Duration` representing the wait time for the Peer Forwarder to complete processing data before shutdown. | ## SSL configuration The SSL configuration for setting up trust manager for the peer forwarding client to connect to other Data Prepper instances. @@ -115,15 +115,15 @@ See the table below for optional SSL configuration descriptions. | Value | Description | | ----- | ----------- | | `ssl` | A `boolean` that enables TLS/SSL. Default value is `true`. | -| `ssl_certificate_file`| A `String` representing the SSL certificate chain file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_certificate.pem` which is default certificate file. Read more about how the certificate file is generated [here](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates). | -| `ssl_key_file`| A `String` represents the SSL key file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_private_key.pem` which is default private key file. Read more about how the private key file is generated [here](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates). | +| `ssl_certificate_file`| A `String` representing the SSL certificate chain file path or AWS S3 path. S3 path example `s3:///`. Defaults to the default certificate file,`config/default_certificate.pem`. Read more about how the certificate file is generated at the [Default Certificates](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates) page. | +| `ssl_key_file`| A `String` represents the SSL key file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_private_key.pem` which is default private key file. Read more about how the private key file is generated at the [Default Certificates](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates) page. | | `ssl_insecure_disable_verification` | A `boolean` that disables the verification of server's TLS certificate chain. Default value is `false`. | | `ssl_fingerprint_verification_only` | A `boolean` that disables the verification of server's TLS certificate chain and instead verifies only the certificate fingerprint. Default value is `false`. | | `use_acm_certificate_for_ssl` | A `boolean` that enables TLS/SSL using certificate and private key from AWS Certificate Manager (ACM). Default is `false`. | | `acm_certificate_arn`| A `String` represents the ACM certificate ARN. ACM certificate take preference over S3 or local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. | -| `acm_private_key_password` | A `String` that represents the ACM private key password which that will be used to decrypt the private key. If it's not provided, a random password will be generated. | +| `acm_private_key_password` | A `String` that represents the ACM private key password that will be used to decrypt the private key. If it's not provided, a random password will be generated. | | `acm_certificate_timeout_millis` | An `int` representing the timeout in milliseconds for ACM to get certificates. Default value is `120000`. | -| `aws_region` | A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. | +| `aws_region` | A `String` represents the AWS Region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. | #### Example config @@ -154,10 +154,10 @@ Core Peer Forwarder introduces the following custom metrics and all the metrics ### Timer -This section describes the metrics feature, timer. +This section describes the timer metrics feature. -- `requestForwardingLatency`: measures latency of forwarding requests by peer forwarder client. -- `requestProcessingLatency`: measures latency of processing requests by peer forwarder server. +- `requestForwardingLatency`: measures latency of forwarding requests by the Peer Forwarder client. +- `requestProcessingLatency`: measures latency of processing requests by Peer Forwarder server. ### Counter @@ -165,22 +165,22 @@ See the table below for counter metric options. | Value | Description | | ----- | ----------- | -| `requests`| measures total number of forwarded requests. | -| `requestsFailed`| measures total number of failed requests. Requests with HTTP response code other than `200`. | -| `requestsSuccessful`| measures total number of successful requests. Requests with HTTP response code `200`. | -| `requestsTooLarge`| measures total number of requests that are too large to be written to Peer Forwarder buffer. Requests with HTTP response code `413`. | -| `requestTimeouts`| measures the total number of requests that timed out while writing content to peer forwarder buffer. Requests with HTTP response code `408`. | -| `requestsUnprocessable`| measures total number of requests that failed due to unprocessable entity. Requests with HTTP response code `422`. | -| `badRequests`| measures total number of requests with bad request format. Requests with HTTP response code `400`. | -| `recordsSuccessfullyForwarded`| measures total number of forwarded records successfully. | -| `recordsFailedForwarding`| measures total number of records failed to be forwarded. | -| `recordsToBeForwarded` | measures total number of records to be forwarded. | -| `recordsToBeProcessedLocally` | measures total number of records to be processed locally. | -| `recordsActuallyProcessedLocally`| measures total number of records actually processed locally. Sum of `recordsToBeProcessedLocally` and `recordsFailedForwarding`. | -| `recordsReceivedFromPeers`| measures total number of records received from remote peers. | +| `requests`| Measures total number of forwarded requests. | +| `requestsFailed`| Measures total number of failed requests. Requests with HTTP response code other than `200`. | +| `requestsSuccessful`| Measures total number of successful requests. Requests with HTTP response code `200`. | +| `requestsTooLarge`| Measures total number of requests that are too large to be written to Peer Forwarder buffer. Requests with HTTP response code `413`. | +| `requestTimeouts`| Measures the total number of requests that timed out while writing content to Peer Forwarder buffer. Requests with HTTP response code `408`. | +| `requestsUnprocessable`| Measures total number of requests that failed due to unprocessable entity. Requests with HTTP response code `422`. | +| `badRequests`| Measures total number of requests with bad request format. Requests with HTTP response code `400`. | +| `recordsSuccessfullyForwarded`| Measures total number of forwarded records successfully. | +| `recordsFailedForwarding`| Measures total number of records failed to be forwarded. | +| `recordsToBeForwarded` | Measures total number of records to be forwarded. | +| `recordsToBeProcessedLocally` | Measures total number of records to be processed locally. | +| `recordsActuallyProcessedLocally`| Measures total number of records actually processed locally. Sum of `recordsToBeProcessedLocally` and `recordsFailedForwarding`. | +| `recordsReceivedFromPeers`| Measures total number of records received from remote peers. | ### Gauge This section describes gauge metric options. -`peerEndpoints` measures number of dynamically discovered peer Data Prepper endpoints. For `static` mode, the size is fixed. +`peerEndpoints` Measures number of dynamically discovered peer Data Prepper endpoints. For `static` mode, the size is fixed. From 9f5313eb1d2ef54e70b0927bab66c599a5d71142 Mon Sep 17 00:00:00 2001 From: carolxob Date: Wed, 14 Dec 2022 19:04:14 -0700 Subject: [PATCH 28/49] Made updates based on editorial feedback. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 40 +++++++++++++++------------------ 1 file changed, 18 insertions(+), 22 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index e9be7c2f0e..b38aa6112f 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -6,20 +6,19 @@ nav_order: 12 # Peer Forwarder -Peer Forwarder is an HTTP service that performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, Peer Forwarder is supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. +Peer Forwarder is an HTTP service that performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, Peer Forwarder is supported by the `aggregate`, `service_map_stateful`, and `otel_trace_raw` processors. Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The `aggregate` processor is configured using the `identification_keys` configuration option. From here, you can specify which keys to use for Peer Forwarder. You can find more information about [identification keys](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) on the aggregate processor page. - -Peer discovery is currently provided by either a static list or by a DNS record lookup or AWS Cloud Map. +Peer discovery is currently provided by a static list, a DNS record lookup, or AWS Cloud Map. ## Discovery modes -See the following information about discovery modes below. +The following sections provide information about discovery modes. ### Static -Static discovery mode allows a Data Prepper node to discover nodes using a list of IP addresses or domain names. +Static discovery mode allows a Data Prepper node to discover nodes using a list of IP addresses or domain names. See the following yaml file for an example of static discovery mode: ```yaml peer_forwarder:4 @@ -28,7 +27,8 @@ peer_forwarder:4 ``` ### DNS lookup -DNS discovery is preferred over static discovery when scaling out a Data Prepper cluster. The core concept is to configure a DNS provider to return a list of Data Prepper hosts when given a single domain name. This is a [DNS A Record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/), which indicates a list of IP addresses of a given domain. + +DNS discovery is preferred over static discovery when scaling out a Data Prepper cluster. The core concept is to configure a DNS provider to return a list of Data Prepper hosts when given a single domain name. This is a [DNS A Record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/), which indicates a list of IP addresses of a given domain. See the following yaml file for an example of DNS lookup: ```yaml peer_forwarder: @@ -154,8 +154,6 @@ Core Peer Forwarder introduces the following custom metrics and all the metrics ### Timer -This section describes the timer metrics feature. - - `requestForwardingLatency`: measures latency of forwarding requests by the Peer Forwarder client. - `requestProcessingLatency`: measures latency of processing requests by Peer Forwarder server. @@ -165,22 +163,20 @@ See the table below for counter metric options. | Value | Description | | ----- | ----------- | -| `requests`| Measures total number of forwarded requests. | -| `requestsFailed`| Measures total number of failed requests. Requests with HTTP response code other than `200`. | -| `requestsSuccessful`| Measures total number of successful requests. Requests with HTTP response code `200`. | -| `requestsTooLarge`| Measures total number of requests that are too large to be written to Peer Forwarder buffer. Requests with HTTP response code `413`. | +| `requests`| Measures the total number of forwarded requests. | +| `requestsFailed`| Measures the total number of failed requests. Requests with HTTP response code other than `200`. | +| `requestsSuccessful`| Measures the total number of successful requests. Requests with HTTP response code `200`. | +| `requestsTooLarge`| Measures the total number of requests that are too large to be written to Peer Forwarder buffer. Requests with HTTP response code `413`. | | `requestTimeouts`| Measures the total number of requests that timed out while writing content to Peer Forwarder buffer. Requests with HTTP response code `408`. | -| `requestsUnprocessable`| Measures total number of requests that failed due to unprocessable entity. Requests with HTTP response code `422`. | -| `badRequests`| Measures total number of requests with bad request format. Requests with HTTP response code `400`. | -| `recordsSuccessfullyForwarded`| Measures total number of forwarded records successfully. | -| `recordsFailedForwarding`| Measures total number of records failed to be forwarded. | -| `recordsToBeForwarded` | Measures total number of records to be forwarded. | -| `recordsToBeProcessedLocally` | Measures total number of records to be processed locally. | -| `recordsActuallyProcessedLocally`| Measures total number of records actually processed locally. Sum of `recordsToBeProcessedLocally` and `recordsFailedForwarding`. | -| `recordsReceivedFromPeers`| Measures total number of records received from remote peers. | +| `requestsUnprocessable`| Measures the total number of requests that failed due to unprocessable entity. Requests with HTTP response code `422`. | +| `badRequests`| Measures the total number of requests with bad request format. Requests with HTTP response code `400`. | +| `recordsSuccessfullyForwarded`| Measures the total number of forwarded records successfully. | +| `recordsFailedForwarding`| Measures the total number of records failed to be forwarded. | +| `recordsToBeForwarded` | Measures the total number of records to be forwarded. | +| `recordsToBeProcessedLocally` | Measures the total number of records to be processed locally. | +| `recordsActuallyProcessedLocally`| Measures the total number of records actually processed locally. This value is the sum of `recordsToBeProcessedLocally` and `recordsFailedForwarding`. | +| `recordsReceivedFromPeers`| Measures the total number of records received from remote peers. | ### Gauge -This section describes gauge metric options. - `peerEndpoints` Measures number of dynamically discovered peer Data Prepper endpoints. For `static` mode, the size is fixed. From 156ddc7d15db9f2975c996a05185ce37799dfe65 Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 16 Dec 2022 16:01:26 -0700 Subject: [PATCH 29/49] Made extensive changes based on editorial feedback. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 90 ++++++++++++++++----------------- 1 file changed, 45 insertions(+), 45 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index b38aa6112f..769929c995 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -8,7 +8,7 @@ nav_order: 12 Peer Forwarder is an HTTP service that performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, Peer Forwarder is supported by the `aggregate`, `service_map_stateful`, and `otel_trace_raw` processors. -Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The `aggregate` processor is configured using the `identification_keys` configuration option. From here, you can specify which keys to use for Peer Forwarder. You can find more information about [identification keys](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) on the aggregate processor page. +Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The `aggregate` processor is configured using the `identification_keys` configuration option. From here, you can specify which keys to use for Peer Forwarder. See [Aggregate Processor Page](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) for more information about identification keys. Peer discovery is currently provided by a static list, a DNS record lookup, or AWS Cloud Map. @@ -28,7 +28,7 @@ peer_forwarder:4 ### DNS lookup -DNS discovery is preferred over static discovery when scaling out a Data Prepper cluster. The core concept is to configure a DNS provider to return a list of Data Prepper hosts when given a single domain name. This is a [DNS A Record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/), which indicates a list of IP addresses of a given domain. See the following yaml file for an example of DNS lookup: +DNS discovery is preferred over static discovery when scaling out a Data Prepper cluster. The core concept is to configure a DNS provider to return a list of Data Prepper hosts when given a single domain name. This is a [DNS A Record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/), which indicates a list of IP addresses of a given domain. See the following yaml file example of DNS lookup: ```yaml peer_forwarder: @@ -40,19 +40,21 @@ peer_forwarder: [AWS Cloud Map](https://docs.aws.amazon.com/cloud-map/latest/dg/what-is-cloud-map.html) provides API-based service discovery as well as DNS-based service discovery. -Peer Forwarder can use the API-based service discovery. To support this, you must have an existing Namespace configured for API instance discovery. You can create a new one following the instructions provided by the [Cloud Map documentation](https://docs.aws.amazon.com/cloud-map/latest/dg/working-with-namespaces.html). +Peer Forwarder can use the API-based service discovery. To support this, you must have an existing namespace configured for API instance discovery. You can create a new one by following the instructions provided by the [AWS Cloud Map documentation](https://docs.aws.amazon.com/cloud-map/latest/dg/working-with-namespaces.html). -Your Data Prepper configuration needs to include: -* `aws_cloud_map_namespace_name` - Set to your Cloud Map Namespace name -* `aws_cloud_map_service_name` - Set to the service name within your specified Namespace -* `aws_region` - The AWS Region where your Namespace exists. -* `discovery_mode` - Set to `aws_cloud_map` +Your Data Prepper configuration needs to include the following: +* `aws_cloud_map_namespace_name` – Set to your AWS Cloud Map namespace name. +* `aws_cloud_map_service_name` – Set to the service name within your specified namespace. +* `aws_region` – Set to the AWS Region where your namespace exists. +* `discovery_mode` – Set to `aws_cloud_map`. -Your Data Prepper configuration can optionally include: -* `aws_cloud_map_query_parameters` - Key-value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. +Your Data Prepper configuration can optionally include the following: +* `aws_cloud_map_query_parameters` – Key-value pairs are used to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. #### Example configuration +See the following yaml file example of AWS Cloud Map configuration: + ```yaml peer_forwarder: discovery_mode: aws_cloud_map @@ -65,7 +67,7 @@ peer_forwarder: ### IAM policy with necessary permissions -The Data Prepper must also be running with the necessary permissions. The following IAM policy shows the necessary permissions: +Data Prepper must also be running with the necessary permissions. The following AWS Identity and Access Management (IAM) policy shows the necessary permissions: ```json { @@ -83,49 +85,47 @@ The Data Prepper must also be running with the necessary permissions. The follow --- ## Configuration -See the following for optional configuration values. +The following table provides optional configuration values. -### Optional configuration | Value | Description | | ---- | ----------- | -| `port` | An `int` between 0 and 65535 represents the port Peer Forwarder server is running on. Default value is `4994`.| -| `request_timeout` | Duration - An `int` representing the request timeout in milliseconds for Peer Forwarder HTTP server. Default value is `10000`. | -| `server_thread_count` | An `int` representing number of threads used by the Peer Forwarder server. Defaults to `200`.| -| `client_thread_count` | An `int` representing number of threads used by the Peer Forwarder client. Defaults to `200`.| -| `maxConnectionCount` | An `int` representing maximum number of open connections for Peer Forwarder server. Default value is `500`. | -| `discovery_mode` | A `String` representing the peer discovery mode to be used. Allowable values are `local_node`, `static`, `dns`, and `aws_cloud_map`. Defaults to `local_node` which processes events locally. | -| `static_endpoints` | A `list` containing endpoints of all Data Prepper instances. Required if `discovery_mode` is set to `static`. | -| `domain_name` | A `String` representing single domain name to query DNS against. Typically, used by creating multiple [DNS A Records](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) for the same domain. Required if `discovery_mode` is set to `dns`. | -| `aws_cloud_map_namespace_name` | A `String` representing the Cloud Map Namespace when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | +| `port` | An `int` between 0 and 65535 represents the port that the Peer Forwarder server is running on. Default value is `4994`. | +| `request_timeout` | An `int` representing the request timeout duration in milliseconds for the Peer Forwarder HTTP server. Default value is `10000`. | +| `server_thread_count` | An `int` representing the number of threads used by the Peer Forwarder server. Default value is `200`.| +| `client_thread_count` | An `int` representing the number of threads used by the Peer Forwarder client. Default value is `200`.| +| `maxConnectionCount` | An `int` representing the maximum number of open connections for Peer Forwarder server. Default value is `500`. | +| `discovery_mode` | A `String` representing the peer discovery mode to be used. Allowable values are `local_node`, `static`, `dns`, and `aws_cloud_map`. Defaults to `local_node`, which processes events locally. | +| `static_endpoints` | A `list` containing the endpoints of all Data Prepper instances. Required if `discovery_mode` is set to `static`. | +| `domain_name` | A `String` representing the single domain name to query DNS against. Typically, used by creating multiple [DNS A records](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) for the same domain. Required if `discovery_mode` is set to `dns`. | +| `aws_cloud_map_namespace_name` | A `String` representing the AWS Cloud Map namespace when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | | `aws_cloud_map_service_name` | A `String` representing the Cloud Map service when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | | `aws_cloud_map_query_parameters` | A `Map` of Key/value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. | -| `buffer_size` | An `int` representing max number of unchecked records the buffer accepts (num of unchecked records equals the number of records written into the buffer + the number of in-flight records not yet checked by the Checkpointing API). Default is `512`. | +| `buffer_size` | An `int` representing the maximum number of unchecked records the buffer accepts (the number of unchecked records equals the number of records written into the buffer plus the number of in-flight records not yet checked by the Checkpointing API). Default is `512`. | | `batch_size` | An `int` representing max number of records the buffer returns on read. Default is `48`. | -| `aws_region` | A `String` represents the AWS region to use `ACM`, `S3`, or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path, or if `discovery_mode` is set to `aws_cloud_map`. | +| `aws_region` | A `String` represents the AWS Region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file`, and `ssl_key_file` is set to the `AWS S3` path, or if `discovery_mode` is set to `aws_cloud_map`. | | `drain_timeout` | A `Duration` representing the wait time for the Peer Forwarder to complete processing data before shutdown. | ## SSL configuration -The SSL configuration for setting up trust manager for the peer forwarding client to connect to other Data Prepper instances. - -### Optional SSL configuration -See the table below for optional SSL configuration descriptions. +The following SSL configuration table provides optional SSL configuration values and allows you to set up trust manager for the peer forwarding client to connect to other Data Prepper instances. | Value | Description | | ----- | ----------- | -| `ssl` | A `boolean` that enables TLS/SSL. Default value is `true`. | -| `ssl_certificate_file`| A `String` representing the SSL certificate chain file path or AWS S3 path. S3 path example `s3:///`. Defaults to the default certificate file,`config/default_certificate.pem`. Read more about how the certificate file is generated at the [Default Certificates](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates) page. | +| `ssl` | A `Boolean` that enables TLS/SSL. Default value is `true`. | +| `ssl_certificate_file`| A `String` representing the SSL certificate chain file path or Amazon Simple Storage Service (Amazon S3) path. The following is an example of an Amazon S3 path: `s3:///`. Defaults to the default certificate file,`config/default_certificate.pem`. See [Default Certificates](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates) for more information about how the certificate is generated. | | `ssl_key_file`| A `String` represents the SSL key file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_private_key.pem` which is default private key file. Read more about how the private key file is generated at the [Default Certificates](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates) page. | -| `ssl_insecure_disable_verification` | A `boolean` that disables the verification of server's TLS certificate chain. Default value is `false`. | -| `ssl_fingerprint_verification_only` | A `boolean` that disables the verification of server's TLS certificate chain and instead verifies only the certificate fingerprint. Default value is `false`. | -| `use_acm_certificate_for_ssl` | A `boolean` that enables TLS/SSL using certificate and private key from AWS Certificate Manager (ACM). Default is `false`. | -| `acm_certificate_arn`| A `String` represents the ACM certificate ARN. ACM certificate take preference over S3 or local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. | -| `acm_private_key_password` | A `String` that represents the ACM private key password that will be used to decrypt the private key. If it's not provided, a random password will be generated. | -| `acm_certificate_timeout_millis` | An `int` representing the timeout in milliseconds for ACM to get certificates. Default value is `120000`. | -| `aws_region` | A `String` represents the AWS Region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. | +| `ssl_insecure_disable_verification` | A `Boolean` that disables the verification of the server's TLS certificate chain. Default value is `false`. | +| `ssl_fingerprint_verification_only` | A `Boolean` that disables the verification of the server's TLS certificate chain and instead verifies only the certificate fingerprint. Default value is `false`. | +| `use_acm_certificate_for_ssl` | A `Boolean` that enables TLS/SSL using the certificate and private key from AWS Certificate Manager (ACM). Default value is `false`. | +| `acm_certificate_arn`| A `String` representing the ACM certificate Amazon Resource Name (ARN). The ACM certificate takes precedence over S3 or the local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. | +| `acm_private_key_password` | A `String` representing the ACM private key password that will be used to decrypt the private key. If it's not provided, a random password will be generated. | +| `acm_certificate_timeout_millis` | An `Int` representing the timeout in milliseconds for ACM to get certificates. Default value is `120000`. | +| `aws_region` | A `String` representing the AWS Region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file`, and `ssl_key_file` is set to the `AWS S3` path, or if `discovery_mode` is set to `aws_cloud_map`. | + +#### Example configuration -#### Example config +The following yaml file provides an example configuration: ```yaml peer_forwarder: @@ -138,7 +138,7 @@ peer_forwarder: This section describes optional authentication. -`authentication`(optional) : A `Map` that enables mTLS. It can either be `mutual_tls` or `unauthenticated`. The default value is `unauthenticated`. +`authentication` is optional and is a `Map` that enables mTLS. It can either be `mutual_tls` or `unauthenticated`. The default value is `unauthenticated`. The following yaml file provides an exmaple for authentication: ```yaml peer_forwarder: @@ -150,24 +150,24 @@ peer_forwarder: This section describes Peer Forwarder's metrics capabilities, including timer, counter, and gauge. -Core Peer Forwarder introduces the following custom metrics and all the metrics are prefixed by `core.peerForwarder`. +Core Peer Forwarder introduces the following custom metrics. All the metrics are prefixed by `core.peerForwarder`. ### Timer -- `requestForwardingLatency`: measures latency of forwarding requests by the Peer Forwarder client. -- `requestProcessingLatency`: measures latency of processing requests by Peer Forwarder server. +- `requestForwardingLatency`: Measures latency of requestes forwarded by the Peer Forwarder client. +- `requestProcessingLatency`: Measures latency of requests processed by the Peer Forwarder server. ### Counter -See the table below for counter metric options. +The following table provides counter metric options. | Value | Description | | ----- | ----------- | | `requests`| Measures the total number of forwarded requests. | | `requestsFailed`| Measures the total number of failed requests. Requests with HTTP response code other than `200`. | | `requestsSuccessful`| Measures the total number of successful requests. Requests with HTTP response code `200`. | -| `requestsTooLarge`| Measures the total number of requests that are too large to be written to Peer Forwarder buffer. Requests with HTTP response code `413`. | -| `requestTimeouts`| Measures the total number of requests that timed out while writing content to Peer Forwarder buffer. Requests with HTTP response code `408`. | +| `requestsTooLarge`| Measures the total number of requests that are too large to be written to the Peer Forwarder buffer. Requests with HTTP response code `413`. | +| `requestTimeouts`| Measures the total number of requests that time out while writing content to the Peer Forwarder buffer. Requests with HTTP response code `408`. | | `requestsUnprocessable`| Measures the total number of requests that failed due to unprocessable entity. Requests with HTTP response code `422`. | | `badRequests`| Measures the total number of requests with bad request format. Requests with HTTP response code `400`. | | `recordsSuccessfullyForwarded`| Measures the total number of forwarded records successfully. | From 5c640ca8bdad0aba52b3e3f4a3e6068bd6433a80 Mon Sep 17 00:00:00 2001 From: carolxob Date: Mon, 19 Dec 2022 16:16:55 -0800 Subject: [PATCH 30/49] Incorporated minor editorial feedback changes. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 769929c995..979597b1e3 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -8,7 +8,7 @@ nav_order: 12 Peer Forwarder is an HTTP service that performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, Peer Forwarder is supported by the `aggregate`, `service_map_stateful`, and `otel_trace_raw` processors. -Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The `aggregate` processor is configured using the `identification_keys` configuration option. From here, you can specify which keys to use for Peer Forwarder. See [Aggregate Processor Page](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) for more information about identification keys. +Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The `aggregate` processor is configured using the `identification_keys` configuration option. From here, you can specify which keys to use for Peer Forwarder. See [Aggregate Processor page](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) for more information about identification keys. Peer discovery is currently provided by a static list, a DNS record lookup, or AWS Cloud Map. @@ -28,7 +28,7 @@ peer_forwarder:4 ### DNS lookup -DNS discovery is preferred over static discovery when scaling out a Data Prepper cluster. The core concept is to configure a DNS provider to return a list of Data Prepper hosts when given a single domain name. This is a [DNS A Record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/), which indicates a list of IP addresses of a given domain. See the following yaml file example of DNS lookup: +DNS discovery is preferred over static discovery when scaling out a Data Prepper cluster. DNS discovery configures a DNS provider to return a list of Data Prepper hosts when given a single domain name. This list that is returned is a [DNS A Record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/), which indicates a list of IP addresses of a given domain. See the following yaml file example of DNS lookup: ```yaml peer_forwarder: From a0eb2ecf234dade8050257a9553e0fccd04e82c4 Mon Sep 17 00:00:00 2001 From: carolxob Date: Tue, 3 Jan 2023 15:26:38 -0700 Subject: [PATCH 31/49] Incorporated more editorial feedback. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 979597b1e3..71b16fe957 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -28,7 +28,7 @@ peer_forwarder:4 ### DNS lookup -DNS discovery is preferred over static discovery when scaling out a Data Prepper cluster. DNS discovery configures a DNS provider to return a list of Data Prepper hosts when given a single domain name. This list that is returned is a [DNS A Record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/), which indicates a list of IP addresses of a given domain. See the following yaml file example of DNS lookup: +DNS discovery is preferred over static discovery when scaling out a Data Prepper cluster. DNS discovery configures a DNS provider to return a list of Data Prepper hosts when given a single domain name. This list that is returned is a [DNS A record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/), which indicates a list of IP addresses of a given domain. See the following yaml file example of DNS lookup: ```yaml peer_forwarder: @@ -94,13 +94,13 @@ The following table provides optional configuration values. | `request_timeout` | An `int` representing the request timeout duration in milliseconds for the Peer Forwarder HTTP server. Default value is `10000`. | | `server_thread_count` | An `int` representing the number of threads used by the Peer Forwarder server. Default value is `200`.| | `client_thread_count` | An `int` representing the number of threads used by the Peer Forwarder client. Default value is `200`.| -| `maxConnectionCount` | An `int` representing the maximum number of open connections for Peer Forwarder server. Default value is `500`. | +| `maxConnectionCount` | An `int` representing the maximum number of open connections for the Peer Forwarder server. Default value is `500`. | | `discovery_mode` | A `String` representing the peer discovery mode to be used. Allowable values are `local_node`, `static`, `dns`, and `aws_cloud_map`. Defaults to `local_node`, which processes events locally. | | `static_endpoints` | A `list` containing the endpoints of all Data Prepper instances. Required if `discovery_mode` is set to `static`. | | `domain_name` | A `String` representing the single domain name to query DNS against. Typically, used by creating multiple [DNS A records](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) for the same domain. Required if `discovery_mode` is set to `dns`. | | `aws_cloud_map_namespace_name` | A `String` representing the AWS Cloud Map namespace when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | -| `aws_cloud_map_service_name` | A `String` representing the Cloud Map service when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | -| `aws_cloud_map_query_parameters` | A `Map` of Key/value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. | +| `aws_cloud_map_service_name` | A `String` representing the AWS Cloud Map service when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | +| `aws_cloud_map_query_parameters` | A `Map` of key-value pairs used to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. | | `buffer_size` | An `int` representing the maximum number of unchecked records the buffer accepts (the number of unchecked records equals the number of records written into the buffer plus the number of in-flight records not yet checked by the Checkpointing API). Default is `512`. | | `batch_size` | An `int` representing max number of records the buffer returns on read. Default is `48`. | | `aws_region` | A `String` represents the AWS Region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file`, and `ssl_key_file` is set to the `AWS S3` path, or if `discovery_mode` is set to `aws_cloud_map`. | @@ -120,8 +120,8 @@ The following SSL configuration table provides optional SSL configuration values | `use_acm_certificate_for_ssl` | A `Boolean` that enables TLS/SSL using the certificate and private key from AWS Certificate Manager (ACM). Default value is `false`. | | `acm_certificate_arn`| A `String` representing the ACM certificate Amazon Resource Name (ARN). The ACM certificate takes precedence over S3 or the local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. | | `acm_private_key_password` | A `String` representing the ACM private key password that will be used to decrypt the private key. If it's not provided, a random password will be generated. | -| `acm_certificate_timeout_millis` | An `Int` representing the timeout in milliseconds for ACM to get certificates. Default value is `120000`. | -| `aws_region` | A `String` representing the AWS Region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file`, and `ssl_key_file` is set to the `AWS S3` path, or if `discovery_mode` is set to `aws_cloud_map`. | +| `acm_certificate_timeout_millis` | An `Int` representing the timeout in milliseconds required for ACM to get certificates. Default value is `120000`. | +| `aws_region` | A `String` representing the AWS Region that uses `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file`. Also required when the `ssl_key_file` is set to the `AWS S3` path, or if `discovery_mode` is set to `aws_cloud_map`. | #### Example configuration @@ -138,7 +138,7 @@ peer_forwarder: This section describes optional authentication. -`authentication` is optional and is a `Map` that enables mTLS. It can either be `mutual_tls` or `unauthenticated`. The default value is `unauthenticated`. The following yaml file provides an exmaple for authentication: +`Authentication` is optional and is a `Map` that enables mTLS. It can either be `mutual_tls` or `unauthenticated`. The default value is `unauthenticated`. The following yaml file provides an exmaple for authentication: ```yaml peer_forwarder: @@ -154,7 +154,9 @@ Core Peer Forwarder introduces the following custom metrics. All the metrics are ### Timer -- `requestForwardingLatency`: Measures latency of requestes forwarded by the Peer Forwarder client. +Peer Forwarder's timer capability provides the following information: + +- `requestForwardingLatency`: Measures latency of requests forwarded by the Peer Forwarder client. - `requestProcessingLatency`: Measures latency of requests processed by the Peer Forwarder server. ### Counter @@ -168,10 +170,10 @@ The following table provides counter metric options. | `requestsSuccessful`| Measures the total number of successful requests. Requests with HTTP response code `200`. | | `requestsTooLarge`| Measures the total number of requests that are too large to be written to the Peer Forwarder buffer. Requests with HTTP response code `413`. | | `requestTimeouts`| Measures the total number of requests that time out while writing content to the Peer Forwarder buffer. Requests with HTTP response code `408`. | -| `requestsUnprocessable`| Measures the total number of requests that failed due to unprocessable entity. Requests with HTTP response code `422`. | +| `requestsUnprocessable`| Measures the total number of requests that fail due to an unprocessable entity. Requests with HTTP response code `422`. | | `badRequests`| Measures the total number of requests with bad request format. Requests with HTTP response code `400`. | -| `recordsSuccessfullyForwarded`| Measures the total number of forwarded records successfully. | -| `recordsFailedForwarding`| Measures the total number of records failed to be forwarded. | +| `recordsSuccessfullyForwarded`| Measures the total number of successfully forwarded records. | +| `recordsFailedForwarding`| Measures the total number of records fail to be forwarded. | | `recordsToBeForwarded` | Measures the total number of records to be forwarded. | | `recordsToBeProcessedLocally` | Measures the total number of records to be processed locally. | | `recordsActuallyProcessedLocally`| Measures the total number of records actually processed locally. This value is the sum of `recordsToBeProcessedLocally` and `recordsFailedForwarding`. | From 069ef713a9f6a1a91c515bfbeb53a562405ca92a Mon Sep 17 00:00:00 2001 From: carolxob Date: Wed, 4 Jan 2023 10:00:55 -0700 Subject: [PATCH 32/49] Minor changes. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 71b16fe957..f2c5358350 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -181,4 +181,4 @@ The following table provides counter metric options. ### Gauge -`peerEndpoints` Measures number of dynamically discovered peer Data Prepper endpoints. For `static` mode, the size is fixed. +`peerEndpoints` Measures the number of dynamically discovered peer Data Prepper endpoints. For `static` mode, the size is fixed. From f4cc653fba659142a112ee41fc21ef5cf65c8900 Mon Sep 17 00:00:00 2001 From: carolxob Date: Tue, 10 Jan 2023 15:28:26 -0700 Subject: [PATCH 33/49] Committed existing Peer Forwarder file to new PR to simplify feedback. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index f2c5358350..badf2eee5d 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -82,7 +82,8 @@ Data Prepper must also be running with the necessary permissions. The following ] } ``` ---- + + ## Configuration The following table provides optional configuration values. @@ -136,8 +137,6 @@ peer_forwarder: ## Authentication -This section describes optional authentication. - `Authentication` is optional and is a `Map` that enables mTLS. It can either be `mutual_tls` or `unauthenticated`. The default value is `unauthenticated`. The following yaml file provides an exmaple for authentication: ```yaml @@ -148,8 +147,6 @@ peer_forwarder: ## Metrics -This section describes Peer Forwarder's metrics capabilities, including timer, counter, and gauge. - Core Peer Forwarder introduces the following custom metrics. All the metrics are prefixed by `core.peerForwarder`. ### Timer From ccf12e759c447c93128acab6e83d2bad184749e5 Mon Sep 17 00:00:00 2001 From: carolxob Date: Thu, 19 Jan 2023 16:22:01 -0700 Subject: [PATCH 34/49] Incorporated doc review feedback. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 30 ++++++++++++++++-------------- 1 file changed, 16 insertions(+), 14 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index badf2eee5d..54c762bd04 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -6,7 +6,9 @@ nav_order: 12 # Peer Forwarder -Peer Forwarder is an HTTP service that performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, Peer Forwarder is supported by the `aggregate`, `service_map_stateful`, and `otel_trace_raw` processors. +Peer Forwarder is an HTTP service that performs peer forwarding of an `event` between Data Prepper nodes for aggregation. This HTTP service uses a hash-ring approach to aggregate events and determine which Data Prepper node should handle on a given trace before rerouting it to that node. Currently, Peer Forwarder is supported by the `aggregate`, `service_map_stateful`, and `otel_trace_raw` processors. + + Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The `aggregate` processor is configured using the `identification_keys` configuration option. From here, you can specify which keys to use for Peer Forwarder. See [Aggregate Processor page](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) for more information about identification keys. @@ -18,7 +20,7 @@ The following sections provide information about discovery modes. ### Static -Static discovery mode allows a Data Prepper node to discover nodes using a list of IP addresses or domain names. See the following yaml file for an example of static discovery mode: +Static discovery mode allows a Data Prepper node to discover nodes using a list of IP addresses or domain names. See the following YAML file for an example of static discovery mode: ```yaml peer_forwarder:4 @@ -28,7 +30,7 @@ peer_forwarder:4 ### DNS lookup -DNS discovery is preferred over static discovery when scaling out a Data Prepper cluster. DNS discovery configures a DNS provider to return a list of Data Prepper hosts when given a single domain name. This list that is returned is a [DNS A record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/), which indicates a list of IP addresses of a given domain. See the following yaml file example of DNS lookup: +DNS discovery is preferred over static discovery when scaling out a Data Prepper cluster. DNS discovery configures a DNS provider to return a list of Data Prepper hosts when given a single domain name. This list is a [DNS A record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/), and a list of IP addresses of a given domain. See the following YAML file example of DNS lookup: ```yaml peer_forwarder: @@ -49,11 +51,11 @@ Your Data Prepper configuration needs to include the following: * `discovery_mode` – Set to `aws_cloud_map`. Your Data Prepper configuration can optionally include the following: -* `aws_cloud_map_query_parameters` – Key-value pairs are used to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. +* `aws_cloud_map_query_parameters` – Key-value pairs are used to filter the results based on the custom attributes attached to an instance. Results include only those instances that match all of the specified key-value pairs. #### Example configuration -See the following yaml file example of AWS Cloud Map configuration: +See the following YAML file example of AWS Cloud Map configuration: ```yaml peer_forwarder: @@ -115,7 +117,7 @@ The following SSL configuration table provides optional SSL configuration values | ----- | ----------- | | `ssl` | A `Boolean` that enables TLS/SSL. Default value is `true`. | | `ssl_certificate_file`| A `String` representing the SSL certificate chain file path or Amazon Simple Storage Service (Amazon S3) path. The following is an example of an Amazon S3 path: `s3:///`. Defaults to the default certificate file,`config/default_certificate.pem`. See [Default Certificates](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates) for more information about how the certificate is generated. | -| `ssl_key_file`| A `String` represents the SSL key file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_private_key.pem` which is default private key file. Read more about how the private key file is generated at the [Default Certificates](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates) page. | +| `ssl_key_file`| A `String` that represents the SSL key file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_private_key.pem` which is default private key file. Read more about how the private key file is generated at the [Default Certificates](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates) page. | | `ssl_insecure_disable_verification` | A `Boolean` that disables the verification of the server's TLS certificate chain. Default value is `false`. | | `ssl_fingerprint_verification_only` | A `Boolean` that disables the verification of the server's TLS certificate chain and instead verifies only the certificate fingerprint. Default value is `false`. | | `use_acm_certificate_for_ssl` | A `Boolean` that enables TLS/SSL using the certificate and private key from AWS Certificate Manager (ACM). Default value is `false`. | @@ -126,7 +128,7 @@ The following SSL configuration table provides optional SSL configuration values #### Example configuration -The following yaml file provides an example configuration: +The following YAML file provides an example configuration: ```yaml peer_forwarder: @@ -137,7 +139,7 @@ peer_forwarder: ## Authentication -`Authentication` is optional and is a `Map` that enables mTLS. It can either be `mutual_tls` or `unauthenticated`. The default value is `unauthenticated`. The following yaml file provides an exmaple for authentication: +`Authentication` is optional and is a `Map` that enables mutual TLS (mTLS). It can either be `mutual_tls` or `unauthenticated`. The default value is `unauthenticated`. The following YAML file provides an exmaple for authentication: ```yaml peer_forwarder: @@ -163,12 +165,12 @@ The following table provides counter metric options. | Value | Description | | ----- | ----------- | | `requests`| Measures the total number of forwarded requests. | -| `requestsFailed`| Measures the total number of failed requests. Requests with HTTP response code other than `200`. | -| `requestsSuccessful`| Measures the total number of successful requests. Requests with HTTP response code `200`. | -| `requestsTooLarge`| Measures the total number of requests that are too large to be written to the Peer Forwarder buffer. Requests with HTTP response code `413`. | -| `requestTimeouts`| Measures the total number of requests that time out while writing content to the Peer Forwarder buffer. Requests with HTTP response code `408`. | -| `requestsUnprocessable`| Measures the total number of requests that fail due to an unprocessable entity. Requests with HTTP response code `422`. | -| `badRequests`| Measures the total number of requests with bad request format. Requests with HTTP response code `400`. | +| `requestsFailed`| Measures the total number of failed requests. Applies to requests with HTTP response code other than `200`. | +| `requestsSuccessful`| Measures the total number of successful requests. Applies to requests with HTTP response code `200`. | +| `requestsTooLarge`| Measures the total number of requests that are too large to be written to the Peer Forwarder buffer. Applies to requests with HTTP response code `413`. | +| `requestTimeouts`| Measures the total number of requests that time out while writing content to the Peer Forwarder buffer. Applies to requests with HTTP response code `408`. | +| `requestsUnprocessable`| Measures the total number of requests that fail due to an unprocessable entity. Applies to requests with HTTP response code `422`. | +| `badRequests`| Measures the total number of requests with bad request format. Applies to requests with HTTP response code `400`. | | `recordsSuccessfullyForwarded`| Measures the total number of successfully forwarded records. | | `recordsFailedForwarding`| Measures the total number of records fail to be forwarded. | | `recordsToBeForwarded` | Measures the total number of records to be forwarded. | From 2cbdd7b1d1e57fbf116c0c0fbb68d1683fe7f4b3 Mon Sep 17 00:00:00 2001 From: carolxob Date: Thu, 19 Jan 2023 16:25:51 -0700 Subject: [PATCH 35/49] Minor updates. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 54c762bd04..ef8a49c55c 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -8,8 +8,6 @@ nav_order: 12 Peer Forwarder is an HTTP service that performs peer forwarding of an `event` between Data Prepper nodes for aggregation. This HTTP service uses a hash-ring approach to aggregate events and determine which Data Prepper node should handle on a given trace before rerouting it to that node. Currently, Peer Forwarder is supported by the `aggregate`, `service_map_stateful`, and `otel_trace_raw` processors. - - Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The `aggregate` processor is configured using the `identification_keys` configuration option. From here, you can specify which keys to use for Peer Forwarder. See [Aggregate Processor page](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) for more information about identification keys. Peer discovery is currently provided by a static list, a DNS record lookup, or AWS Cloud Map. From 05e7a061e51c29ea616159bd062e77b5dd79d165 Mon Sep 17 00:00:00 2001 From: carolxob Date: Mon, 23 Jan 2023 13:11:27 -0700 Subject: [PATCH 36/49] Minor changes. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index ef8a49c55c..048f9356e5 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -10,7 +10,7 @@ Peer Forwarder is an HTTP service that performs peer forwarding of an `event` be Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The `aggregate` processor is configured using the `identification_keys` configuration option. From here, you can specify which keys to use for Peer Forwarder. See [Aggregate Processor page](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) for more information about identification keys. -Peer discovery is currently provided by a static list, a DNS record lookup, or AWS Cloud Map. +Peer discovery allows Data Prepper to find other nodes that it will communicate with. Currently, peer discovery is currently provided by a static list, a DNS record lookup, or AWS Cloud Map. ## Discovery modes From ae8070f68df6f4eebe66c5d4bf5e300258b651f0 Mon Sep 17 00:00:00 2001 From: carolxob Date: Tue, 24 Jan 2023 15:24:25 -0700 Subject: [PATCH 37/49] Updates specifically to table formatting. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 58 ++++++++++++++++----------------- 1 file changed, 29 insertions(+), 29 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 048f9356e5..5d94226e05 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -89,40 +89,40 @@ Data Prepper must also be running with the necessary permissions. The following The following table provides optional configuration values. -| Value | Description | -| ---- | ----------- | -| `port` | An `int` between 0 and 65535 represents the port that the Peer Forwarder server is running on. Default value is `4994`. | -| `request_timeout` | An `int` representing the request timeout duration in milliseconds for the Peer Forwarder HTTP server. Default value is `10000`. | -| `server_thread_count` | An `int` representing the number of threads used by the Peer Forwarder server. Default value is `200`.| -| `client_thread_count` | An `int` representing the number of threads used by the Peer Forwarder client. Default value is `200`.| -| `maxConnectionCount` | An `int` representing the maximum number of open connections for the Peer Forwarder server. Default value is `500`. | -| `discovery_mode` | A `String` representing the peer discovery mode to be used. Allowable values are `local_node`, `static`, `dns`, and `aws_cloud_map`. Defaults to `local_node`, which processes events locally. | -| `static_endpoints` | A `list` containing the endpoints of all Data Prepper instances. Required if `discovery_mode` is set to `static`. | -| `domain_name` | A `String` representing the single domain name to query DNS against. Typically, used by creating multiple [DNS A records](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) for the same domain. Required if `discovery_mode` is set to `dns`. | -| `aws_cloud_map_namespace_name` | A `String` representing the AWS Cloud Map namespace when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | -| `aws_cloud_map_service_name` | A `String` representing the AWS Cloud Map service when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | -| `aws_cloud_map_query_parameters` | A `Map` of key-value pairs used to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. | -| `buffer_size` | An `int` representing the maximum number of unchecked records the buffer accepts (the number of unchecked records equals the number of records written into the buffer plus the number of in-flight records not yet checked by the Checkpointing API). Default is `512`. | -| `batch_size` | An `int` representing max number of records the buffer returns on read. Default is `48`. | -| `aws_region` | A `String` represents the AWS Region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file`, and `ssl_key_file` is set to the `AWS S3` path, or if `discovery_mode` is set to `aws_cloud_map`. | -| `drain_timeout` | A `Duration` representing the wait time for the Peer Forwarder to complete processing data before shutdown. | +| Value | Type | Description | +| ---- | --- | ----------- | +| `port` | Integer | A value between 0 and 65535 represents the port that the Peer Forwarder server is running on. Default value is `4994`. | +| `request_timeout` | Integer | Represents the request timeout duration in milliseconds for the Peer Forwarder HTTP server. Default value is `10000`. | +| `server_thread_count` | Integer | Represents the number of threads used by the Peer Forwarder server. Default value is `200`.| +| `client_thread_count` | Integer | Represents the number of threads used by the Peer Forwarder client. Default value is `200`.| +| `maxConnectionCount` | Integer | Represents the maximum number of open connections for the Peer Forwarder server. Default value is `500`. | +| `discovery_mode` | String | Represents the peer discovery mode to be used. Allowable values are `local_node`, `static`, `dns`, and `aws_cloud_map`. Defaults to `local_node`, which processes events locally. | +| `static_endpoints` | List | Contains the endpoints of all Data Prepper instances. Required if `discovery_mode` is set to `static`. | +| `domain_name` | String | Represents the single domain name to query DNS against. Typically used by creating multiple [DNS A records](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) for the same domain. Required if `discovery_mode` is set to `dns`. | +| `aws_cloud_map_namespace_name` | String | Represents the AWS Cloud Map namespace when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | +| `aws_cloud_map_service_name` | String | Represents the AWS Cloud Map service when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | +| `aws_cloud_map_query_parameters` | Map | Key-value pairs used to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. | +| `buffer_size` | Integer | Represents the maximum number of unchecked records the buffer accepts (the number of unchecked records equals the number of records written into the buffer plus the number of records that are still processing and not yet checked by the Checkpointing API). Default is `512`. | +| `batch_size` | Integer | Representing max number of records the buffer returns on read. Default is `48`. | +| `aws_region` | String | Represents the AWS Region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file`, and `ssl_key_file` is set to the `AWS S3` path, or if `discovery_mode` is set to `aws_cloud_map`. | +| `drain_timeout` | Duration | Represents the wait time for the Peer Forwarder to complete processing data before shutdown. | ## SSL configuration The following SSL configuration table provides optional SSL configuration values and allows you to set up trust manager for the peer forwarding client to connect to other Data Prepper instances. -| Value | Description | -| ----- | ----------- | -| `ssl` | A `Boolean` that enables TLS/SSL. Default value is `true`. | -| `ssl_certificate_file`| A `String` representing the SSL certificate chain file path or Amazon Simple Storage Service (Amazon S3) path. The following is an example of an Amazon S3 path: `s3:///`. Defaults to the default certificate file,`config/default_certificate.pem`. See [Default Certificates](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates) for more information about how the certificate is generated. | -| `ssl_key_file`| A `String` that represents the SSL key file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_private_key.pem` which is default private key file. Read more about how the private key file is generated at the [Default Certificates](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates) page. | -| `ssl_insecure_disable_verification` | A `Boolean` that disables the verification of the server's TLS certificate chain. Default value is `false`. | -| `ssl_fingerprint_verification_only` | A `Boolean` that disables the verification of the server's TLS certificate chain and instead verifies only the certificate fingerprint. Default value is `false`. | -| `use_acm_certificate_for_ssl` | A `Boolean` that enables TLS/SSL using the certificate and private key from AWS Certificate Manager (ACM). Default value is `false`. | -| `acm_certificate_arn`| A `String` representing the ACM certificate Amazon Resource Name (ARN). The ACM certificate takes precedence over S3 or the local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. | -| `acm_private_key_password` | A `String` representing the ACM private key password that will be used to decrypt the private key. If it's not provided, a random password will be generated. | -| `acm_certificate_timeout_millis` | An `Int` representing the timeout in milliseconds required for ACM to get certificates. Default value is `120000`. | -| `aws_region` | A `String` representing the AWS Region that uses `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file`. Also required when the `ssl_key_file` is set to the `AWS S3` path, or if `discovery_mode` is set to `aws_cloud_map`. | +| Value | Type | Description | +| ----- | ---- | ----------- | +| `ssl` | Boolean | Enables TLS/SSL. Default value is `true`. | +| `ssl_certificate_file`| String | Representings the SSL certificate chain file path or Amazon Simple Storage Service (Amazon S3) path. The following is an example of an Amazon S3 path: `s3:///`. Defaults to the default certificate file,`config/default_certificate.pem`. See [Default Certificates](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates) for more information about how the certificate is generated. | +| `ssl_key_file`| String | Represents the SSL key file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_private_key.pem` which is default private key file. Read more about how the private key file is generated at the [Default Certificates](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates) page. | +| `ssl_insecure_disable_verification` | Boolean | that disables the verification of the server's TLS certificate chain. Default value is `false`. | +| `ssl_fingerprint_verification_only` | Boolean | Disables the verification of the server's TLS certificate chain and instead verifies only the certificate fingerprint. Default value is `false`. | +| `use_acm_certificate_for_ssl` | Boolean | Enables TLS/SSL using the certificate and private key from AWS Certificate Manager (ACM). Default value is `false`. | +| `acm_certificate_arn`| String | Representings the ACM certificate Amazon Resource Name (ARN). The ACM certificate takes precedence over S3 or the local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. | +| `acm_private_key_password` | String | Represents the ACM private key password that will be used to decrypt the private key. If it's not provided, a random password will be generated. | +| `acm_certificate_timeout_millis` | Integer |representing the timeout in milliseconds required for ACM to get certificates. Default value is `120000`. | +| `aws_region` | String | Represents the AWS Region that uses `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file`. Also required when the `ssl_key_file` is set to the `AWS S3` path, or if `discovery_mode` is set to `aws_cloud_map`. | #### Example configuration From f1b4f1f4c14a00fd5731a86a6082e7d458390981 Mon Sep 17 00:00:00 2001 From: carolxob Date: Tue, 24 Jan 2023 16:07:39 -0700 Subject: [PATCH 38/49] Minor updates. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 5d94226e05..455f9ddb82 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -100,8 +100,8 @@ The following table provides optional configuration values. | `static_endpoints` | List | Contains the endpoints of all Data Prepper instances. Required if `discovery_mode` is set to `static`. | | `domain_name` | String | Represents the single domain name to query DNS against. Typically used by creating multiple [DNS A records](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) for the same domain. Required if `discovery_mode` is set to `dns`. | | `aws_cloud_map_namespace_name` | String | Represents the AWS Cloud Map namespace when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | -| `aws_cloud_map_service_name` | String | Represents the AWS Cloud Map service when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | -| `aws_cloud_map_query_parameters` | Map | Key-value pairs used to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. | +| `aws_cloud_map_service_name` | String | Represents the AWS Cloud Map service when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | +| `aws_cloud_map_query_parameters` | Map | Key-value pairs used to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. | | `buffer_size` | Integer | Represents the maximum number of unchecked records the buffer accepts (the number of unchecked records equals the number of records written into the buffer plus the number of records that are still processing and not yet checked by the Checkpointing API). Default is `512`. | | `batch_size` | Integer | Representing max number of records the buffer returns on read. Default is `48`. | | `aws_region` | String | Represents the AWS Region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file`, and `ssl_key_file` is set to the `AWS S3` path, or if `discovery_mode` is set to `aws_cloud_map`. | From b1dc95db4df54033099f7e7374c8d949e20f27bd Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 27 Jan 2023 15:41:07 -0700 Subject: [PATCH 39/49] Made minor edit to the end of the Configuration table. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 455f9ddb82..3a1e09562e 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -104,7 +104,7 @@ The following table provides optional configuration values. | `aws_cloud_map_query_parameters` | Map | Key-value pairs used to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. | | `buffer_size` | Integer | Represents the maximum number of unchecked records the buffer accepts (the number of unchecked records equals the number of records written into the buffer plus the number of records that are still processing and not yet checked by the Checkpointing API). Default is `512`. | | `batch_size` | Integer | Representing max number of records the buffer returns on read. Default is `48`. | -| `aws_region` | String | Represents the AWS Region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file`, and `ssl_key_file` is set to the `AWS S3` path, or if `discovery_mode` is set to `aws_cloud_map`. | +| `aws_region` | String | Represents the AWS Region to use `ACM`, `S3` or `AWS Cloud Map` and is required when any of the following conditions are met:
- The `use_acm_certificate_for_ssl` setting is `true`.
- Either `ssl_certificate_file` or `ssl_key_file` specifies an AWS S3 URI (for example, s3://mybucket/path/to/public.cert).
- The `discovery_mode` setting is `aws_cloud_map`. | | `drain_timeout` | Duration | Represents the wait time for the Peer Forwarder to complete processing data before shutdown. | ## SSL configuration From 5ae8f0ccb0c6a9838313f6a2563eec74225aac49 Mon Sep 17 00:00:00 2001 From: carolxob Date: Tue, 31 Jan 2023 15:59:46 -0700 Subject: [PATCH 40/49] Minor edits from doc review feedback. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 3a1e09562e..45b8758c33 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -6,11 +6,11 @@ nav_order: 12 # Peer Forwarder -Peer Forwarder is an HTTP service that performs peer forwarding of an `event` between Data Prepper nodes for aggregation. This HTTP service uses a hash-ring approach to aggregate events and determine which Data Prepper node should handle on a given trace before rerouting it to that node. Currently, Peer Forwarder is supported by the `aggregate`, `service_map_stateful`, and `otel_trace_raw` processors. +Peer Forwarder is an HTTP service that performs peer forwarding of an `event` between Data Prepper nodes for aggregation. This HTTP service uses a hash-ring approach to aggregate events and determine which Data Prepper node should handle on a given trace before rerouting it to that node. Currently, Peer Forwarder is supported by the `aggregate`, `service_map_stateful`, and `otel_trace_raw` [processors]({{site.url}}{{site.baseurl}}/data-prepper/configuration/processors/processors/). -Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The `aggregate` processor is configured using the `identification_keys` configuration option. From here, you can specify which keys to use for Peer Forwarder. See [Aggregate Processor page](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) for more information about identification keys. +Peer Forwarder groups events based on the identification keys provided by the supported processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The `aggregate` processor is configured using the `identification_keys` configuration option. From here, you can specify which keys to use for Peer Forwarder. See [Aggregate Processor page](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) for more information about identification keys. -Peer discovery allows Data Prepper to find other nodes that it will communicate with. Currently, peer discovery is currently provided by a static list, a DNS record lookup, or AWS Cloud Map. +Peer discovery allows Data Prepper to find other nodes that it will communicate with. Currently, peer discovery is provided by a static list, a DNS record lookup, or anAWS Cloud Map. ## Discovery modes @@ -40,7 +40,7 @@ peer_forwarder: [AWS Cloud Map](https://docs.aws.amazon.com/cloud-map/latest/dg/what-is-cloud-map.html) provides API-based service discovery as well as DNS-based service discovery. -Peer Forwarder can use the API-based service discovery. To support this, you must have an existing namespace configured for API instance discovery. You can create a new one by following the instructions provided by the [AWS Cloud Map documentation](https://docs.aws.amazon.com/cloud-map/latest/dg/working-with-namespaces.html). +Peer Forwarder can use the API-based service discovery in AWS Cloud Map. To support this, you must have an existing namespace configured for API instance discovery. You can create a new one by following the instructions provided by the [AWS Cloud Map documentation](https://docs.aws.amazon.com/cloud-map/latest/dg/working-with-namespaces.html). Your Data Prepper configuration needs to include the following: * `aws_cloud_map_namespace_name` – Set to your AWS Cloud Map namespace name. @@ -114,14 +114,14 @@ The following SSL configuration table provides optional SSL configuration values | Value | Type | Description | | ----- | ---- | ----------- | | `ssl` | Boolean | Enables TLS/SSL. Default value is `true`. | -| `ssl_certificate_file`| String | Representings the SSL certificate chain file path or Amazon Simple Storage Service (Amazon S3) path. The following is an example of an Amazon S3 path: `s3:///`. Defaults to the default certificate file,`config/default_certificate.pem`. See [Default Certificates](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates) for more information about how the certificate is generated. | +| `ssl_certificate_file`| String | Represents the SSL certificate chain file path or Amazon Simple Storage Service (Amazon S3) path. The following is an example of an Amazon S3 path: `s3:///`. Defaults to the default certificate file,`config/default_certificate.pem`. See [Default Certificates](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates) for more information about how the certificate is generated. | | `ssl_key_file`| String | Represents the SSL key file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_private_key.pem` which is default private key file. Read more about how the private key file is generated at the [Default Certificates](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates) page. | -| `ssl_insecure_disable_verification` | Boolean | that disables the verification of the server's TLS certificate chain. Default value is `false`. | +| `ssl_insecure_disable_verification` | Boolean | Disables the verification of the server's TLS certificate chain. Default value is `false`. | | `ssl_fingerprint_verification_only` | Boolean | Disables the verification of the server's TLS certificate chain and instead verifies only the certificate fingerprint. Default value is `false`. | | `use_acm_certificate_for_ssl` | Boolean | Enables TLS/SSL using the certificate and private key from AWS Certificate Manager (ACM). Default value is `false`. | -| `acm_certificate_arn`| String | Representings the ACM certificate Amazon Resource Name (ARN). The ACM certificate takes precedence over S3 or the local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. | +| `acm_certificate_arn`| String | Represents the ACM certificate Amazon Resource Name (ARN). The ACM certificate takes precedence over S3 or the local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. | | `acm_private_key_password` | String | Represents the ACM private key password that will be used to decrypt the private key. If it's not provided, a random password will be generated. | -| `acm_certificate_timeout_millis` | Integer |representing the timeout in milliseconds required for ACM to get certificates. Default value is `120000`. | +| `acm_certificate_timeout_millis` | Integer | Represents the timeout in milliseconds required for ACM to get certificates. Default value is `120000`. | | `aws_region` | String | Represents the AWS Region that uses `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file`. Also required when the `ssl_key_file` is set to the `AWS S3` path, or if `discovery_mode` is set to `aws_cloud_map`. | #### Example configuration From 30d51e531b9b7f4c46906a3a3f9e57fb4a8e4ed9 Mon Sep 17 00:00:00 2001 From: carolxob Date: Wed, 1 Feb 2023 09:52:27 -0700 Subject: [PATCH 41/49] Minor updates from doc review. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 45b8758c33..a06513371d 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -6,7 +6,7 @@ nav_order: 12 # Peer Forwarder -Peer Forwarder is an HTTP service that performs peer forwarding of an `event` between Data Prepper nodes for aggregation. This HTTP service uses a hash-ring approach to aggregate events and determine which Data Prepper node should handle on a given trace before rerouting it to that node. Currently, Peer Forwarder is supported by the `aggregate`, `service_map_stateful`, and `otel_trace_raw` [processors]({{site.url}}{{site.baseurl}}/data-prepper/configuration/processors/processors/). +Peer Forwarder is an HTTP service that performs peer forwarding of an `event` between Data Prepper nodes for aggregation. This HTTP service uses a hash-ring approach to aggregate events and determine which Data Prepper node it should handle on a given trace before rerouting it to that node. Currently, Peer Forwarder is supported by the `aggregate`, `service_map_stateful`, and `otel_trace_raw` [processors]({{site.url}}{{site.baseurl}}/data-prepper/configuration/processors/processors/). Peer Forwarder groups events based on the identification keys provided by the supported processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The `aggregate` processor is configured using the `identification_keys` configuration option. From here, you can specify which keys to use for Peer Forwarder. See [Aggregate Processor page](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) for more information about identification keys. @@ -115,7 +115,7 @@ The following SSL configuration table provides optional SSL configuration values | ----- | ---- | ----------- | | `ssl` | Boolean | Enables TLS/SSL. Default value is `true`. | | `ssl_certificate_file`| String | Represents the SSL certificate chain file path or Amazon Simple Storage Service (Amazon S3) path. The following is an example of an Amazon S3 path: `s3:///`. Defaults to the default certificate file,`config/default_certificate.pem`. See [Default Certificates](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates) for more information about how the certificate is generated. | -| `ssl_key_file`| String | Represents the SSL key file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_private_key.pem` which is default private key file. Read more about how the private key file is generated at the [Default Certificates](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates) page. | +| `ssl_key_file`| String | Represents the SSL key file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_private_key.pem` which is the default private key file. Read more about how the private key file is generated at the [Default Certificates](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates) page. | | `ssl_insecure_disable_verification` | Boolean | Disables the verification of the server's TLS certificate chain. Default value is `false`. | | `ssl_fingerprint_verification_only` | Boolean | Disables the verification of the server's TLS certificate chain and instead verifies only the certificate fingerprint. Default value is `false`. | | `use_acm_certificate_for_ssl` | Boolean | Enables TLS/SSL using the certificate and private key from AWS Certificate Manager (ACM). Default value is `false`. | @@ -170,7 +170,7 @@ The following table provides counter metric options. | `requestsUnprocessable`| Measures the total number of requests that fail due to an unprocessable entity. Applies to requests with HTTP response code `422`. | | `badRequests`| Measures the total number of requests with bad request format. Applies to requests with HTTP response code `400`. | | `recordsSuccessfullyForwarded`| Measures the total number of successfully forwarded records. | -| `recordsFailedForwarding`| Measures the total number of records fail to be forwarded. | +| `recordsFailedForwarding`| Measures the total number of records that fail to be forwarded. | | `recordsToBeForwarded` | Measures the total number of records to be forwarded. | | `recordsToBeProcessedLocally` | Measures the total number of records to be processed locally. | | `recordsActuallyProcessedLocally`| Measures the total number of records actually processed locally. This value is the sum of `recordsToBeProcessedLocally` and `recordsFailedForwarding`. | From 320cc450a764f68774bbee99e0635cd58d2725b8 Mon Sep 17 00:00:00 2001 From: carolxob Date: Wed, 1 Feb 2023 11:00:42 -0700 Subject: [PATCH 42/49] Incorporated more editorial feedback. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index a06513371d..8972099519 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -10,7 +10,7 @@ Peer Forwarder is an HTTP service that performs peer forwarding of an `event` be Peer Forwarder groups events based on the identification keys provided by the supported processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The `aggregate` processor is configured using the `identification_keys` configuration option. From here, you can specify which keys to use for Peer Forwarder. See [Aggregate Processor page](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) for more information about identification keys. -Peer discovery allows Data Prepper to find other nodes that it will communicate with. Currently, peer discovery is provided by a static list, a DNS record lookup, or anAWS Cloud Map. +Peer discovery allows Data Prepper to find other nodes that it will communicate with. Currently, peer discovery is provided by a static list, a DNS record lookup, or AWS Cloud Map. ## Discovery modes @@ -28,7 +28,7 @@ peer_forwarder:4 ### DNS lookup -DNS discovery is preferred over static discovery when scaling out a Data Prepper cluster. DNS discovery configures a DNS provider to return a list of Data Prepper hosts when given a single domain name. This list is a [DNS A record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/), and a list of IP addresses of a given domain. See the following YAML file example of DNS lookup: +DNS discovery is preferred over static discovery when scaling out a Data Prepper cluster. DNS discovery configures a DNS provider to return a list of Data Prepper hosts when given a single domain name. This list consists of a [DNS A record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/), and a list of IP addresses of a given domain. See the following YAML file for an example of DNS lookup: ```yaml peer_forwarder: @@ -45,7 +45,7 @@ Peer Forwarder can use the API-based service discovery in AWS Cloud Map. To supp Your Data Prepper configuration needs to include the following: * `aws_cloud_map_namespace_name` – Set to your AWS Cloud Map namespace name. * `aws_cloud_map_service_name` – Set to the service name within your specified namespace. -* `aws_region` – Set to the AWS Region where your namespace exists. +* `aws_region` – Set to the AWS Region in which your namespace exists. * `discovery_mode` – Set to `aws_cloud_map`. Your Data Prepper configuration can optionally include the following: @@ -91,7 +91,7 @@ The following table provides optional configuration values. | Value | Type | Description | | ---- | --- | ----------- | -| `port` | Integer | A value between 0 and 65535 represents the port that the Peer Forwarder server is running on. Default value is `4994`. | +| `port` | Integer | A value between 0 and 65535 that represents the port that the Peer Forwarder server is running on. Default value is `4994`. | | `request_timeout` | Integer | Represents the request timeout duration in milliseconds for the Peer Forwarder HTTP server. Default value is `10000`. | | `server_thread_count` | Integer | Represents the number of threads used by the Peer Forwarder server. Default value is `200`.| | `client_thread_count` | Integer | Represents the number of threads used by the Peer Forwarder client. Default value is `200`.| @@ -103,26 +103,26 @@ The following table provides optional configuration values. | `aws_cloud_map_service_name` | String | Represents the AWS Cloud Map service when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | | `aws_cloud_map_query_parameters` | Map | Key-value pairs used to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. | | `buffer_size` | Integer | Represents the maximum number of unchecked records the buffer accepts (the number of unchecked records equals the number of records written into the buffer plus the number of records that are still processing and not yet checked by the Checkpointing API). Default is `512`. | -| `batch_size` | Integer | Representing max number of records the buffer returns on read. Default is `48`. | -| `aws_region` | String | Represents the AWS Region to use `ACM`, `S3` or `AWS Cloud Map` and is required when any of the following conditions are met:
- The `use_acm_certificate_for_ssl` setting is `true`.
- Either `ssl_certificate_file` or `ssl_key_file` specifies an AWS S3 URI (for example, s3://mybucket/path/to/public.cert).
- The `discovery_mode` setting is `aws_cloud_map`. | -| `drain_timeout` | Duration | Represents the wait time for the Peer Forwarder to complete processing data before shutdown. | +| `batch_size` | Integer | Represents the maximum number of records that the buffer returns on read. Default is `48`. | +| `aws_region` | String | Represents the AWS Region that uses `ACM`, `S3`, or `AWS Cloud Map` and is required when any of the following conditions are met:
- The `use_acm_certificate_for_ssl` setting is set to `true`.
- Either `ssl_certificate_file` or `ssl_key_file` specifies an Amazon Simple Storage Service (Amazon S3) URI (for example, s3://mybucket/path/to/public.cert).
- The `discovery_mode` is set to `aws_cloud_map`. | +| `drain_timeout` | Duration | Represents the amount of time that Peer Forwarder will wait to complete data processing before shutdown. | ## SSL configuration -The following SSL configuration table provides optional SSL configuration values and allows you to set up trust manager for the peer forwarding client to connect to other Data Prepper instances. +The following SSL configuration table provides optional SSL configuration values and allows you to set up Trust Manager for the Peer Forwarder client to connect to other Data Prepper instances. | Value | Type | Description | | ----- | ---- | ----------- | | `ssl` | Boolean | Enables TLS/SSL. Default value is `true`. | -| `ssl_certificate_file`| String | Represents the SSL certificate chain file path or Amazon Simple Storage Service (Amazon S3) path. The following is an example of an Amazon S3 path: `s3:///`. Defaults to the default certificate file,`config/default_certificate.pem`. See [Default Certificates](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates) for more information about how the certificate is generated. | -| `ssl_key_file`| String | Represents the SSL key file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_private_key.pem` which is the default private key file. Read more about how the private key file is generated at the [Default Certificates](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates) page. | +| `ssl_certificate_file`| String | Represents the SSL certificate chain file path or Amazon S3 path. The following is an example of an Amazon S3 path: `s3:///`. Defaults to the default certificate file,`config/default_certificate.pem`. See [Default Certificates](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates) for more information about how the certificate is generated. | +| `ssl_key_file`| String | Represents the SSL key file path or Amazon S3 path. Amazon S3 path example: `s3:///`. Defaults to `config/default_private_key.pem` which is the default private key file. See [Default Certificates](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates) for more information about how the private key file is generated. | | `ssl_insecure_disable_verification` | Boolean | Disables the verification of the server's TLS certificate chain. Default value is `false`. | | `ssl_fingerprint_verification_only` | Boolean | Disables the verification of the server's TLS certificate chain and instead verifies only the certificate fingerprint. Default value is `false`. | | `use_acm_certificate_for_ssl` | Boolean | Enables TLS/SSL using the certificate and private key from AWS Certificate Manager (ACM). Default value is `false`. | -| `acm_certificate_arn`| String | Represents the ACM certificate Amazon Resource Name (ARN). The ACM certificate takes precedence over S3 or the local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. | +| `acm_certificate_arn`| String | Represents the ACM certificate Amazon Resource Name (ARN). The ACM certificate takes precedence over Amazon S3 or the local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. | | `acm_private_key_password` | String | Represents the ACM private key password that will be used to decrypt the private key. If it's not provided, a random password will be generated. | | `acm_certificate_timeout_millis` | Integer | Represents the timeout in milliseconds required for ACM to get certificates. Default value is `120000`. | -| `aws_region` | String | Represents the AWS Region that uses `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file`. Also required when the `ssl_key_file` is set to the `AWS S3` path, or if `discovery_mode` is set to `aws_cloud_map`. | +| `aws_region` | String | Represents the AWS Region that uses `ACM`, `S3`, or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file`. Also required when the `ssl_key_file` is set to the `AWS S3` path, or if `discovery_mode` is set to `aws_cloud_map`. | #### Example configuration @@ -137,7 +137,7 @@ peer_forwarder: ## Authentication -`Authentication` is optional and is a `Map` that enables mutual TLS (mTLS). It can either be `mutual_tls` or `unauthenticated`. The default value is `unauthenticated`. The following YAML file provides an exmaple for authentication: +`Authentication` is optional and is a `Map` that enables mutual TLS (mTLS). It can either be `mutual_tls` or `unauthenticated`. The default value is `unauthenticated`. The following YAML file provides an example of authentication: ```yaml peer_forwarder: @@ -163,12 +163,12 @@ The following table provides counter metric options. | Value | Description | | ----- | ----------- | | `requests`| Measures the total number of forwarded requests. | -| `requestsFailed`| Measures the total number of failed requests. Applies to requests with HTTP response code other than `200`. | +| `requestsFailed`| Measures the total number of failed requests. Applies to requests with an HTTP response code other than `200`. | | `requestsSuccessful`| Measures the total number of successful requests. Applies to requests with HTTP response code `200`. | | `requestsTooLarge`| Measures the total number of requests that are too large to be written to the Peer Forwarder buffer. Applies to requests with HTTP response code `413`. | | `requestTimeouts`| Measures the total number of requests that time out while writing content to the Peer Forwarder buffer. Applies to requests with HTTP response code `408`. | | `requestsUnprocessable`| Measures the total number of requests that fail due to an unprocessable entity. Applies to requests with HTTP response code `422`. | -| `badRequests`| Measures the total number of requests with bad request format. Applies to requests with HTTP response code `400`. | +| `badRequests`| Measures the total number of requests with a bad request format. Applies to requests with HTTP response code `400`. | | `recordsSuccessfullyForwarded`| Measures the total number of successfully forwarded records. | | `recordsFailedForwarding`| Measures the total number of records that fail to be forwarded. | | `recordsToBeForwarded` | Measures the total number of records to be forwarded. | From 0a4fa3bd8f139334bb7016603c893136b6ce0106 Mon Sep 17 00:00:00 2001 From: carolxob Date: Wed, 1 Feb 2023 12:22:38 -0700 Subject: [PATCH 43/49] Minor update to line 112. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 8972099519..590fb4970c 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -109,7 +109,7 @@ The following table provides optional configuration values. ## SSL configuration -The following SSL configuration table provides optional SSL configuration values and allows you to set up Trust Manager for the Peer Forwarder client to connect to other Data Prepper instances. +The following SSL configuration table provides optional SSL configuration values that you to set up a trust manager for the Peer Forwarder client in order to connect to other Data Prepper instances. | Value | Type | Description | | ----- | ---- | ----------- | From e85fec4ea1759645231bc6ce78e70927aa1c80fc Mon Sep 17 00:00:00 2001 From: carolxob Date: Wed, 1 Feb 2023 15:30:48 -0700 Subject: [PATCH 44/49] Made minor edits to references to S3. Changed to Amazon S3 where applicable. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 590fb4970c..185b0b29f1 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -122,7 +122,7 @@ The following SSL configuration table provides optional SSL configuration values | `acm_certificate_arn`| String | Represents the ACM certificate Amazon Resource Name (ARN). The ACM certificate takes precedence over Amazon S3 or the local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. | | `acm_private_key_password` | String | Represents the ACM private key password that will be used to decrypt the private key. If it's not provided, a random password will be generated. | | `acm_certificate_timeout_millis` | Integer | Represents the timeout in milliseconds required for ACM to get certificates. Default value is `120000`. | -| `aws_region` | String | Represents the AWS Region that uses `ACM`, `S3`, or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file`. Also required when the `ssl_key_file` is set to the `AWS S3` path, or if `discovery_mode` is set to `aws_cloud_map`. | +| `aws_region` | String | Represents the AWS Region that uses `ACM`, `Amazon S3`, or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file`. Also required when the `ssl_key_file` is set to the `Amazon S3` path, or if `discovery_mode` is set to `aws_cloud_map`. | #### Example configuration From b3a5b030b813f39b1df2c3867796f9bb7d87c48e Mon Sep 17 00:00:00 2001 From: carolxob Date: Wed, 1 Feb 2023 15:31:06 -0700 Subject: [PATCH 45/49] Minor edits. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 185b0b29f1..9d07143a68 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -104,7 +104,7 @@ The following table provides optional configuration values. | `aws_cloud_map_query_parameters` | Map | Key-value pairs used to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. | | `buffer_size` | Integer | Represents the maximum number of unchecked records the buffer accepts (the number of unchecked records equals the number of records written into the buffer plus the number of records that are still processing and not yet checked by the Checkpointing API). Default is `512`. | | `batch_size` | Integer | Represents the maximum number of records that the buffer returns on read. Default is `48`. | -| `aws_region` | String | Represents the AWS Region that uses `ACM`, `S3`, or `AWS Cloud Map` and is required when any of the following conditions are met:
- The `use_acm_certificate_for_ssl` setting is set to `true`.
- Either `ssl_certificate_file` or `ssl_key_file` specifies an Amazon Simple Storage Service (Amazon S3) URI (for example, s3://mybucket/path/to/public.cert).
- The `discovery_mode` is set to `aws_cloud_map`. | +| `aws_region` | String | Represents the AWS Region that uses `ACM`, `Amazon S3`, or `AWS Cloud Map` and is required when any of the following conditions are met:
- The `use_acm_certificate_for_ssl` setting is set to `true`.
- Either `ssl_certificate_file` or `ssl_key_file` specifies an Amazon Simple Storage Service (Amazon S3) URI (for example, s3://mybucket/path/to/public.cert).
- The `discovery_mode` is set to `aws_cloud_map`. | | `drain_timeout` | Duration | Represents the amount of time that Peer Forwarder will wait to complete data processing before shutdown. | ## SSL configuration From 8aab515527251186b48ce4ac01ac2b52f9f72a23 Mon Sep 17 00:00:00 2001 From: carolxob Date: Thu, 2 Feb 2023 15:58:45 -0700 Subject: [PATCH 46/49] Minor updates to text. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 9d07143a68..f385b6a256 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -109,7 +109,7 @@ The following table provides optional configuration values. ## SSL configuration -The following SSL configuration table provides optional SSL configuration values that you to set up a trust manager for the Peer Forwarder client in order to connect to other Data Prepper instances. +The following table provides optional SSL configuration values that allow you to set up a trust manager for the Peer Forwarder client in order to connect to other Data Prepper instances. | Value | Type | Description | | ----- | ---- | ----------- | @@ -122,7 +122,7 @@ The following SSL configuration table provides optional SSL configuration values | `acm_certificate_arn`| String | Represents the ACM certificate Amazon Resource Name (ARN). The ACM certificate takes precedence over Amazon S3 or the local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. | | `acm_private_key_password` | String | Represents the ACM private key password that will be used to decrypt the private key. If it's not provided, a random password will be generated. | | `acm_certificate_timeout_millis` | Integer | Represents the timeout in milliseconds required for ACM to get certificates. Default value is `120000`. | -| `aws_region` | String | Represents the AWS Region that uses `ACM`, `Amazon S3`, or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file`. Also required when the `ssl_key_file` is set to the `Amazon S3` path, or if `discovery_mode` is set to `aws_cloud_map`. | +| `aws_region` | String | Represents the AWS Region that uses `ACM`, `Amazon S3`, or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file`. Also required when the `ssl_key_file` is set to use an Amazon S3 path, or if `discovery_mode` is set to `aws_cloud_map`. | #### Example configuration From d949d1f49a00dea6d724a0b14d496228896247d8 Mon Sep 17 00:00:00 2001 From: carolxob Date: Thu, 2 Feb 2023 16:09:17 -0700 Subject: [PATCH 47/49] Made updates based on SME and doc review feedback. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index f385b6a256..a4f327c748 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -122,7 +122,7 @@ The following table provides optional SSL configuration values that allow you to | `acm_certificate_arn`| String | Represents the ACM certificate Amazon Resource Name (ARN). The ACM certificate takes precedence over Amazon S3 or the local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. | | `acm_private_key_password` | String | Represents the ACM private key password that will be used to decrypt the private key. If it's not provided, a random password will be generated. | | `acm_certificate_timeout_millis` | Integer | Represents the timeout in milliseconds required for ACM to get certificates. Default value is `120000`. | -| `aws_region` | String | Represents the AWS Region that uses `ACM`, `Amazon S3`, or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file`. Also required when the `ssl_key_file` is set to use an Amazon S3 path, or if `discovery_mode` is set to `aws_cloud_map`. | +| `aws_region` | String | Represents the AWS Region that uses ACM, Amazon S3, or AWS Cloud Map. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file`. Also required when the `ssl_key_file` is set to use an Amazon S3 path, or if `discovery_mode` is set to `aws_cloud_map`. | #### Example configuration From ccc1ab12047ec269d0fc0473909127479f4aa90a Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 3 Feb 2023 12:34:58 -0700 Subject: [PATCH 48/49] Minor change to one word. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index a4f327c748..186ca6bb28 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -122,7 +122,7 @@ The following table provides optional SSL configuration values that allow you to | `acm_certificate_arn`| String | Represents the ACM certificate Amazon Resource Name (ARN). The ACM certificate takes precedence over Amazon S3 or the local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. | | `acm_private_key_password` | String | Represents the ACM private key password that will be used to decrypt the private key. If it's not provided, a random password will be generated. | | `acm_certificate_timeout_millis` | Integer | Represents the timeout in milliseconds required for ACM to get certificates. Default value is `120000`. | -| `aws_region` | String | Represents the AWS Region that uses ACM, Amazon S3, or AWS Cloud Map. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file`. Also required when the `ssl_key_file` is set to use an Amazon S3 path, or if `discovery_mode` is set to `aws_cloud_map`. | +| `aws_region` | String | Represents the AWS Region that uses ACM, Amazon S3, or AWS Cloud Map. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file`. Also required when the `ssl_key_file` is set to use the Amazon S3 path, or if `discovery_mode` is set to `aws_cloud_map`. | #### Example configuration From f165005594a8c7f05983bf335e534a95e262f18f Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 3 Feb 2023 12:36:04 -0700 Subject: [PATCH 49/49] Removed a comma. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 186ca6bb28..efb62dbb0b 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -122,7 +122,7 @@ The following table provides optional SSL configuration values that allow you to | `acm_certificate_arn`| String | Represents the ACM certificate Amazon Resource Name (ARN). The ACM certificate takes precedence over Amazon S3 or the local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. | | `acm_private_key_password` | String | Represents the ACM private key password that will be used to decrypt the private key. If it's not provided, a random password will be generated. | | `acm_certificate_timeout_millis` | Integer | Represents the timeout in milliseconds required for ACM to get certificates. Default value is `120000`. | -| `aws_region` | String | Represents the AWS Region that uses ACM, Amazon S3, or AWS Cloud Map. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file`. Also required when the `ssl_key_file` is set to use the Amazon S3 path, or if `discovery_mode` is set to `aws_cloud_map`. | +| `aws_region` | String | Represents the AWS Region that uses ACM, Amazon S3, or AWS Cloud Map. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file`. Also required when the `ssl_key_file` is set to use the Amazon S3 path or if `discovery_mode` is set to `aws_cloud_map`. | #### Example configuration