Add documentation for Mutate strings (#2950)

* Added placeholder files. Signed-off-by: carolxob <[email protected]> * Added source content from Data Prepper repo docs. Signed-off-by: carolxob <[email protected]> * Edits for consistency and formatting. Signed-off-by: carolxob <[email protected]> * Minor edits to otel-metrics-string. Signed-off-by: carolxob <[email protected]> * Content and formatting edits. Signed-off-by: carolxob <[email protected]> * Minor edits to trace-peer-forwarder Signed-off-by: carolxob <[email protected]> * Minor edits. Signed-off-by: carolxob <[email protected]> * Updated links to existing Data Prepper documentation as relative URLs. Signed-off-by: carolxob <[email protected]> * Minor updates. Signed-off-by: carolxob <[email protected]> * Minor edits. Signed-off-by: carolxob <[email protected]> * Minor edits. Signed-off-by: carolxob <[email protected]> * Minor edits for consistency and clarity. Signed-off-by: carolxob <[email protected]> * Minor updates. Signed-off-by: carolxob <[email protected]> * Minor update. Signed-off-by: carolxob <[email protected]> * minor updates. Signed-off-by: carolxob <[email protected]> * Minor updates. Signed-off-by: carolxob <[email protected]> * Adjustments based on technical feedback. Signed-off-by: carolxob <[email protected]> * Removed 'Developer guide' section. Signed-off-by: carolxob <[email protected]> * Minor adjustments for clarity. Signed-off-by: carolxob <[email protected]> * Minor adjustments for consistency. Signed-off-by: carolxob <[email protected]> * Removed markdown comment. Signed-off-by: carolxob <[email protected]> * Minor edit. Signed-off-by: carolxob <[email protected]> * Updates made based on doc review feedback. Signed-off-by: carolxob <[email protected]> * Minor edits. Signed-off-by: carolxob <[email protected]> * Removed bullet for single item list. Signed-off-by: carolxob <[email protected]> * Edits made based on doc review feedback. Signed-off-by: carolxob <[email protected]> * Modified file title and doc title to more accurately reflect contents. Signed-off-by: carolxob <[email protected]> * Incorporated feedback from tech review. Signed-off-by: carolxob <[email protected]> * Minor edit. Signed-off-by: carolxob <[email protected]> * Added note to convert to table. Signed-off-by: carolxob <[email protected]> * Updated links. Signed-off-by: carolxob <[email protected]> * Incorporated most doc review feedback. Signed-off-by: carolxob <[email protected]> * Removed unaffected files. Signed-off-by: carolxob <[email protected]> * Minor updates. Signed-off-by: carolxob <[email protected]> * Minor adjustements. Signed-off-by: carolxob <[email protected]> * Added some clarification around the substitute string processor. Signed-off-by: carolxob <[email protected]> * Adjusted table boundaries. Signed-off-by: carolxob <[email protected]> * Minor update based on tech review feedback. Signed-off-by: carolxob <[email protected]> * Removed Basic from Usage section titles. Signed-off-by: carolxob <[email protected]> * Added link placeholder for config file instructions. Signed-off-by: carolxob <[email protected]> * Minor edit. Signed-off-by: carolxob <[email protected]> * Minor update with sentence structure. Signed-off-by: carolxob <[email protected]> * Refined steps around Data Prepper configuration files. Signed-off-by: carolxob <[email protected]> * Renamed file for consistency. Signed-off-by: carolxob <[email protected]> * Changed phrasing for consistency. Signed-off-by: carolxob <[email protected]> * Removed redundant processor docs. Signed-off-by: carolxob <[email protected]> * Minor updates from doc review feedback. Signed-off-by: carolxob <[email protected]> * Fixed header justification alignment. Signed-off-by: carolxob <[email protected]> * Made edits based on editorial review. Signed-off-by: carolxob <[email protected]> --------- Signed-off-by: carolxob <[email protected]> (cherry picked from commit 0690c47) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
opensearch-project · Mar 16, 2023 · 094ef34 · 094ef34
1 parent b63efc1
commit 094ef34
Showing 1 changed file with 256 additions and 0 deletions.
diff --git a/_data-prepper/pipelines/configuration/processors/mutate-string.md b/_data-prepper/pipelines/configuration/processors/mutate-string.md
@@ -0,0 +1,256 @@
+---
+layout: default
+title: Mutate string
+parent: Processors
+grand_parent: Pipelines
+nav_order: 45
+---
+
+# Mutate string processors
+
+You can change the way that a string appears by using a mutate string processesor. For example, you can use the `uppercase_string` processor to convert a string to uppercase, and you can use the `lowercase_string` processor to convert a string to lowercase. The following is a list of processors that allow you to mutate a string:
+
+* [substitute_string](#substitute_string)
+* [split_string](#split_string)
+* [uppercase_string](#uppercase_string)
+* [lowercase_string](#lowercase_string)
+* [trim_string](#trim_string)
+
+## substitute_string
+
+The `substitute_string` processor matches a key's value against a regular expression (regex) and replaces all returned matches with a replacement string.
+
+### Configuration
+
+You can configure the `substitute_string` processor with the following options.
+
+Option | Required | Description
+:--- | :--- | :---
+`entries` | Yes | A list of entries to add to an event. |
+`source` | Yes | The key to be modified. |
+`from` | Yes | The regex string to be replaced. Special regex characters such as `[` and `]` must be escaped using `\\` when using double quotes and `\` when using single quotes. For more information, see [Class Pattern](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/util/regex/Pattern.html) in the Java documentation. |
+`to` | Yes | The string that replaces each match of `from`. |
+
+### Usage
+
+To get started, create the following `pipeline.yaml` file: 
+
+```yaml
+pipeline:
+  source:
+    file:
+      path: "/full/path/to/logs_json.log"
+      record_type: "event"
+      format: "json"
+  processor:
+    - substitute_string:
+        entries:
+          - source: "message"
+            from: ":"
+            to: "-"
+  sink:
+    - stdout:
+```
+{% include copy.html %}
+
+Next, create a log file named `logs_json.log`. After that, replace the `path` of the file source in your `pipeline.yaml` file with your file path. For more detailed information, see [Configuring Data Prepper]({{site.url}}{{site.baseurl}}/data-prepper/getting-started/#2-configuring-data-prepper). 
+
+Before you run Data Prepper, the source appears in the following format:
+
+```json
+{"message": "ab:cd:ab:cd"}
+```
+
+After you run Data Prepper, the source is converted to the following format:
+
+```json
+{"message": "ab-cd-ab-cd"}
+```
+
+`from` defines which string is replaced, and `to` defines the string that replaces the `from` string. In the preceding example, string `ab:cd:ab:cd` becomes `ab-cd-ab-cd`. If the `from` regex string does not return a match, the key is returned without any changes. 
+
+## split_string
+
+The `split_string` processor splits a field into an array using a delimiter character.
+
+### Configuration
+
+You can configure the `split_string` processor with the following options.
+
+Option | Required | Description
+:--- | :--- | :---
+ `entries` | Yes | A list of entries to add to an event. |
+ `source` | Yes | The key to be split. |
+ `delimiter` | No | The separator character responsible for the split. Cannot be defined at the same time as `delimiter_regex`. At least `delimiter` or `delimiter_regex` must be defined. |
+`delimiter_regex` | No | A regex string responsible for the split. Cannot be defined at the same time as `delimiter`. Either `delimiter` or `delimiter_regex` must be defined. |
+
+### Usage
+
+To get started, create the following `pipeline.yaml` file:
+
+```yaml
+pipeline:
+  source:
+    file:
+      path: "/full/path/to/logs_json.log"
+      record_type: "event"
+      format: "json"
+  processor:
+    - split_string:
+        entries:
+          - source: "message"
+            delimiter: ","
+  sink:
+    - stdout:
+```
+{% include copy.html %}
+
+Next, create a log file named `logs_json.log`. After that, replace the `path` in the file source of your `pipeline.yaml` file with your file path. For more detailed information, see [Configuring Data Prepper]({{site.url}}{{site.baseurl}}/data-prepper/getting-started/#2-configuring-data-prepper). 
+
+Before you run Data Prepper, the source appears in the following format:
+
+```json
+{"message": "hello,world"}
+```
+After you run Data Prepper, the source is converted to the following format:
+
+```json
+{"message":["hello","world"]}
+```
+
+## uppercase_string
+
+The `uppercase_string` processor converts the value (a string) of a key from its current case to uppercase. 
+
+### Configuration
+
+You can configure the `uppercase_string` processor with the following options.
+
+Option | Required | Description
+:--- | :--- | :---
+ `with_keys` | Yes | A list of keys to convert to uppercase. |
+
+### Usage
+
+To get started, create the following `pipeline.yaml` file:
+
+```yaml
+pipeline:
+  source:
+    file:
+      path: "/full/path/to/logs_json.log"
+      record_type: "event"
+      format: "json"
+  processor:
+    - uppercase_string:
+        with_keys:
+          - "uppercaseField"
+  sink:
+    - stdout:
+```
+{% include copy.html %}
+
+Next, create a log file named `logs_json.log`. After that, replace the `path` in the file source of your `pipeline.yaml` file with the correct file path. For more detailed information, see [Configuring Data Prepper]({{site.url}}{{site.baseurl}}/data-prepper/getting-started/#2-configuring-data-prepper). 
+
+Before you run Data Prepper, the source appears in the following format:
+
+```json
+{"uppercaseField": "hello"}
+```
+After you run Data Prepper, the source is converted to the following format:
+
+```json
+{"uppercaseField": "HELLO"}
+```
+
+## lowercase_string
+
+The `lowercase string` processor converts a string to lowercase.
+
+### Configuration
+
+You can configure the `lowercase string` processor with the following options.
+
+Option | Required | Description
+:--- | :--- | :---
+ `with_keys` | Yes | A list of keys to convert to lowercase. |
+
+### Usage
+
+To get started, create the following `pipeline.yaml` file:
+
+```yaml
+pipeline:
+  source:
+    file:
+      path: "/full/path/to/logs_json.log"
+      record_type: "event"
+      format: "json"
+  processor:
+    - lowercase_string:
+        with_keys:
+          - "lowercaseField"
+  sink:
+    - stdout:
+```
+{% include copy.html %}
+
+Next, create a log file named `logs_json.log`. After that, replace the `path` in the file source of your `pipeline.yaml` file with the correct file path. For more detailed information, see [Configuring Data Prepper]({{site.url}}{{site.baseurl}}/data-prepper/getting-started/#2-configuring-data-prepper). 
+
+Before you run Data Prepper, the source appears in the following format:
+
+```json
+{"lowercaseField": "TESTmeSSage"}
+```
+
+After you run Data Prepper, the source is converted to the following format:
+
+```json
+{"lowercaseField": "testmessage"}
+```
+
+## trim_string
+
+The `trim_string` processor removes whitespace from the beginning and end of a key.
+
+### Configuration
+
+You can configure the `trim_string` processor with the following options.
+
+Option | Required | Description
+:--- | :--- | :---
+ `with_keys` | Yes | A list of keys from which to trim the whitespace. |
+
+### Usage
+
+To get started, create the following `pipeline.yaml` file:
+
+```yaml
+pipeline:
+  source:
+    file:
+      path: "/full/path/to/logs_json.log"
+      record_type: "event"
+      format: "json"
+  processor:
+    - trim_string:
+        with_keys:
+          - "trimField"
+  sink:
+    - stdout:
+```
+{% include copy.html %}
+
+Next, create a log file named `logs_json.log`. After that, replace the `path` in the file source of your `pipeline.yaml` file with the correct file path. For more detailed information, see [Configuring Data Prepper]({{site.url}}{{site.baseurl}}/data-prepper/getting-started/#2-configuring-data-prepper). 
+
+Before you run Data Prepper, the source appears in the following format:
+
+```json
+{"trimField": " Space Ship "}
+```
+
+After you run Data Prepper, the source is converted to the following format:
+
+```json
+{"trimField": "Space Ship"}
+```