Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add documentation for Mutate strings #2950

Merged
merged 50 commits into from
Mar 16, 2023
Merged
Show file tree
Hide file tree
Changes from 49 commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
8077116
Added placeholder files.
carolxob Feb 17, 2023
f9dc552
Added source content from Data Prepper repo docs.
carolxob Feb 17, 2023
8b0091a
Edits for consistency and formatting.
carolxob Feb 20, 2023
52bcb93
Minor edits to otel-metrics-string.
carolxob Feb 20, 2023
d3a11c3
Content and formatting edits.
carolxob Feb 20, 2023
1da45b8
Minor edits to trace-peer-forwarder
carolxob Feb 20, 2023
fbb7aa1
Minor edits.
carolxob Feb 20, 2023
7bf3eec
Updated links to existing Data Prepper documentation as relative URLs.
carolxob Feb 20, 2023
c6c6478
Minor updates.
carolxob Feb 23, 2023
0bf8376
Minor edits.
carolxob Feb 28, 2023
c70652a
Minor edits.
carolxob Feb 28, 2023
cbaa376
Minor edits for consistency and clarity.
carolxob Feb 28, 2023
ea845e3
Minor updates.
carolxob Feb 28, 2023
c192dc3
Minor update.
carolxob Mar 1, 2023
d44a4ec
minor updates.
carolxob Mar 2, 2023
d0bef9e
Minor updates.
carolxob Mar 2, 2023
852ef1d
Adjustments based on technical feedback.
carolxob Mar 2, 2023
c49f88c
Removed 'Developer guide' section.
carolxob Mar 2, 2023
272a987
Minor adjustments for clarity.
carolxob Mar 2, 2023
cba951e
Minor adjustments for consistency.
carolxob Mar 2, 2023
2ca9cf8
Removed markdown comment.
carolxob Mar 2, 2023
bb7850e
Minor edit.
carolxob Mar 2, 2023
41ffd68
Updates made based on doc review feedback.
carolxob Mar 2, 2023
300d980
Minor edits.
carolxob Mar 2, 2023
f67888c
Removed bullet for single item list.
carolxob Mar 2, 2023
b258c0d
Edits made based on doc review feedback.
carolxob Mar 3, 2023
26a4567
Modified file title and doc title to more accurately reflect contents.
carolxob Mar 6, 2023
8d5dd02
Incorporated feedback from tech review.
carolxob Mar 6, 2023
9f4fab0
Minor edit.
carolxob Mar 6, 2023
b29bca1
Added note to convert to table.
carolxob Mar 6, 2023
b552822
Updated links.
carolxob Mar 6, 2023
00d4af7
Incorporated most doc review feedback.
carolxob Mar 7, 2023
e0135e3
Removed unaffected files.
carolxob Mar 7, 2023
54403c3
Minor updates.
carolxob Mar 7, 2023
008e1ee
Minor adjustements.
carolxob Mar 7, 2023
c840765
Added some clarification around the substitute string processor.
carolxob Mar 8, 2023
f653d79
Adjusted table boundaries.
carolxob Mar 8, 2023
32d6a29
Minor update based on tech review feedback.
carolxob Mar 8, 2023
4719163
Removed Basic from Usage section titles.
carolxob Mar 8, 2023
b7f221e
Added link placeholder for config file instructions.
carolxob Mar 13, 2023
2a2a9b8
Minor edit.
carolxob Mar 13, 2023
99234fc
Minor update with sentence structure.
carolxob Mar 13, 2023
13de038
Refined steps around Data Prepper configuration files.
carolxob Mar 13, 2023
7a7b2f1
Renamed file for consistency.
carolxob Mar 13, 2023
bc81921
Changed phrasing for consistency.
carolxob Mar 13, 2023
e8a975d
Removed redundant processor docs.
carolxob Mar 14, 2023
29966e1
Minor updates from doc review feedback.
carolxob Mar 14, 2023
c9ce370
Fixed header justification alignment.
carolxob Mar 14, 2023
03685a6
Made edits based on editorial review.
carolxob Mar 16, 2023
543d9c8
Merging.
carolxob Mar 16, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

This file was deleted.

256 changes: 256 additions & 0 deletions _data-prepper/pipelines/configuration/processors/mutate-string.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,256 @@
---
layout: default
title: Mutate string
parent: Processors
grand_parent: Pipelines
nav_order: 45
---

# Mutate string processors

You can change the way that a string appears by using a mutate string processesor. For example, you can use the `uppercase_string` processor to convert a string to uppercase, and you can use the `lowercase_string` processor to convert a string to lowercase. The following is a list of processors that allow you to mutate a string:

* [substitute_string](#substitute_string)
* [split_string](#split_string)
* [uppercase_string](#uppercase_string)
* [lowercase_string](#lowercase_string)
* [trim_string](#trim_string)

## substitute_string

The `substitute_string` processor matches a key's value against a regular expression (regex) and replaces all returned matches with a replacement string.

### Configuration

You can configure the `substitute_string` processor with the following options.

Option | Required | Description
:--- | :--- | :---
`entries` | Yes | A list of entries to add to an event. |
`source` | Yes | The key to be modified. |
`from` | Yes | The regex string to be replaced. Special regex characters such as `[` and `]` must be escaped using `\\` when using double quotes and `\` when using single quotes. For more information, see [Class Pattern](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/util/regex/Pattern.html) in the Java documentation. |
`to` | Yes | The string that replaces each match of `from`. |

### Usage

To get started, create the following `pipeline.yaml` file:

```yaml
pipeline:
source:
file:
path: "/full/path/to/logs_json.log"
record_type: "event"
format: "json"
processor:
- substitute_string:
entries:
- source: "message"
from: ":"
to: "-"
sink:
- stdout:
```
{% include copy.html %}
Next, create a log file named `logs_json.log`. After that, replace the `path` of the file source in your `pipeline.yaml` file with your file path. For more detailed information, see [Configuring Data Prepper]({{site.url}}{{site.baseurl}}/data-prepper/getting-started/#2-configuring-data-prepper).

Before you run Data Prepper, the source appears in the following format:

```json
{"message": "ab:cd:ab:cd"}
```

After you run Data Prepper, the source is converted to the following format:

```json
{"message": "ab-cd-ab-cd"}
```

`from` defines which string is replaced, and `to` defines the string that replaces the `from` string. In the preceding example, string `ab:cd:ab:cd` becomes `ab-cd-ab-cd`. If the `from` regex string does not return a match, the key is returned without any changes.

## split_string

The `split_string` processor splits a field into an array using a delimiter character.

### Configuration

You can configure the `split_string` processor with the following options.

Option | Required | Description
:--- | :--- | :---
`entries` | Yes | A list of entries to add to an event. |
`source` | Yes | The key to be split. |
`delimiter` | No | The separator character responsible for the split. Cannot be defined at the same time as `delimiter_regex`. At least `delimiter` or `delimiter_regex` must be defined. |
carolxob marked this conversation as resolved.
Show resolved Hide resolved
`delimiter_regex` | No | A regex string responsible for the split. Cannot be defined at the same time as `delimiter`. Either `delimiter` or `delimiter_regex` must be defined. |

### Usage

To get started, create the following `pipeline.yaml` file:

```yaml
pipeline:
source:
file:
path: "/full/path/to/logs_json.log"
record_type: "event"
format: "json"
processor:
- split_string:
entries:
- source: "message"
delimiter: ","
sink:
- stdout:
```
{% include copy.html %}

Next, create a log file named `logs_json.log`. After that, replace the `path` in the file source of your `pipeline.yaml` file with your file path. For more detailed information, see [Configuring Data Prepper]({{site.url}}{{site.baseurl}}/data-prepper/getting-started/#2-configuring-data-prepper).

Before you run Data Prepper, the source appears in the following format:

```json
{"message": "hello,world"}
```
After you run Data Prepper, the source is converted to the following format:

```json
{"message":["hello","world"]}
```

## uppercase_string

The `uppercase_string` processor converts the value (a string) of a key from its current case to uppercase.

### Configuration

You can configure the `uppercase_string` processor with the following options.

Option | Required | Description
:--- | :--- | :---
`with_keys` | Yes | A list of keys to convert to uppercase. |

### Usage

To get started, create the following `pipeline.yaml` file:

```yaml
pipeline:
source:
file:
path: "/full/path/to/logs_json.log"
record_type: "event"
format: "json"
processor:
- uppercase_string:
with_keys:
- "uppercaseField"
sink:
- stdout:
```
{% include copy.html %}

Next, create a log file named `logs_json.log`. After that, replace the `path` in the file source of your `pipeline.yaml` file with the correct file path. For more detailed information, see [Configuring Data Prepper]({{site.url}}{{site.baseurl}}/data-prepper/getting-started/#2-configuring-data-prepper).

Before you run Data Prepper, the source appears in the following format:

```json
{"uppercaseField": "hello"}
```
After you run Data Prepper, the source is converted to the following format:

```json
{"uppercaseField": "HELLO"}
```

## lowercase_string

The `lowercase string` processor converts a string to lowercase.

### Configuration

You can configure the `lowercase string` processor with the following options.

Option | Required | Description
:--- | :--- | :---
`with_keys` | Yes | A list of keys to convert to lowercase. |

### Usage

To get started, create the following `pipeline.yaml` file:

```yaml
pipeline:
source:
file:
path: "/full/path/to/logs_json.log"
record_type: "event"
format: "json"
processor:
- lowercase_string:
with_keys:
- "lowercaseField"
sink:
- stdout:
```
{% include copy.html %}

Next, create a log file named `logs_json.log`. After that, replace the `path` in the file source of your `pipeline.yaml` file with the correct file path. For more detailed information, see [Configuring Data Prepper]({{site.url}}{{site.baseurl}}/data-prepper/getting-started/#2-configuring-data-prepper).

Before you run Data Prepper, the source appears in the following format:

```json
{"lowercaseField": "TESTmeSSage"}
```

After you run Data Prepper, the source is converted to the following format:

```json
{"lowercaseField": "testmessage"}
```

## trim_string

The `trim_string` processor removes whitespace from the beginning and end of a key.

### Configuration

You can configure the `trim_string` processor with the following options.

Option | Required | Description
:--- | :--- | :---
`with_keys` | Yes | A list of keys from which to trim the whitespace. |

### Usage

To get started, create the following `pipeline.yaml` file:

```yaml
pipeline:
source:
file:
path: "/full/path/to/logs_json.log"
record_type: "event"
format: "json"
processor:
- trim_string:
with_keys:
- "trimField"
sink:
- stdout:
```
{% include copy.html %}

Next, create a log file named `logs_json.log`. After that, replace the `path` in the file source of your `pipeline.yaml` file with the correct file path. For more detailed information, see [Configuring Data Prepper]({{site.url}}{{site.baseurl}}/data-prepper/getting-started/#2-configuring-data-prepper).

Before you run Data Prepper, the source appears in the following format:

```json
{"trimField": " Space Ship "}
```

After you run Data Prepper, the source is converted to the following format:

```json
{"trimField": "Space Ship"}
```
28 changes: 0 additions & 28 deletions _data-prepper/pipelines/configuration/processors/split-string.md

This file was deleted.

This file was deleted.

25 changes: 0 additions & 25 deletions _data-prepper/pipelines/configuration/processors/trim-string.md

This file was deleted.

This file was deleted.