Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add documentation for Mutate strings #2950

Merged
merged 50 commits into from
Mar 16, 2023
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
8077116
Added placeholder files.
carolxob Feb 17, 2023
f9dc552
Added source content from Data Prepper repo docs.
carolxob Feb 17, 2023
8b0091a
Edits for consistency and formatting.
carolxob Feb 20, 2023
52bcb93
Minor edits to otel-metrics-string.
carolxob Feb 20, 2023
d3a11c3
Content and formatting edits.
carolxob Feb 20, 2023
1da45b8
Minor edits to trace-peer-forwarder
carolxob Feb 20, 2023
fbb7aa1
Minor edits.
carolxob Feb 20, 2023
7bf3eec
Updated links to existing Data Prepper documentation as relative URLs.
carolxob Feb 20, 2023
c6c6478
Minor updates.
carolxob Feb 23, 2023
0bf8376
Minor edits.
carolxob Feb 28, 2023
c70652a
Minor edits.
carolxob Feb 28, 2023
cbaa376
Minor edits for consistency and clarity.
carolxob Feb 28, 2023
ea845e3
Minor updates.
carolxob Feb 28, 2023
c192dc3
Minor update.
carolxob Mar 1, 2023
d44a4ec
minor updates.
carolxob Mar 2, 2023
d0bef9e
Minor updates.
carolxob Mar 2, 2023
852ef1d
Adjustments based on technical feedback.
carolxob Mar 2, 2023
c49f88c
Removed 'Developer guide' section.
carolxob Mar 2, 2023
272a987
Minor adjustments for clarity.
carolxob Mar 2, 2023
cba951e
Minor adjustments for consistency.
carolxob Mar 2, 2023
2ca9cf8
Removed markdown comment.
carolxob Mar 2, 2023
bb7850e
Minor edit.
carolxob Mar 2, 2023
41ffd68
Updates made based on doc review feedback.
carolxob Mar 2, 2023
300d980
Minor edits.
carolxob Mar 2, 2023
f67888c
Removed bullet for single item list.
carolxob Mar 2, 2023
b258c0d
Edits made based on doc review feedback.
carolxob Mar 3, 2023
26a4567
Modified file title and doc title to more accurately reflect contents.
carolxob Mar 6, 2023
8d5dd02
Incorporated feedback from tech review.
carolxob Mar 6, 2023
9f4fab0
Minor edit.
carolxob Mar 6, 2023
b29bca1
Added note to convert to table.
carolxob Mar 6, 2023
b552822
Updated links.
carolxob Mar 6, 2023
00d4af7
Incorporated most doc review feedback.
carolxob Mar 7, 2023
e0135e3
Removed unaffected files.
carolxob Mar 7, 2023
54403c3
Minor updates.
carolxob Mar 7, 2023
008e1ee
Minor adjustements.
carolxob Mar 7, 2023
c840765
Added some clarification around the substitute string processor.
carolxob Mar 8, 2023
f653d79
Adjusted table boundaries.
carolxob Mar 8, 2023
32d6a29
Minor update based on tech review feedback.
carolxob Mar 8, 2023
4719163
Removed Basic from Usage section titles.
carolxob Mar 8, 2023
b7f221e
Added link placeholder for config file instructions.
carolxob Mar 13, 2023
2a2a9b8
Minor edit.
carolxob Mar 13, 2023
99234fc
Minor update with sentence structure.
carolxob Mar 13, 2023
13de038
Refined steps around Data Prepper configuration files.
carolxob Mar 13, 2023
7a7b2f1
Renamed file for consistency.
carolxob Mar 13, 2023
bc81921
Changed phrasing for consistency.
carolxob Mar 13, 2023
e8a975d
Removed redundant processor docs.
carolxob Mar 14, 2023
29966e1
Minor updates from doc review feedback.
carolxob Mar 14, 2023
c9ce370
Fixed header justification alignment.
carolxob Mar 14, 2023
03685a6
Made edits based on editorial review.
carolxob Mar 16, 2023
543d9c8
Merging.
carolxob Mar 16, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
288 changes: 288 additions & 0 deletions _data-prepper/pipelines/configuration/processors/mutate-event.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,288 @@
---
layout: default
title: Mutate event
parent: Processors
grand_parent: Pipelines
nav_order: 45
---

# Mutate event processors

The following processors allow you to mutate an event.
carolxob marked this conversation as resolved.
Show resolved Hide resolved

<!--- Why would users want to mutate an event? What does it achieve?--->
carolxob marked this conversation as resolved.
Show resolved Hide resolved

## AddEntry

The `AddEntry` processor adds entries to an event.

### Basic usage

To get started, create the following `pipeline.yaml` file:

```yaml
pipeline:
source:
file:
path: "/full/path/to/logs_json.log"
record_type: "event"
format: "json"
processor:
- add_entries:
entries:
- key: "newMessage"
value: 3
overwrite_if_key_exists: true
sink:
- stdout:
```

Create the following file named `logs_json.log` and replace the `path` in the file source of your `pipeline.yaml` with this filepath.

```json
{"message": "value"}
carolxob marked this conversation as resolved.
Show resolved Hide resolved
```

When run, the processor parses the message into the following output:

```json
{"message": "value", "newMessage": 3}
carolxob marked this conversation as resolved.
Show resolved Hide resolved
```

> If `newMessage` already exists, its existing value is overwritten with a value of `3`.

### Configuration

* `entries` (required): A list of entries to add to an event
carolxob marked this conversation as resolved.
Show resolved Hide resolved
* `key` (required): The key of the new entry to be added
* `value` (required): The value of the new entry to be added. Strings, booleans, numbers, null, nested objects, and arrays containing the aforementioned data types are valid to use
carolxob marked this conversation as resolved.
Show resolved Hide resolved
* `overwrite_if_key_exists` (optional): When set to `true`, if `key` already exists in the event, then the existing value is overwritten. The default value is `false`.

## Copy value

The `copy value` processor copies values within an event.
carolxob marked this conversation as resolved.
Show resolved Hide resolved

### Basic usage

To get started, create the following `pipeline.yaml` file:

```yaml
pipeline:
source:
file:
path: "/full/path/to/logs_json.log"
record_type: "event"
format: "json"
processor:
- copy_values:
entries:
- from_key: "message"
to_key: "newMessage"
overwrite_if_to_key_exists: true
sink:
- stdout:
```

Create the following file named `logs_json.log` and replace the `path` in the file source of your `pipeline.yaml` with the path of this file:

```json
{"message": "value"}
```

When run, the processor parses the message into the following output:

```json
{"message": "value", "newMessage": "value"}
```

> If `newMessage` had already existed, its existing value would have been overwritten with `value`

### Configuration
* `entries` - (required) - A list of entries to be copied in an event
carolxob marked this conversation as resolved.
Show resolved Hide resolved
* `from_key` - (required) - The key of the entry to be copied
* `to_key` - (required) - The key of the new entry to be added
* `overwrite_if_to_key_exists` - (optional) - When set to `true`, if `to_key` already exists in the event, then the existing value will be overwritten. The default is `false`.


## DeleteEntry

The `DeleteEntry` processor deletes entries in an event.
carolxob marked this conversation as resolved.
Show resolved Hide resolved

### Basic usage

To get started, create the following `pipeline.yaml` file:

```yaml
pipeline:
source:
file:
path: "/full/path/to/logs_json.log"
record_type: "event"
format: "json"
processor:
- delete_entries:
with_keys: ["message"]
sink:
- stdout:
```

Create the following file named `logs_json.log` and replace the `path` in the file source of your `pipeline.yaml` with the path of this file.

```json
{"message": "value", "message2": "value2"}
```

When run, the processor parses the message into the following output:

```json
{"message2": "value2"}
```

> If `message` had not existed in the event, then nothing would have happened

### Configuration

<!---Need some intro text.--->
carolxob marked this conversation as resolved.
Show resolved Hide resolved

* `with_keys` - (required) - An array of keys of the entries to be deleted.


## RenameKey

The `rename key` processor renames keys in an event.

### Basic usage

To get started, create the following `pipeline.yaml`.
```yaml
pipeline:
source:
file:
path: "/full/path/to/logs_json.log"
record_type: "event"
format: "json"
processor:
- rename_keys:
entries:
- from_key: "message"
to_key: "newMessage"
overwrite_if_to_key_exists: true
sink:
- stdout:
```

Create the following file named `logs_json.log` and replace the `path` in the file source of your `pipeline.yaml` with the path of this file.

```json
{"message": "value"}
```

When run, the processor parses the message into the following output:
carolxob marked this conversation as resolved.
Show resolved Hide resolved

```json
{"newMessage": "value"}
```

> If `newMessage` already exists, its existing value is overwritten with `value`.

### Configuration

<!--- Need some intro text here.--->
carolxob marked this conversation as resolved.
Show resolved Hide resolved

* `entries` - (required) - A list of entries to rename in an event
* `from_key` - (required) - The key of the entry to be renamed
* `to_key` - (required) - The new key of the entry
* `overwrite_if_to_key_exists` - (optional) - When set to `true`, if `to_key` already exists in the event, then the existing value will be overwritten. The default is `false`.

### Special considerations

The renaming operation occurs in a defined order. <!--- Where is this order defined?---> This means that chaining is implicit with the `RenameKey` processor. See the following `piplines.yaml` file example:
carolxob marked this conversation as resolved.
Show resolved Hide resolved

```yaml
pipeline:
source:
file:
path: "/full/path/to/logs_json.log"
record_type: "event"
format: "json"
processor:
- rename_key:
entries:
- from_key: "message"
to_key: "message2"
- from_key: "message2"
to_key: "message3"
sink:
- stdout:
```

Add the following contents to the `logs_json.log` file:

```json
{"message": "value"}
```

After the processor runs, the following output appears:

```json
{"message3": "value"}
carolxob marked this conversation as resolved.
Show resolved Hide resolved
```

## ConvertEntry

The `ConvertEntry` processor converts the type of value associated with the specified key in a message to the specified type. It is a casting processor that changes the types of some fields in the event or message. Some of inputted data may need to be converted to different types, such as an integer or a double. The data may need to be converted so that it will pass the events through condition-based processors, or to perform conditional routing.

## Basic usage

To get started with type conversion processor using Data Prepper, create the following `pipeline.yaml` file:
carolxob marked this conversation as resolved.
Show resolved Hide resolved

```yaml
type-conv-pipeline:
source:
file:
path: "/full/path/to/logs_json.log"
record_type: "event"
format: "json"
processor:
- grok:
match:
message: ['%{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] %{NUMBER:response_status}']
- convert_entry_type:
key: "response_status"
type: "integer"
sink:
- stdout:
```

Create the following file named `logs_json.log` and replace the `path` in the file source of your `pipeline.yaml` with the path of this file.

```json
{"message": "10.10.10.19 [19/Feb/2015:15:50:36 -0500] 200"}
```

When run, the `Grok` processor parses the message into the following output:

```json
{"message": "10.10.10.10 [19/Feb/2015:15:50:36 -0500] 200", "clientip":"10.10.10.10", "timestamp": "19/Feb/2015:15:50:36 -0500", "response_status": "200"}
```

The type conversion processor changes the output received into the following output, where the type of `response_status` value changes to an integer:

```json
{"message": "10.10.10.10 [19/Feb/2015:15:50:36 -0500] 200", "clientip":"10.10.10.10", "timestamp": "19/Feb/2015:15:50:36 -0500", "response_status": 200}
```

### Configuration

* `key` (required): Keys whose value needs to be converted to a different type
carolxob marked this conversation as resolved.
Show resolved Hide resolved
* `type`: Target type for key value. Possible values are `integer`, `double`, `string`, and `boolean`. Default value is `integer`.


## Developer guide

This plugin is compatible with Java 14. See the following:

- [Contributing](https://github.com/opensearch-project/data-prepper/blob/main/CONTRIBUTING.md)
- [Monitoring]({{site.url}}{{site.baseurl}}/data-prepper/monitoring/)


Loading