-
Notifications
You must be signed in to change notification settings - Fork 503
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Add list to map processor. Signed-off-by: Naarcha-AWS <[email protected]> * Tweak one last file Signed-off-by: Naarcha-AWS <[email protected]> * Fix typo Signed-off-by: Naarcha-AWS <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <[email protected]> * Update mutate-event.md * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Chris Moore <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Chris Moore <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Chris Moore <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md * Add Chris' feedback Signed-off-by: Naarcha-AWS <[email protected]> * A couple more wording tweaks Signed-off-by: Naarcha-AWS <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Nathan Bower <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Nathan Bower <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> --------- Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Hai Yan <[email protected]> Co-authored-by: Chris Moore <[email protected]> Co-authored-by: Nathan Bower <[email protected]>
- Loading branch information
Showing
26 changed files
with
657 additions
and
428 deletions.
There are no files selected for viewing
61 changes: 49 additions & 12 deletions
61
_data-prepper/pipelines/configuration/processors/add-entries.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,25 +1,62 @@ | ||
| --- | ||
| layout: default | ||
| title: add_entries | ||
| title: Add entries processor | ||
| parent: Processors | ||
| grand_parent: Pipelines | ||
| nav_order: 45 | ||
| nav_order: 40 | ||
| --- | ||
|
|
||
| # add_entries | ||
|
|
||
| ## Overview | ||
| The `add_entries` processor adds entries to an event. | ||
|
|
||
| The `add_entries` processor adds an entry to the event and is a [mutate event](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/mutate-event-processors#mutate-event-processors) processor. The following table describes the options you can use to configure the `add_entries` processor. | ||
| ### Configuration | ||
|
|
||
| Option | Required | Type | Description | ||
| :--- | :--- | :--- | :--- | ||
| entries | Yes | List | List of events to be added. Valid entries are `key`, `value`, and `overwrite_if_key_exists`. | ||
| key | N/A | N/A | Key of the new event to be added. | ||
| value | N/A | N/A | Value of the new entry to be added. Valid data types are strings, booleans, numbers, null, nested objects, and arrays containing the aforementioned data types. | ||
| overwrite_if_key_exists | No | Boolean | If true, the existing value is overwritten if the key already exists within the event. Default value is `false`. | ||
| You can configure the `add_entries` processor with the following options. | ||
|
|
||
| <!--- ## Configuration | ||
| | Option | Required | Description | | ||
| | :--- | :--- | :--- | | ||
| | `entries` | Yes | A list of entries to add to an event. | | ||
| | `key` | Yes | The key of the new entry to be added. Some examples of keys include `my_key`, `myKey`, and `object/sub_Key`. | | ||
| | `value` | Yes | The value of the new entry to be added. You can use the following data types: strings, Booleans, numbers, null, nested objects, and arrays. | | ||
| | `overwrite_if_key_exists` | No | When set to `true`, the existing value is overwritten if `key` already exists in the event. The default value is `false`. | | ||
|
|
||
| Content will be added to this section.---> | ||
| ### Usage | ||
|
|
||
| To get started, create the following `pipeline.yaml` file: | ||
|
|
||
| ```yaml | ||
| pipeline: | ||
| source: | ||
| file: | ||
| path: "/full/path/to/logs_json.log" | ||
| record_type: "event" | ||
| format: "json" | ||
| processor: | ||
| - add_entries: | ||
| entries: | ||
| - key: "newMessage" | ||
| value: 3 | ||
| overwrite_if_key_exists: true | ||
| sink: | ||
| - stdout: | ||
| ``` | ||
| {% include copy.html %} | ||
| Next, create a log file named `logs_json.log` and replace the `path` in the file source of your `pipeline.yaml` file with that filepath. For more information, see [Configuring Data Prepper]({{site.url}}{{site.baseurl}}/data-prepper/getting-started/#2-configuring-data-prepper). | ||
|
|
||
| For example, before you run the `add_entries` processor, if the `logs_json.log` file contains the following event record: | ||
|
|
||
| ```json | ||
| {"message": "hello"} | ||
| ``` | ||
|
|
||
| Then when you run the `add_entries` processor using the previous configuration, it adds a new entry `{"newMessage": 3}` to the existing event `{"message": "hello"}` so that the new event contains two entries in the final output: | ||
|
|
||
| ```json | ||
| {"message": "hello", "newMessage": 3} | ||
| ``` | ||
|
|
||
| > If `newMessage` already exists, its existing value is overwritten with a value of `3`. | ||
|
|
11 changes: 7 additions & 4 deletions
11
_data-prepper/pipelines/configuration/processors/aggregate.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
55 changes: 55 additions & 0 deletions
55
_data-prepper/pipelines/configuration/processors/convert_entry_type.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| --- | ||
| layout: default | ||
| title: Convert entry type processor | ||
| parent: Processors | ||
| grand_parent: Pipelines | ||
| nav_order: 47 | ||
| --- | ||
|
|
||
| # convert_entry_type_type | ||
|
|
||
| The `convert_entry_type` processor converts a value type associated with the specified key in a event to the specified type. It is a casting processor that changes the types of some fields in events. Some data must be converted to a different type, such as an integer to a double, or a string to an integer, so that it will pass the events through condition-based processors or perform conditional routing. | ||
|
|
||
| ## Configuration | ||
|
|
||
| You can configure the `convert_entry_type` processor with the following options. | ||
|
|
||
| | Option | Required | Description | | ||
| | :--- | :--- | :--- | | ||
| | `key`| Yes | Keys whose value needs to be converted to a different type. | | ||
| | `type` | No | Target type for the key-value pair. Possible values are `integer`, `double`, `string`, and `Boolean`. Default value is `integer`. | | ||
|
|
||
| ## Usage | ||
|
|
||
| To get started, create the following `pipeline.yaml` file: | ||
|
|
||
| ```yaml | ||
| type-conv-pipeline: | ||
| source: | ||
| file: | ||
| path: "/full/path/to/logs_json.log" | ||
| record_type: "event" | ||
| format: "json" | ||
| processor: | ||
| - convert_entry_type_type: | ||
| key: "response_status" | ||
| type: "integer" | ||
| sink: | ||
| - stdout: | ||
| ``` | ||
| {% include copy.html %} | ||
| Next, create a log file named `logs_json.log` and replace the `path` in the file source of your `pipeline.yaml` file with that filepath. For more information, see [Configuring Data Prepper]({{site.url}}{{site.baseurl}}/data-prepper/getting-started/#2-configuring-data-prepper). | ||
|
|
||
| For example, before you run the `convert_entry_type` processor, if the `logs_json.log` file contains the following event record: | ||
|
|
||
|
|
||
| ```json | ||
| {"message": "value", "response_status":"200"} | ||
| ``` | ||
|
|
||
| The `convert_entry_type` processor converts the output received to the following output, where the type of `response_status` value changes from a string to an integer: | ||
|
|
||
| ```json | ||
| {"message":"value","response_status":200} | ||
| ``` |
60 changes: 48 additions & 12 deletions
60
_data-prepper/pipelines/configuration/processors/copy-values.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,24 +1,60 @@ | ||
| --- | ||
| layout: default | ||
| title: copy_values | ||
| title: Copy values processor | ||
| parent: Processors | ||
| grand_parent: Pipelines | ||
| nav_order: 45 | ||
| nav_order: 48 | ||
| --- | ||
|
|
||
| # copy_values | ||
|
|
||
| ## Overview | ||
| The `copy_values` processor copies values within an event and is a [mutate event]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/mutate-event/) processor. | ||
|
|
||
| The `copy_values` processor copies values within an event and is a [mutate event](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/mutate-event-processors#mutate-event-processors) processor. The following table describes the options you can use to configure the `copy_values` processor. | ||
| ## Configuration | ||
|
|
||
| Option | Required | Type | Description | ||
| :--- | :--- | :--- | :--- | ||
| entries | Yes | List | The list of entries to be copied. Valid values are `from_key`, `to_key`, and `overwrite_if_key_exists`. | ||
| from_key | N/A | N/A | The key of the entry to be copied. | ||
| to_key | N/A | N/A | The key of the new entry to be added. | ||
| overwrite_if_to_key_exists | No | Boolean | If true, the existing value is overwritten if the key already exists within the event. Default value is `false`. | ||
| You can configure the `copy_values` processor with the following options. | ||
|
|
||
| <!---## Configuration | ||
| | Option | Required | Description | | ||
| :--- | :--- | :--- | ||
| | `entries` | Yes | A list of entries to be copied in an event. | | ||
| | `from_key` | Yes | The key of the entry to be copied. | | ||
| | `to_key` | Yes | The key of the new entry to be added. | | ||
| | `overwrite_if_key_exists` | No | When set to `true`, the existing value is overwritten if `key` already exists in the event. The default value is `false`. | | ||
|
|
||
| Content will be added to this section.---> | ||
| ## Usage | ||
|
|
||
| To get started, create the following `pipeline.yaml` file: | ||
|
|
||
| ```yaml | ||
| pipeline: | ||
| source: | ||
| file: | ||
| path: "/full/path/to/logs_json.log" | ||
| record_type: "event" | ||
| format: "json" | ||
| processor: | ||
| - copy_values: | ||
| entries: | ||
| - from_key: "message" | ||
| to_key: "newMessage" | ||
| overwrite_if_to_key_exists: true | ||
| sink: | ||
| - stdout: | ||
| ``` | ||
| {% include copy.html %} | ||
| Next, create a log file named `logs_json.log` and replace the `path` in the file source of your `pipeline.yaml` file with that filepath. For more information, see [Configuring Data Prepper]({{site.url}}{{site.baseurl}}/data-prepper/getting-started/#2-configuring-data-prepper). | ||
|
|
||
| For example, before you run the `copy_values` processor, if the `logs_json.log` file contains the following event record: | ||
|
|
||
| ```json | ||
| {"message": "hello"} | ||
| ``` | ||
|
|
||
| When you run this processor, it parses the message into the following output: | ||
|
|
||
| ```json | ||
| {"message": "hello", "newMessage": "hello"} | ||
| ``` | ||
|
|
||
| > If `newMessage` already exists, its existing value is overwritten with `value`. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
5 changes: 2 additions & 3 deletions
5
_data-prepper/pipelines/configuration/processors/drop-events.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
5 changes: 2 additions & 3 deletions
5
_data-prepper/pipelines/configuration/processors/key-value.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.