Data Prepper documentation updates: autogeneration campaign (opensear…

…ch-project#7707) Updates Data Prepper documentation with some missing fields. Adds support for autogeneration of processors by naming to match the processor and including the autogenerated comment. Signed-off-by: David Venable <[email protected]> Signed-off-by: David Venable <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> Co-authored-by: Melissa Vagi <[email protected]> Co-authored-by: Heather Halter <[email protected]> Signed-off-by: Sander van de Geijn <[email protected]>
sandervandegeijn · Jul 30, 2024 · b57f352 · b57f352
1 parent 4ff2760
commit b57f352
Show file tree

Hide file tree

Showing 6 changed files with 56 additions and 7 deletions.
diff --git a/_data-prepper/pipelines/configuration/processors/convert_entry_type.md b/_data-prepper/pipelines/configuration/processors/convert_entry_type.md
@@ -14,10 +14,22 @@ The `convert_entry_type` processor converts a value type associated with the spe
 
 You can configure the `convert_entry_type` processor with the following options.
 
+<!--
+This table is autogenerated. Do not edit it.
+- name: convert_entry_type
+- pluginType: processor
+- source: https://github.com/opensearch-project/data-prepper/blob/7d15115c281687aab50e5c471fd210cb1ef90fc5/data-prepper-plugins/mutate-event-processors/src/main/java/org/opensearch/dataprepper/plugins/processor/mutateevent/ConvertEntryTypeProcessorConfig.java
+-->
+
 | Option | Required | Description |
 | :--- | :--- | :--- |
-| `key`| Yes | Keys whose value needs to be converted to a different type. |
-| `type` | No | Target type for the key-value pair. Possible values are `integer`, `double`, `string`, and `Boolean`. Default value is `integer`. |
+| `key`| Yes | Key whose value needs to be converted to a different type. |
+| `keys`| Yes | Keys whose value needs to be converted to a different type. |
+| `type` | No | Target type for the key-value pair. Possible values are `integer`, `long`, `double`, `big_decimal`, `string`, and `boolean`. Default value is `integer`. |
+| `null_values` | No | String representation of what constitutes a `null` value. If the field value equals one of these strings, then the value is considered `null` and is converted to `null`. |
+| `scale` | No | Modifies the scale of the `big_decimal` when converting to a `big_decimal`. The default value is `0`. |
+| `tags_on_failure` | No | A list of tags to be added to the event metadata when the event fails to convert. |
+| `convert_when` | No | Specifies a condition using a [Data Prepper expression]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/) for performing the `convert_entry_type` operation. If specified, the `convert_entry_type` operation runs only when the expression evaluates to `true`. |
 
 ## Usage
 

diff --git a/...nes/configuration/processors/parse-ion.md → ...nes/configuration/processors/parse_ion.md b/...nes/configuration/processors/parse-ion.md → ...nes/configuration/processors/parse_ion.md
@@ -14,12 +14,22 @@ The `parse_ion` processor parses [Amazon Ion](https://amazon-ion.github.io/ion-d
 
 You can configure the `parse_ion` processor with the following options.
 
+<!--
+This table is autogenerated. Do not edit it.
+- name: parse_ion
+- pluginType: processor
+- source: https://github.com/opensearch-project/data-prepper/blob/253e59245fd9c39c959c1c8caaeff1b226a5a0ab/data-prepper-plugins/parse-json-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/parse/ion/ParseIonProcessorConfig.java
+-->
+
 | Option | Required | Type | Description |
 | :--- | :--- | :--- | :--- | 
 | `source` | No | String | The field in the `event` that is parsed. Default value is `message`. |
 | `destination` | No | String | The destination field of the parsed JSON. Defaults to the root of the `event`. Cannot be `""`, `/`, or any white-space-only `string` because these are not valid `event` fields. |
 | `pointer` | No | String | A JSON pointer to the field to be parsed. There is no `pointer` by default, meaning that the entire `source` is parsed. The `pointer` can access JSON array indexes as well. If the JSON pointer is invalid, then the entire `source` data is parsed into the outgoing `event`. If the key that is pointed to already exists in the `event` and the `destination` is the root, then the pointer uses the entire path of the key. |
-| `tags_on_failure` | No | String | A list of strings that specify the tags to be set in the event that the processors fails or an unknown exception occurs while parsing. 
+| `parse_when` | No | String | Specifies under which conditions the processor should perform parsing. Default is no condition. Accepts a Data Prepper expression string following the [Expression syntax]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/). |
+| `overwrite_if_destination_exists` | No | Boolean | Overwrites the destination if set to `true`. Set to `false` to prevent changing a destination value that exists. Defaults is `true`. |
+| `delete_source` | No | Boolean | If set to `true`, then the source field is deleted. Defaults is `false`. |
+| `tags_on_failure` | No | String | A list of strings specifying the tags to be set in the event that the processor fails or an unknown exception occurs during parsing.
 
 ## Usage
 

diff --git a/...es/configuration/processors/parse-json.md → ...es/configuration/processors/parse_json.md b/...es/configuration/processors/parse-json.md → ...es/configuration/processors/parse_json.md
@@ -15,11 +15,22 @@ The `parse_json` processor parses JSON data for an event, including any nested f
 
 You can configure the `parse_json` processor with the following options.
 
+<!--
+This table is autogenerated. Do not edit it.
+- name: parse_json
+- pluginType: processor
+- source: https://github.com/opensearch-project/data-prepper/blob/253e59245fd9c39c959c1c8caaeff1b226a5a0ab/data-prepper-plugins/parse-json-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/parse/json/ParseJsonProcessorConfig.java
+-->
+
 | Option | Required | Type | Description |
 | :--- | :--- | :--- | :--- | 
 | `source` | No | String | The field in the `event` that will be parsed. Default value is `message`. |
 | `destination` | No | String | The destination field of the parsed JSON. Defaults to the root of the `event`. Cannot be `""`, `/`, or any white-space-only `string` because these are not valid `event` fields. |
 | `pointer` | No | String | A JSON pointer to the field to be parsed. There is no `pointer` by default, meaning the entire `source` is parsed. The `pointer` can access JSON array indexes as well. If the JSON pointer is invalid then the entire `source` data is parsed into the outgoing `event`. If the key that is pointed to already exists in the `event` and the `destination` is the root, then the pointer uses the entire path of the key. |
+| `parse_when` | No | String | Specifies under which conditions the processor should perform parsing. Default is no condition. Accepts a Data Prepper expression string following the [Expression syntax]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/). |
+| `overwrite_if_destination_exists` | No | Boolean | Overwrites the destination if set to `true`. Set to `false` to prevent changing a destination value that exists. Defaults to `true`. |
+| `delete_source` | No | Boolean | If set to `true` then this will delete the source field. Defaults to `false`. |
+| `tags_on_failure` | No | String | A list of strings specifying the tags to be set in the event that the processor fails or an unknown exception occurs during parsing.
 
 ## Usage
 

diff --git a/...nes/configuration/processors/parse-xml.md → ...nes/configuration/processors/parse_xml.md b/...nes/configuration/processors/parse-xml.md → ...nes/configuration/processors/parse_xml.md
@@ -14,13 +14,22 @@ The `parse_xml` processor parses XML data for an event.
 
 You can configure the `parse_xml` processor with the following options.
 
+<!--
+This table is autogenerated. Do not edit it.
+- name: parse_xml
+- pluginType: processor
+- source: https://github.com/opensearch-project/data-prepper/blob/253e59245fd9c39c959c1c8caaeff1b226a5a0ab/data-prepper-plugins/parse-json-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/parse/xml/ParseXmlProcessorConfig.java
+-->
+
 | Option | Required | Type | Description |
 | :--- | :--- | :--- | :--- | 
 | `source` | No | String | Specifies which `event` field to parse. |
 | `destination` | No | String | The destination field of the parsed XML. Defaults to the root of the `event`. Cannot be `""`, `/`, or any white-space-only string because these are not valid `event` fields. |
 | `pointer` | No | String | A JSON pointer to the field to be parsed. The value is null by default, meaning that the entire `source` is parsed. The `pointer` can access JSON array indexes as well. If the JSON pointer is invalid, then the entire `source` data is parsed into the outgoing `event` object. If the key that is pointed to already exists in the `event` object and the `destination` is the root, then the pointer uses the entire path of the key. |
 | `parse_when` | No | String | Specifies under what conditions the processor should perform parsing. Default is no condition. Accepts a Data Prepper expression string following the [Data Prepper Expression Syntax]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/). |
-| `tags_on_failure` | No | String | A list of strings that specify the tags to be set if the processor fails or an unknown exception occurs while parsing. 
+| `overwrite_if_destination_exists` | No | Boolean | Overwrites the destination if set to `true`. Set to `false` to prevent changing a destination value that exists. Defaults to `true`. |
+| `delete_source` | No | Boolean | If set to `true` then this will delete the source field. Defaults to `false`. |
+| `tags_on_failure` | No | String | A list of strings specifying the tags to be set in the event that the processor fails or an unknown exception occurs during parsing.
 
 ## Usage
 

diff --git a/_data-prepper/pipelines/configuration/processors/write_json.md b/_data-prepper/pipelines/configuration/processors/write_json.md
@@ -11,8 +11,15 @@ nav_order: 56
 
 The `write_json` processor converts an object in an event into a JSON string. You can customize the processor to choose the source and target field names.
 
-Option | Description | Example 
-:--- | :--- | :--- 
+<!--
+This table is autogenerated. Do not edit it.
+- name: write_json
+- pluginType: processor
+- source: https://github.com/opensearch-project/data-prepper/blob/f0bd8d8e4773dc3d7318c19b8623c2213a099a86/data-prepper-plugins/write-json-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/write_json/WriteJsonProcessorConfig.java
+-->
+
+Option | Description | Example
+:--- | :--- | :---
 source | Mandatory field that specifies the name of the field in the event containing the message or object to be parsed. | If `source` is set to `"message"` and the input is `{"message": {"key1":"value1", "key2":{"key3":"value3"}}}`, then the `write_json` processor outputs the event as `"{\"key1\":\"value1\",\"key2\":{\"key3\":\"value3\"}}"`.
 target | An optional field that specifies the name of the field in which the resulting JSON string should be stored. If `target` is not specified, then the `source` field is used. | `key1`
 
diff --git a/_data-prepper/pipelines/configuration/sources/s3.md b/_data-prepper/pipelines/configuration/sources/s3.md
@@ -138,7 +138,7 @@ The `codec` determines how the `s3` source parses each Amazon S3 object. For inc
 
 ### `newline` codec
 
-The `newline` codec parses each single line as a single log event. This is ideal for most application logs because each event parses per single line. It can also be suitable for S3 objects that have individual JSON objects on each line, which matches well when used with the [parse_json]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/parse-json/) processor to parse each line.
+The `newline` codec parses each single line as a single log event. This is ideal for most application logs because each event parses per single line. It can also be suitable for S3 objects that have individual JSON objects on each line, which matches well when used with the [parse_json]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/parse_json/) processor to parse each line.
 
 Use the following options to configure the `newline` codec.