From 5d1bbc41a189c8b85a78f9e3f9a5899d8e0733d7 Mon Sep 17 00:00:00 2001 From: carolxob Date: Mon, 6 Mar 2023 13:25:46 -0500 Subject: [PATCH 01/21] Add Parse JSON processor doc. Signed-off-by: carolxob --- .../processors/parse-json-processor.md | 86 +++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 _data-prepper/pipelines/configuration/processors/parse-json-processor.md diff --git a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md new file mode 100644 index 0000000000..31d90a83fa --- /dev/null +++ b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md @@ -0,0 +1,86 @@ +--- +layout: default +title: Parse JSON +parent: Processors +grand_parent: Pipelines +nav_order: 45 +--- + + + +# Parse JSON processor + +The `Parse JSON` processor takes in an event and parses its JSON data, including any nested fields. + +## Basic usage + +To get started, create the following `pipelines.yaml` file: + +```yaml +parse-json-pipeline: + source: + stdin: + processor: + - parse_json: + sink: + - stdout: +``` +### Basic example + +You can test the JSON Processor with the above configuration by using the following example. + +Run the pipeline and paste the following line into your console, then enter `exit` on a new line: + +``` +{"outer_key": {"inner_key": "inner_value"}} +``` + +The processor parses the message into the following format: + +``` +{"message": {"outer_key": {"inner_key": "inner_value"}}", "outer_key":{"inner_key":"inner_value"}}} +``` + +### Example with JSON pointer + +You can parse a selection of the JSON data by specifying a JSON pointer and using the `pointer` option in the configuration. The following configuration file and example demonstrate a basic pointer use case: + +```yaml +parse-json-pipeline: + source: + stdin: + processor: + - parse_json: + pointer: "outer_key/inner_key" + sink: + - stdout: +``` + +Run the pipeline and paste the following line into your console, then enter `exit` on a new line. + +``` +{"outer_key": {"inner_key": "inner_value"}} +``` + +The processor parses the message into the following format: + +``` +{"message": {"outer_key": {"inner_key": "inner_value"}}", "inner_key": "inner_value"} +``` + +## Configuration + +You can configure the `Parse JSON` processor with the following options: + +* `source` (optional): The field in the `Event` that is parsed. + * Default: `message` + +* `destination` (optional): The destination field of the parsed JSON. Defaults to the root of the `Event`. + * Defaults to writing to the root of the `Event`. The processor writes to root when the `destination` has a value of `null`. + * Cannot be `""`, `/`, or any whitespace-only `String` because these options are not valid `Event` fields. + +* `pointer` (optional): A JSON pointer to the field that must be parsed. + * There is no `pointer` by default, and the entire `source` is parsed. + * The `pointer` can access JSON array indexes. + * If the JSON pointer is invalid, then the entire `source` data is parsed into the outgoing `Event`. + * If the key that is pointed to exists in the `Event`, and the `destination` is the root, then the entire path of the key will be used. \ No newline at end of file From efcdfa963bbeddc5b52de2bc59d436793137293d Mon Sep 17 00:00:00 2001 From: carolxob Date: Mon, 13 Mar 2023 13:07:17 -0400 Subject: [PATCH 02/21] Minor edit. Signed-off-by: carolxob --- .../pipelines/configuration/processors/parse-json-processor.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md index 31d90a83fa..41e51430d7 100644 --- a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md +++ b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md @@ -6,8 +6,6 @@ grand_parent: Pipelines nav_order: 45 --- - - # Parse JSON processor The `Parse JSON` processor takes in an event and parses its JSON data, including any nested fields. From b05f8607acd7da8a347cb561a478fd554a5e1b1f Mon Sep 17 00:00:00 2001 From: carolxob Date: Mon, 13 Mar 2023 13:29:54 -0400 Subject: [PATCH 03/21] Formatted some information as a table instead of a list. Signed-off-by: carolxob --- .../processors/parse-json-processor.md | 18 ++++++------------ 1 file changed, 6 insertions(+), 12 deletions(-) diff --git a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md index 41e51430d7..cbafb19593 100644 --- a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md +++ b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md @@ -68,17 +68,11 @@ The processor parses the message into the following format: ## Configuration -You can configure the `Parse JSON` processor with the following options: +You can configure the `Parse JSON` processor with the following options. -* `source` (optional): The field in the `Event` that is parsed. - * Default: `message` +Option | Optional | Default value | Description +:--- | :--- | :--- | :--- -* `destination` (optional): The destination field of the parsed JSON. Defaults to the root of the `Event`. - * Defaults to writing to the root of the `Event`. The processor writes to root when the `destination` has a value of `null`. - * Cannot be `""`, `/`, or any whitespace-only `String` because these options are not valid `Event` fields. - -* `pointer` (optional): A JSON pointer to the field that must be parsed. - * There is no `pointer` by default, and the entire `source` is parsed. - * The `pointer` can access JSON array indexes. - * If the JSON pointer is invalid, then the entire `source` data is parsed into the outgoing `Event`. - * If the key that is pointed to exists in the `Event`, and the `destination` is the root, then the entire path of the key will be used. \ No newline at end of file + `source` | Yes | `message` | The field in the `Event` that is parsed. + `destination` | Yes | | * The destination field of the parsed JSON. The processor writes to root when the `destination` has a value of `null`. Cannot be `""`, `/`, or any whitespace-only `String` because these options are not valid `Event` fields. +`pointer` | Yes | A JSON pointer that points to the field that must be parsed. No `pointer` exists by default and the entire `source` is parsed. The `pointer` can access JSON array indexes. If the JSON `pointer` is invalid, all `source` data is parsed into the outgoing `event`. If the key that is pointed to exists in `Event`, and the `destination` is the root, then the full path of the key is used. | \ No newline at end of file From aaac99847c8f26200b61bed788b477f7fbb787ee Mon Sep 17 00:00:00 2001 From: carolxob Date: Mon, 13 Mar 2023 13:33:15 -0400 Subject: [PATCH 04/21] Minor edits. Signed-off-by: carolxob --- .../processors/parse-json-processor.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md index cbafb19593..95d0acf963 100644 --- a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md +++ b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md @@ -1,6 +1,6 @@ --- layout: default -title: Parse JSON +title: Parse JSON processor parent: Processors grand_parent: Pipelines nav_order: 45 @@ -8,9 +8,9 @@ nav_order: 45 # Parse JSON processor -The `Parse JSON` processor takes in an event and parses its JSON data, including any nested fields. +The `parse_json` processor parses JSON data for an event, including any nested fields. -## Basic usage +## Usage To get started, create the following `pipelines.yaml` file: @@ -70,9 +70,8 @@ The processor parses the message into the following format: You can configure the `Parse JSON` processor with the following options. -Option | Optional | Default value | Description +Option | Required | Type | Description :--- | :--- | :--- | :--- - - `source` | Yes | `message` | The field in the `Event` that is parsed. - `destination` | Yes | | * The destination field of the parsed JSON. The processor writes to root when the `destination` has a value of `null`. Cannot be `""`, `/`, or any whitespace-only `String` because these options are not valid `Event` fields. -`pointer` | Yes | A JSON pointer that points to the field that must be parsed. No `pointer` exists by default and the entire `source` is parsed. The `pointer` can access JSON array indexes. If the JSON `pointer` is invalid, all `source` data is parsed into the outgoing `event`. If the key that is pointed to exists in `Event`, and the `destination` is the root, then the full path of the key is used. | \ No newline at end of file +`source` | No | String | The field in the `Event` that will be parsed. Default value is `message`. +`destination` | No | String | The destination field of the parsed JSON. Defaults to the root of the `Event`. Cannot be `""`, `/`, or any whitespace-only `String` because these are not valid `Event` fields. +`pointer` | No | String | A JSON Pointer to the field to be parsed. There is no `pointer` by default, meaning the entire `source` is parsed. The `pointer` can access JSON Array indices as well. If the JSON Pointer is invalid then the entire `source` data is parsed into the outgoing `Event`. If the pointed-to key already exists in the `Event` and the `destination` is the root, then the pointer uses the entire path of the key. \ No newline at end of file From 49316ff06b082532f834bad8b43e5237a3dd5656 Mon Sep 17 00:00:00 2001 From: carolxob Date: Mon, 13 Mar 2023 13:34:34 -0400 Subject: [PATCH 05/21] Removed smaller Parse JSON file. Signed-off-by: carolxob --- .../configuration/processors/parse-json.md | 27 ------------------- 1 file changed, 27 deletions(-) delete mode 100644 _data-prepper/pipelines/configuration/processors/parse-json.md diff --git a/_data-prepper/pipelines/configuration/processors/parse-json.md b/_data-prepper/pipelines/configuration/processors/parse-json.md deleted file mode 100644 index e32b4e5110..0000000000 --- a/_data-prepper/pipelines/configuration/processors/parse-json.md +++ /dev/null @@ -1,27 +0,0 @@ ---- -layout: default -title: Parse JSON -parent: Processors -grand_parent: Pipelines -nav_order: 45 ---- - -# Parse JSON - -## Overview - -The `parse_json` processor parses JSON data for an event, including any nested fields. The following table describes several optional parameters you can configure in the `parse_json` processor. - -Option | Required | Type | Description -:--- | :--- | :--- | :--- -source | No | String | The field in the `Event` that will be parsed. Default value is `message`. -destination | No | String | The destination field of the parsed JSON. Defaults to the root of the `Event`. Cannot be `""`, `/`, or any whitespace-only `String` because these are not valid `Event` fields. -pointer | No | String | A JSON Pointer to the field to be parsed. There is no `pointer` by default, meaning the entire `source` is parsed. The `pointer` can access JSON Array indices as well. If the JSON Pointer is invalid then the entire `source` data is parsed into the outgoing `Event`. If the pointed-to key already exists in the `Event` and the `destination` is the root, then the pointer uses the entire path of the key. - - \ No newline at end of file From 778e8f9fba2b1791b840d3b32238cd3a6034e4c9 Mon Sep 17 00:00:00 2001 From: carolxob Date: Tue, 14 Mar 2023 20:52:29 -0400 Subject: [PATCH 06/21] Minor updates. Signed-off-by: carolxob --- .../configuration/processors/parse-json-processor.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md index 95d0acf963..9ed214127d 100644 --- a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md +++ b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md @@ -12,7 +12,7 @@ The `parse_json` processor parses JSON data for an event, including any nested f ## Usage -To get started, create the following `pipelines.yaml` file: +To get started, create the following `pipeline.yaml` file: ```yaml parse-json-pipeline: @@ -23,9 +23,12 @@ parse-json-pipeline: sink: - stdout: ``` + + + ### Basic example -You can test the JSON Processor with the above configuration by using the following example. +You can test the JSON Processor with the previous configuration by using the following example. Run the pipeline and paste the following line into your console, then enter `exit` on a new line: From bfe52eb1416127f614cee399e53c41424abf3e54 Mon Sep 17 00:00:00 2001 From: carolxob Date: Tue, 21 Mar 2023 17:18:47 -0600 Subject: [PATCH 07/21] Minor edits to table. Signed-off-by: carolxob --- .../configuration/processors/parse-json-processor.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md index 9ed214127d..d0623f634c 100644 --- a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md +++ b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md @@ -35,6 +35,7 @@ Run the pipeline and paste the following line into your console, then enter `exi ``` {"outer_key": {"inner_key": "inner_value"}} ``` +{% include copy.html %} The processor parses the message into the following format: @@ -62,6 +63,7 @@ Run the pipeline and paste the following line into your console, then enter `exi ``` {"outer_key": {"inner_key": "inner_value"}} ``` +{% include copy.html %} The processor parses the message into the following format: @@ -73,8 +75,8 @@ The processor parses the message into the following format: You can configure the `Parse JSON` processor with the following options. -Option | Required | Type | Description -:--- | :--- | :--- | :--- -`source` | No | String | The field in the `Event` that will be parsed. Default value is `message`. -`destination` | No | String | The destination field of the parsed JSON. Defaults to the root of the `Event`. Cannot be `""`, `/`, or any whitespace-only `String` because these are not valid `Event` fields. -`pointer` | No | String | A JSON Pointer to the field to be parsed. There is no `pointer` by default, meaning the entire `source` is parsed. The `pointer` can access JSON Array indices as well. If the JSON Pointer is invalid then the entire `source` data is parsed into the outgoing `Event`. If the pointed-to key already exists in the `Event` and the `destination` is the root, then the pointer uses the entire path of the key. \ No newline at end of file +| Option | Required | Type | Description | +| :--- | :--- | :--- | :--- | +| `source` | No | String | The field in the `Event` that will be parsed. Default value is `message`. | +| `destination` | No | String | The destination field of the parsed JSON. Defaults to the root of the `Event`. Cannot be `""`, `/`, or any whitespace-only `String` because these are not valid `Event` fields. | +| `pointer` | No | String | A JSON Pointer to the field to be parsed. There is no `pointer` by default, meaning the entire `source` is parsed. The `pointer` can access JSON Array indices as well. If the JSON Pointer is invalid then the entire `source` data is parsed into the outgoing `Event`. If the pointed-to key already exists in the `Event` and the `destination` is the root, then the pointer uses the entire path of the key. | \ No newline at end of file From f6df00ff7cd15b37d164cd378adb93cca020c3f7 Mon Sep 17 00:00:00 2001 From: carolxob Date: Wed, 22 Mar 2023 11:40:17 -0600 Subject: [PATCH 08/21] Minor updates. Signed-off-by: carolxob --- .../pipelines/configuration/processors/parse-json-processor.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md index d0623f634c..9594ce95d0 100644 --- a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md +++ b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md @@ -45,7 +45,7 @@ The processor parses the message into the following format: ### Example with JSON pointer -You can parse a selection of the JSON data by specifying a JSON pointer and using the `pointer` option in the configuration. The following configuration file and example demonstrate a basic pointer use case: +You can parse a selection of the JSON data by specifying a JSON pointer and using the `pointer` option in the configuration. See the following YAML configuration file and example of pointer use: ```yaml parse-json-pipeline: From 7720a9986fb49c96656e15a45a40b0ce1704cfc5 Mon Sep 17 00:00:00 2001 From: carolxob Date: Tue, 28 Mar 2023 14:27:35 -0600 Subject: [PATCH 09/21] Minor update based on tech feedback. Signed-off-by: carolxob --- .../pipelines/configuration/processors/parse-json-processor.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md index 9594ce95d0..232f6f34bd 100644 --- a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md +++ b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md @@ -8,7 +8,7 @@ nav_order: 45 # Parse JSON processor -The `parse_json` processor parses JSON data for an event, including any nested fields. +The `parse_json` processor parses JSON data for an event, including any nested fields. The processor extracts the JSON pointer data and adds the input event to the extracted fields. ## Usage From 13b1b42f0af51763a63d1eb78282a74c9b4b523c Mon Sep 17 00:00:00 2001 From: carolxob Date: Tue, 28 Mar 2023 14:30:15 -0600 Subject: [PATCH 10/21] Minor updates. Signed-off-by: carolxob --- .../pipelines/configuration/processors/parse-json-processor.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md index 232f6f34bd..27d84ffc9b 100644 --- a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md +++ b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md @@ -24,8 +24,6 @@ parse-json-pipeline: - stdout: ``` - - ### Basic example You can test the JSON Processor with the previous configuration by using the following example. From 24f2561e52e04192328f1d31dded49940fd2cf7d Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 31 Mar 2023 15:32:31 -0600 Subject: [PATCH 11/21] Modified 'indices' to 'indexes' Signed-off-by: carolxob --- .../pipelines/configuration/processors/parse-json-processor.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md index 27d84ffc9b..cec3913761 100644 --- a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md +++ b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md @@ -77,4 +77,4 @@ You can configure the `Parse JSON` processor with the following options. | :--- | :--- | :--- | :--- | | `source` | No | String | The field in the `Event` that will be parsed. Default value is `message`. | | `destination` | No | String | The destination field of the parsed JSON. Defaults to the root of the `Event`. Cannot be `""`, `/`, or any whitespace-only `String` because these are not valid `Event` fields. | -| `pointer` | No | String | A JSON Pointer to the field to be parsed. There is no `pointer` by default, meaning the entire `source` is parsed. The `pointer` can access JSON Array indices as well. If the JSON Pointer is invalid then the entire `source` data is parsed into the outgoing `Event`. If the pointed-to key already exists in the `Event` and the `destination` is the root, then the pointer uses the entire path of the key. | \ No newline at end of file +| `pointer` | No | String | A JSON Pointer to the field to be parsed. There is no `pointer` by default, meaning the entire `source` is parsed. The `pointer` can access JSON Array indexes as well. If the JSON Pointer is invalid then the entire `source` data is parsed into the outgoing `Event`. If the pointed-to key already exists in the `Event` and the `destination` is the root, then the pointer uses the entire path of the key. | \ No newline at end of file From 64b8b5f99685debda90e6706a29a89bf4a105ec2 Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 31 Mar 2023 15:32:58 -0600 Subject: [PATCH 12/21] Minor edits. Signed-off-by: carolxob --- .../pipelines/configuration/processors/parse-json-processor.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md index cec3913761..7f6eeb1907 100644 --- a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md +++ b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md @@ -77,4 +77,4 @@ You can configure the `Parse JSON` processor with the following options. | :--- | :--- | :--- | :--- | | `source` | No | String | The field in the `Event` that will be parsed. Default value is `message`. | | `destination` | No | String | The destination field of the parsed JSON. Defaults to the root of the `Event`. Cannot be `""`, `/`, or any whitespace-only `String` because these are not valid `Event` fields. | -| `pointer` | No | String | A JSON Pointer to the field to be parsed. There is no `pointer` by default, meaning the entire `source` is parsed. The `pointer` can access JSON Array indexes as well. If the JSON Pointer is invalid then the entire `source` data is parsed into the outgoing `Event`. If the pointed-to key already exists in the `Event` and the `destination` is the root, then the pointer uses the entire path of the key. | \ No newline at end of file +| `pointer` | No | String | A JSON pointer to the field to be parsed. There is no `pointer` by default, meaning the entire `source` is parsed. The `pointer` can access JSON Array indexes as well. If the JSON Pointer is invalid then the entire `source` data is parsed into the outgoing `Event`. If the pointed-to key already exists in the `Event` and the `destination` is the root, then the pointer uses the entire path of the key. | \ No newline at end of file From 052c6fcd7a939450f9631b9592f5e102adff3461 Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 31 Mar 2023 15:33:30 -0600 Subject: [PATCH 13/21] Minor edits. Signed-off-by: carolxob --- .../pipelines/configuration/processors/parse-json-processor.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md index 7f6eeb1907..f1799f070d 100644 --- a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md +++ b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md @@ -77,4 +77,4 @@ You can configure the `Parse JSON` processor with the following options. | :--- | :--- | :--- | :--- | | `source` | No | String | The field in the `Event` that will be parsed. Default value is `message`. | | `destination` | No | String | The destination field of the parsed JSON. Defaults to the root of the `Event`. Cannot be `""`, `/`, or any whitespace-only `String` because these are not valid `Event` fields. | -| `pointer` | No | String | A JSON pointer to the field to be parsed. There is no `pointer` by default, meaning the entire `source` is parsed. The `pointer` can access JSON Array indexes as well. If the JSON Pointer is invalid then the entire `source` data is parsed into the outgoing `Event`. If the pointed-to key already exists in the `Event` and the `destination` is the root, then the pointer uses the entire path of the key. | \ No newline at end of file +| `pointer` | No | String | A JSON pointer to the field to be parsed. There is no `pointer` by default, meaning the entire `source` is parsed. The `pointer` can access JSON Array indexes as well. If the JSON pointer is invalid then the entire `source` data is parsed into the outgoing `Event`. If the pointed-to key already exists in the `Event` and the `destination` is the root, then the pointer uses the entire path of the key. | \ No newline at end of file From 5ac580a65c911c78d80f54c844a0ec4e5dc13354 Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 31 Mar 2023 15:34:09 -0600 Subject: [PATCH 14/21] Minor edits. Signed-off-by: carolxob --- .../pipelines/configuration/processors/parse-json-processor.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md index f1799f070d..710eb8d27f 100644 --- a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md +++ b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md @@ -77,4 +77,4 @@ You can configure the `Parse JSON` processor with the following options. | :--- | :--- | :--- | :--- | | `source` | No | String | The field in the `Event` that will be parsed. Default value is `message`. | | `destination` | No | String | The destination field of the parsed JSON. Defaults to the root of the `Event`. Cannot be `""`, `/`, or any whitespace-only `String` because these are not valid `Event` fields. | -| `pointer` | No | String | A JSON pointer to the field to be parsed. There is no `pointer` by default, meaning the entire `source` is parsed. The `pointer` can access JSON Array indexes as well. If the JSON pointer is invalid then the entire `source` data is parsed into the outgoing `Event`. If the pointed-to key already exists in the `Event` and the `destination` is the root, then the pointer uses the entire path of the key. | \ No newline at end of file +| `pointer` | No | String | A JSON pointer to the field to be parsed. There is no `pointer` by default, meaning the entire `source` is parsed. The `pointer` can access JSON Array indexes as well. If the JSON pointer is invalid then the entire `source` data is parsed into the outgoing `Event`. If the key that is pointed to already exists in the `Event` and the `destination` is the root, then the pointer uses the entire path of the key. | \ No newline at end of file From ee925705e447304f5d211ca88e20d7cc2614a88d Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 31 Mar 2023 15:34:50 -0600 Subject: [PATCH 15/21] Minor edits. Signed-off-by: carolxob --- .../pipelines/configuration/processors/parse-json-processor.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md index 710eb8d27f..5ef25a098b 100644 --- a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md +++ b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md @@ -26,7 +26,7 @@ parse-json-pipeline: ### Basic example -You can test the JSON Processor with the previous configuration by using the following example. +You can test the JSON processor with the previous configuration by using the following example. Run the pipeline and paste the following line into your console, then enter `exit` on a new line: From 4b414c370002a56367a81a20ceafc8ae207d4a76 Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 31 Mar 2023 15:38:31 -0600 Subject: [PATCH 16/21] Minor edits. Signed-off-by: carolxob --- .../configuration/processors/parse-json-processor.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md index 5ef25a098b..9222f4d185 100644 --- a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md +++ b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md @@ -1,6 +1,6 @@ --- layout: default -title: Parse JSON processor +title: Parse_JSON processor parent: Processors grand_parent: Pipelines nav_order: 45 @@ -8,7 +8,7 @@ nav_order: 45 # Parse JSON processor -The `parse_json` processor parses JSON data for an event, including any nested fields. The processor extracts the JSON pointer data and adds the input event to the extracted fields. +The `Parse_json` processor parses JSON data for an event, including any nested fields. The processor extracts the JSON pointer data and adds the input event to the extracted fields. ## Usage @@ -56,7 +56,7 @@ parse-json-pipeline: - stdout: ``` -Run the pipeline and paste the following line into your console, then enter `exit` on a new line. +Run the pipeline and paste the following line into your console, then enter `exit` on a new line: ``` {"outer_key": {"inner_key": "inner_value"}} @@ -71,7 +71,7 @@ The processor parses the message into the following format: ## Configuration -You can configure the `Parse JSON` processor with the following options. +You can configure the `Parse_JSON` processor with the following options. | Option | Required | Type | Description | | :--- | :--- | :--- | :--- | From 1cf75cdd5a25200cd2eae708c8b36d1c78a0ea4c Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 31 Mar 2023 15:43:28 -0600 Subject: [PATCH 17/21] Modified title. Signed-off-by: carolxob --- .../processors/parse-json-processor.md | 27 ++++++++++--------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md index 9222f4d185..be92c4fb45 100644 --- a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md +++ b/_data-prepper/pipelines/configuration/processors/parse-json-processor.md @@ -1,15 +1,26 @@ --- layout: default -title: Parse_JSON processor +title: Parse_JSON parent: Processors grand_parent: Pipelines nav_order: 45 --- -# Parse JSON processor +# Parse JSON The `Parse_json` processor parses JSON data for an event, including any nested fields. The processor extracts the JSON pointer data and adds the input event to the extracted fields. + +## Configuration + +You can configure the `Parse_JSON` processor with the following options. + +| Option | Required | Type | Description | +| :--- | :--- | :--- | :--- | +| `source` | No | String | The field in the `Event` that will be parsed. Default value is `message`. | +| `destination` | No | String | The destination field of the parsed JSON. Defaults to the root of the `Event`. Cannot be `""`, `/`, or any whitespace-only `String` because these are not valid `Event` fields. | +| `pointer` | No | String | A JSON pointer to the field to be parsed. There is no `pointer` by default, meaning the entire `source` is parsed. The `pointer` can access JSON Array indexes as well. If the JSON pointer is invalid then the entire `source` data is parsed into the outgoing `Event`. If the key that is pointed to already exists in the `Event` and the `destination` is the root, then the pointer uses the entire path of the key. | + ## Usage To get started, create the following `pipeline.yaml` file: @@ -67,14 +78,4 @@ The processor parses the message into the following format: ``` {"message": {"outer_key": {"inner_key": "inner_value"}}", "inner_key": "inner_value"} -``` - -## Configuration - -You can configure the `Parse_JSON` processor with the following options. - -| Option | Required | Type | Description | -| :--- | :--- | :--- | :--- | -| `source` | No | String | The field in the `Event` that will be parsed. Default value is `message`. | -| `destination` | No | String | The destination field of the parsed JSON. Defaults to the root of the `Event`. Cannot be `""`, `/`, or any whitespace-only `String` because these are not valid `Event` fields. | -| `pointer` | No | String | A JSON pointer to the field to be parsed. There is no `pointer` by default, meaning the entire `source` is parsed. The `pointer` can access JSON Array indexes as well. If the JSON pointer is invalid then the entire `source` data is parsed into the outgoing `Event`. If the key that is pointed to already exists in the `Event` and the `destination` is the root, then the pointer uses the entire path of the key. | \ No newline at end of file +``` \ No newline at end of file From 77d510c5c2973fd5bcca3683c67202ba1fe34355 Mon Sep 17 00:00:00 2001 From: carolxob Date: Tue, 4 Apr 2023 10:28:27 -0600 Subject: [PATCH 18/21] Content edits from doc and editorial reviews. Signed-off-by: carolxob --- .../{parse-json-processor.md => parse-json.md} | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) rename _data-prepper/pipelines/configuration/processors/{parse-json-processor.md => parse-json.md} (77%) diff --git a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md b/_data-prepper/pipelines/configuration/processors/parse-json.md similarity index 77% rename from _data-prepper/pipelines/configuration/processors/parse-json-processor.md rename to _data-prepper/pipelines/configuration/processors/parse-json.md index be92c4fb45..885f5ea4b2 100644 --- a/_data-prepper/pipelines/configuration/processors/parse-json-processor.md +++ b/_data-prepper/pipelines/configuration/processors/parse-json.md @@ -6,9 +6,9 @@ grand_parent: Pipelines nav_order: 45 --- -# Parse JSON +# Parse_JSON -The `Parse_json` processor parses JSON data for an event, including any nested fields. The processor extracts the JSON pointer data and adds the input event to the extracted fields. +The `Parse_JSON` processor parses JSON data for an event, including any nested fields. The processor extracts the JSON pointer data and adds the input event to the extracted fields. ## Configuration @@ -37,9 +37,7 @@ parse-json-pipeline: ### Basic example -You can test the JSON processor with the previous configuration by using the following example. - -Run the pipeline and paste the following line into your console, then enter `exit` on a new line: +To test the `Parse JSON` processor with the previous configuration, run the pipeline and paste the following line into your console, then enter `exit` on a new line: ``` {"outer_key": {"inner_key": "inner_value"}} @@ -54,7 +52,7 @@ The processor parses the message into the following format: ### Example with JSON pointer -You can parse a selection of the JSON data by specifying a JSON pointer and using the `pointer` option in the configuration. See the following YAML configuration file and example of pointer use: +Use a JSON pointer to parse a selection of the JSON data by specifying the `pointer` option in the configuration. To get started, create the following `pipeline.yaml` file: ```yaml parse-json-pipeline: @@ -67,7 +65,7 @@ parse-json-pipeline: - stdout: ``` -Run the pipeline and paste the following line into your console, then enter `exit` on a new line: +To test the JSON processor with the pointer option, run the pipeline, paste the following line into your console, and then enter `exit` on a new line: ``` {"outer_key": {"inner_key": "inner_value"}} From 4ad5e29d1b48ac247fb42f37a6b43104f9cd3d6c Mon Sep 17 00:00:00 2001 From: carolxob Date: Tue, 4 Apr 2023 10:30:05 -0600 Subject: [PATCH 19/21] Minor edit. Signed-off-by: carolxob --- _data-prepper/pipelines/configuration/processors/parse-json.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/processors/parse-json.md b/_data-prepper/pipelines/configuration/processors/parse-json.md index 885f5ea4b2..2bb118d4c0 100644 --- a/_data-prepper/pipelines/configuration/processors/parse-json.md +++ b/_data-prepper/pipelines/configuration/processors/parse-json.md @@ -19,7 +19,7 @@ You can configure the `Parse_JSON` processor with the following options. | :--- | :--- | :--- | :--- | | `source` | No | String | The field in the `Event` that will be parsed. Default value is `message`. | | `destination` | No | String | The destination field of the parsed JSON. Defaults to the root of the `Event`. Cannot be `""`, `/`, or any whitespace-only `String` because these are not valid `Event` fields. | -| `pointer` | No | String | A JSON pointer to the field to be parsed. There is no `pointer` by default, meaning the entire `source` is parsed. The `pointer` can access JSON Array indexes as well. If the JSON pointer is invalid then the entire `source` data is parsed into the outgoing `Event`. If the key that is pointed to already exists in the `Event` and the `destination` is the root, then the pointer uses the entire path of the key. | +| `pointer` | No | String | A JSON pointer to the field to be parsed. There is no `pointer` by default, meaning the entire `source` is parsed. The `pointer` can access JSON array indexes as well. If the JSON pointer is invalid then the entire `source` data is parsed into the outgoing `Event`. If the key that is pointed to already exists in the `Event` and the `destination` is the root, then the pointer uses the entire path of the key. | ## Usage From 8865ea2ccdfdb4a214373e3ecbef67f9c06d906f Mon Sep 17 00:00:00 2001 From: carolxob Date: Tue, 4 Apr 2023 14:32:10 -0600 Subject: [PATCH 20/21] Updates. Signed-off-by: carolxob --- .../pipelines/configuration/processors/parse-json.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_data-prepper/pipelines/configuration/processors/parse-json.md b/_data-prepper/pipelines/configuration/processors/parse-json.md index 2bb118d4c0..2116f336e9 100644 --- a/_data-prepper/pipelines/configuration/processors/parse-json.md +++ b/_data-prepper/pipelines/configuration/processors/parse-json.md @@ -50,9 +50,9 @@ The processor parses the message into the following format: {"message": {"outer_key": {"inner_key": "inner_value"}}", "outer_key":{"inner_key":"inner_value"}}} ``` -### Example with JSON pointer +### Example with a JSON pointer -Use a JSON pointer to parse a selection of the JSON data by specifying the `pointer` option in the configuration. To get started, create the following `pipeline.yaml` file: +You can use a JSON pointer to parse a selection of the JSON data by specifying the `pointer` option in the configuration. To get started, create the following `pipeline.yaml` file: ```yaml parse-json-pipeline: From df6eeeff95a7de21b776c0515511362b6b0ce77e Mon Sep 17 00:00:00 2001 From: carolxob Date: Tue, 4 Apr 2023 14:45:07 -0600 Subject: [PATCH 21/21] Final review comments incorporated. Signed-off-by: carolxob --- .../configuration/processors/parse-json.md | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/_data-prepper/pipelines/configuration/processors/parse-json.md b/_data-prepper/pipelines/configuration/processors/parse-json.md index 2116f336e9..306327a8d7 100644 --- a/_data-prepper/pipelines/configuration/processors/parse-json.md +++ b/_data-prepper/pipelines/configuration/processors/parse-json.md @@ -1,25 +1,25 @@ --- layout: default -title: Parse_JSON +title: parse_json parent: Processors grand_parent: Pipelines nav_order: 45 --- -# Parse_JSON +# parse_json -The `Parse_JSON` processor parses JSON data for an event, including any nested fields. The processor extracts the JSON pointer data and adds the input event to the extracted fields. +The `parse_json` processor parses JSON data for an event, including any nested fields. The processor extracts the JSON pointer data and adds the input event to the extracted fields. ## Configuration -You can configure the `Parse_JSON` processor with the following options. +You can configure the `parse_json` processor with the following options. | Option | Required | Type | Description | | :--- | :--- | :--- | :--- | -| `source` | No | String | The field in the `Event` that will be parsed. Default value is `message`. | -| `destination` | No | String | The destination field of the parsed JSON. Defaults to the root of the `Event`. Cannot be `""`, `/`, or any whitespace-only `String` because these are not valid `Event` fields. | -| `pointer` | No | String | A JSON pointer to the field to be parsed. There is no `pointer` by default, meaning the entire `source` is parsed. The `pointer` can access JSON array indexes as well. If the JSON pointer is invalid then the entire `source` data is parsed into the outgoing `Event`. If the key that is pointed to already exists in the `Event` and the `destination` is the root, then the pointer uses the entire path of the key. | +| `source` | No | String | The field in the `event` that will be parsed. Default value is `message`. | +| `destination` | No | String | The destination field of the parsed JSON. Defaults to the root of the `event`. Cannot be `""`, `/`, or any whitespace-only `string` because these are not valid `event` fields. | +| `pointer` | No | String | A JSON pointer to the field to be parsed. There is no `pointer` by default, meaning the entire `source` is parsed. The `pointer` can access JSON array indexes as well. If the JSON pointer is invalid then the entire `source` data is parsed into the outgoing `event`. If the key that is pointed to already exists in the `event` and the `destination` is the root, then the pointer uses the entire path of the key. | ## Usage @@ -37,14 +37,14 @@ parse-json-pipeline: ### Basic example -To test the `Parse JSON` processor with the previous configuration, run the pipeline and paste the following line into your console, then enter `exit` on a new line: +To test the `parse_json` processor with the previous configuration, run the pipeline and paste the following line into your console, then enter `exit` on a new line: ``` {"outer_key": {"inner_key": "inner_value"}} ``` {% include copy.html %} -The processor parses the message into the following format: +The `parse_json` processor parses the message into the following format: ``` {"message": {"outer_key": {"inner_key": "inner_value"}}", "outer_key":{"inner_key":"inner_value"}}} @@ -65,7 +65,7 @@ parse-json-pipeline: - stdout: ``` -To test the JSON processor with the pointer option, run the pipeline, paste the following line into your console, and then enter `exit` on a new line: +To test the `parse_json` processor with the pointer option, run the pipeline, paste the following line into your console, and then enter `exit` on a new line: ``` {"outer_key": {"inner_key": "inner_value"}}