Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Parse JSON processor doc. #3237

Merged
merged 21 commits into from
Apr 6, 2023
Merged
Changes from 19 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 65 additions & 13 deletions _data-prepper/pipelines/configuration/processors/parse-json.md
Original file line number Diff line number Diff line change
@@ -1,27 +1,79 @@
---
layout: default
title: Parse JSON
title: Parse_JSON
carolxob marked this conversation as resolved.
Show resolved Hide resolved
parent: Processors
grand_parent: Pipelines
nav_order: 45
---

# Parse JSON
# Parse_JSON
carolxob marked this conversation as resolved.
Show resolved Hide resolved

## Overview
The `Parse_JSON` processor parses JSON data for an event, including any nested fields. The processor extracts the JSON pointer data and adds the input event to the extracted fields.
carolxob marked this conversation as resolved.
Show resolved Hide resolved

The `parse_json` processor parses JSON data for an event, including any nested fields. The following table describes several optional parameters you can configure in the `parse_json` processor.

Option | Required | Type | Description
:--- | :--- | :--- | :---
source | No | String | The field in the `Event` that will be parsed. Default value is `message`.
destination | No | String | The destination field of the parsed JSON. Defaults to the root of the `Event`. Cannot be `""`, `/`, or any whitespace-only `String` because these are not valid `Event` fields.
pointer | No | String | A JSON Pointer to the field to be parsed. There is no `pointer` by default, meaning the entire `source` is parsed. The `pointer` can access JSON Array indices as well. If the JSON Pointer is invalid then the entire `source` data is parsed into the outgoing `Event`. If the pointed-to key already exists in the `Event` and the `destination` is the root, then the pointer uses the entire path of the key.
## Configuration

<!---## Configuration
You can configure the `Parse_JSON` processor with the following options.
carolxob marked this conversation as resolved.
Show resolved Hide resolved

Content will be added to this section.
| Option | Required | Type | Description |
| :--- | :--- | :--- | :--- |
| `source` | No | String | The field in the `Event` that will be parsed. Default value is `message`. |
carolxob marked this conversation as resolved.
Show resolved Hide resolved
| `destination` | No | String | The destination field of the parsed JSON. Defaults to the root of the `Event`. Cannot be `""`, `/`, or any whitespace-only `String` because these are not valid `Event` fields. |
carolxob marked this conversation as resolved.
Show resolved Hide resolved
| `pointer` | No | String | A JSON pointer to the field to be parsed. There is no `pointer` by default, meaning the entire `source` is parsed. The `pointer` can access JSON array indexes as well. If the JSON pointer is invalid then the entire `source` data is parsed into the outgoing `Event`. If the key that is pointed to already exists in the `Event` and the `destination` is the root, then the pointer uses the entire path of the key. |
carolxob marked this conversation as resolved.
Show resolved Hide resolved

## Metrics
## Usage

Content will be added to this section.--->
To get started, create the following `pipeline.yaml` file:

```yaml
parse-json-pipeline:
source:
stdin:
processor:
- parse_json:
sink:
- stdout:
```

### Basic example

To test the `Parse JSON` processor with the previous configuration, run the pipeline and paste the following line into your console, then enter `exit` on a new line:
carolxob marked this conversation as resolved.
Show resolved Hide resolved

```
{"outer_key": {"inner_key": "inner_value"}}
```
{% include copy.html %}

The processor parses the message into the following format:

```
{"message": {"outer_key": {"inner_key": "inner_value"}}", "outer_key":{"inner_key":"inner_value"}}}
```

### Example with JSON pointer

Use a JSON pointer to parse a selection of the JSON data by specifying the `pointer` option in the configuration. To get started, create the following `pipeline.yaml` file:

```yaml
parse-json-pipeline:
source:
stdin:
processor:
- parse_json:
pointer: "outer_key/inner_key"
sink:
- stdout:
```

To test the JSON processor with the pointer option, run the pipeline, paste the following line into your console, and then enter `exit` on a new line:

```
{"outer_key": {"inner_key": "inner_value"}}
```
{% include copy.html %}

The processor parses the message into the following format:

```
{"message": {"outer_key": {"inner_key": "inner_value"}}", "inner_key": "inner_value"}
```