Skip to content

Commit

Permalink
Update _data-prepper/pipelines/configuration/sinks/s3.md
Browse files Browse the repository at this point in the history 
Signed-off-by: Melissa Vagi <[email protected]>
  • Loading branch information
@vagimeli
vagimeli authored Jun 25, 2024
1 parent 8ef550e commit f2ab4ae
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion _data-prepper/pipelines/configuration/sinks/s3.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,7 @@ The `avro` codec writes an event as an [Apache Avro](https://avro.apache.org/) d

When you provide your own Avro schema, that schema defines the final structure of your data. Any extra values in any incoming events that are not mapped in the Avro schema will not be included in the final destination. Data Prepper does not allow the use of `include_keys` or `exclude_keys` with a custom schema so as to avoid confusion between a custom Avro schema and the `include_keys` or `exclude_keys` sink configurations.

Check failure on line 172 in _data-prepper/pipelines/configuration/sinks/s3.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: Avro. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: Avro. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/configuration/sinks/s3.md", "range": {"start": {"line": 172, "column": 27}}}, "severity": "ERROR"}

Check failure on line 172 in _data-prepper/pipelines/configuration/sinks/s3.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: Avro. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: Avro. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/configuration/sinks/s3.md", "range": {"start": {"line": 172, "column": 161}}}, "severity": "ERROR"}

Check failure on line 172 in _data-prepper/pipelines/configuration/sinks/s3.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: Avro. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: Avro. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/configuration/sinks/s3.md", "range": {"start": {"line": 172, "column": 355}}}, "severity": "ERROR"}

In cases where your data is uniform, you may be able to automatically generate a schema. Automatically generated schemas are based on the first event that the codec receives. The schema will only contain keys from this event, and all keys must be present in all events in order for the automatically generated schema to produce a working schema. Automatically generated schemas make all fields nullable. Use the `include_keys` and `exclude_keys` sink configurations to control which data is included in the automatically generated schema.
In cases where your data is uniform, you may be able to automatically generate a schema. Automatically generated schemas are based on the first event that the codec receives. The schema will only contain keys from this event, and all keys must be present in all events in order to automatically generate a working schema. Automatically generated schemas make all fields nullable. Use the `include_keys` and `exclude_keys` sink configurations to control which data is included in the automatically generated schema.

Check failure on line 174 in _data-prepper/pipelines/configuration/sinks/s3.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: nullable. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: nullable. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/configuration/sinks/s3.md", "range": {"start": {"line": 174, "column": 371}}}, "severity": "ERROR"}

Avro fields should use a null [union](https://avro.apache.org/docs/current/specification/#unions) because this will allow missing values. Otherwise, all required fields must be present for each event. Use non-nullable fields only when you are certain they exist.

Expand Down

0 comments on commit f2ab4ae

Please sign in to comment.