From f2ab4ae6639c5ae21657d62f2bea9f998c2ac88e Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Tue, 25 Jun 2024 15:41:20 -0600 Subject: [PATCH] Update _data-prepper/pipelines/configuration/sinks/s3.md Signed-off-by: Melissa Vagi --- _data-prepper/pipelines/configuration/sinks/s3.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/sinks/s3.md b/_data-prepper/pipelines/configuration/sinks/s3.md index 42d26134fd..d1413f6ffc 100644 --- a/_data-prepper/pipelines/configuration/sinks/s3.md +++ b/_data-prepper/pipelines/configuration/sinks/s3.md @@ -171,7 +171,7 @@ The `avro` codec writes an event as an [Apache Avro](https://avro.apache.org/) d When you provide your own Avro schema, that schema defines the final structure of your data. Any extra values in any incoming events that are not mapped in the Avro schema will not be included in the final destination. Data Prepper does not allow the use of `include_keys` or `exclude_keys` with a custom schema so as to avoid confusion between a custom Avro schema and the `include_keys` or `exclude_keys` sink configurations. -In cases where your data is uniform, you may be able to automatically generate a schema. Automatically generated schemas are based on the first event that the codec receives. The schema will only contain keys from this event, and all keys must be present in all events in order for the automatically generated schema to produce a working schema. Automatically generated schemas make all fields nullable. Use the `include_keys` and `exclude_keys` sink configurations to control which data is included in the automatically generated schema. +In cases where your data is uniform, you may be able to automatically generate a schema. Automatically generated schemas are based on the first event that the codec receives. The schema will only contain keys from this event, and all keys must be present in all events in order to automatically generate a working schema. Automatically generated schemas make all fields nullable. Use the `include_keys` and `exclude_keys` sink configurations to control which data is included in the automatically generated schema. Avro fields should use a null [union](https://avro.apache.org/docs/current/specification/#unions) because this will allow missing values. Otherwise, all required fields must be present for each event. Use non-nullable fields only when you are certain they exist.