Support for Event Json input and output codecs #4436

kkondaka · 2024-04-17T23:49:04Z

Description

Support for event_json input and output codecs

Issues Resolved

Resolves #4404

Check List

[X ] New functionality includes testing.
New functionality has a documentation issue. Please link to it in this PR.
- New functionality has javadoc added
[X ] Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Krishna Kondaka <[email protected]>

graytaylor0 · 2024-04-18T17:03:20Z

...cs/src/main/java/org/opensearch/dataprepper/plugins/codec/eventjson/EventJsonInputCodec.java

+                    .getThis();
+                final JacksonEvent event = (JacksonEvent)logBuilder.build();
+                final Record<Event> record =  new Record<>(event);
+                final String externalOriginationTime = (String)metadata.get(EventJsonDefines.EXTERNAL_ORIGINATION_TIME);


You could probably have Jackson handle these conversions by annotating the DefaultEventMetadata properties and deserializing the metadata to the DefaultEventMetadata class

Is the time an Instant in production use cases?

graytaylor0 · 2024-04-18T17:05:24Z

...cs/src/main/java/org/opensearch/dataprepper/plugins/codec/eventjson/EventJsonInputCodec.java

+                final JacksonLog.Builder logBuilder = JacksonLog.builder()
+                    .withData(data)
+                    .withEventMetadataAttributes((Map<String, Object>)metadata.get(EventJsonDefines.ATTRIBUTES))
+                    .withTimeReceived(Instant.parse((String)metadata.get(EventJsonDefines.TIME_RECEIVED)))


We should make this configurable and default to not passing anything here so that the new time the Event reaches the source is applied to the Event.

event time received should not be optional, I think. Tags, attributes and externalorigination time are optional.

The default behavior when using this codec should be to create a new time received time, which is the default behavior when an Event is created.

Signed-off-by: Krishna Kondaka <[email protected]>

graytaylor0 · 2024-04-18T21:18:30Z

There's still some missing test coverage in data-prepper-api

dinujoh · 2024-04-18T21:20:12Z

data-prepper-api/src/main/java/org/opensearch/dataprepper/model/event/DefaultEventMetadata.java

+/*
+    public void setEventType(String eType) {
+        eventType = eType;
+    }
+
+    public void setTimeReceived(Instant receivedTime) {
+        timeReceived = receivedTime;
+    }
+*/


Do we need this ?

Signed-off-by: Krishna Kondaka <[email protected]>

dlvenable · 2024-04-18T22:49:58Z

...odecs/src/main/java/org/opensearch/dataprepper/plugins/codec/eventjson/EventJsonDefines.java

+ * Copyright OpenSearch Contributors
+ * SPDX-License-Identifier: Apache-2.0
+ */
+package org.opensearch.dataprepper.plugins.codec.eventjson;


The Java convention for packages with spaces is to use underscore. So you could rename to:

package org.opensearch.dataprepper.plugins.codec.event_json;

dlvenable · 2024-04-18T22:53:39Z

...t/java/org/opensearch/dataprepper/plugins/codec/eventjson/EventJsonInputOutputCodecTest.java

+        outputCodec.writeEvent(event, outputStream);
+        outputCodec.complete(outputStream);
+        assertThat(outputCodec.getExtension(), equalTo(EventJsonOutputCodec.EVENT_JSON));
+        inputCodec.parse(new ByteArrayInputStream(outputStream.toByteArray()), record -> {


There is no guarantee that this consumer is even called.

A simple change is to update the consumer to put the records into a collection. Then assert the size. Then assert the format.

List<Record> records = new LinkedList<>(); inputCodec.parse(new ByteArrayInputStream(outputStream.toByteArray()), records::add); assertThat(records.size(), equalTo(1)); for(Record record : records) { Event e = record.getData(); // All the same from here. }

dlvenable · 2024-04-18T22:56:07Z

...s/src/main/java/org/opensearch/dataprepper/plugins/codec/eventjson/EventJsonOutputCodec.java

+    public void start(OutputStream outputStream, Event event, OutputCodecContext context) throws IOException {
+        Objects.requireNonNull(outputStream);
+        generator = factory.createGenerator(outputStream, JsonEncoding.UTF8);
+        generator.writeStartObject();


It appears that you are starting an object. But, there is no array. So won't it look like this?

{{"data":..., "metadata"...},{"data":..., "metadata"...},{"data":..., "metadata"...}}

Good point. Modified to generate an array.

dlvenable · 2024-04-18T22:56:43Z

...t/java/org/opensearch/dataprepper/plugins/codec/eventjson/EventJsonInputOutputCodecTest.java

+        outputCodec.start(outputStream, null, null);
+        outputCodec.writeEvent(event, outputStream);
+        outputCodec.complete(outputStream);
+        inputCodec.parse(new ByteArrayInputStream(outputStream.toByteArray()), record -> {


Similar comment as below about the assertions here.

dlvenable · 2024-04-18T22:57:02Z

...t/java/org/opensearch/dataprepper/plugins/codec/eventjson/EventJsonInputOutputCodecTest.java

+    }
+
+    @Test
+    public void extendedTest() throws Exception {


We should have a test that operates against multiple events.

dlvenable · 2024-04-18T22:58:36Z

...t/java/org/opensearch/dataprepper/plugins/codec/eventjson/EventJsonInputOutputCodecTest.java

+import java.util.UUID;
+import java.util.stream.Collectors;
+
+public class EventJsonInputOutputCodecTest {


I like that this test is asserting symmetry between the codecs.

We should also have test suites for the individual codecs with known values to ensure that the format is consistent. That is, somebody could make a change which causes the input codec to no longer work with the same values. The current tests would pass, but we would have broken backward compatibility which could pose problems when using this to write to S3.

dlvenable · 2024-04-18T22:59:20Z

...main/java/org/opensearch/dataprepper/plugins/codec/eventjson/EventJsonOutputCodecConfig.java

+ */
+package org.opensearch.dataprepper.plugins.codec.eventjson;
+
+public class EventJsonOutputCodecConfig {


Do we need this?

Just keeping it for future use where we may want to control what fields should be included in the output.

dlvenable · 2024-04-18T23:02:23Z

...t/java/org/opensearch/dataprepper/plugins/codec/eventjson/EventJsonInputOutputCodecTest.java

+    public void emptyTest() throws Exception {
+        ByteArrayInputStream inputStream = new ByteArrayInputStream("".getBytes());
+        inputCodec = createInputCodec();
+        inputCodec.parse(inputStream, record -> { });


Make these different tests for isolation:

@ParameterizedTest @ValueSource(strings = { "[]", "[{}]", "{}" }) ...

dlvenable · 2024-04-18T23:03:16Z

...t/java/org/opensearch/dataprepper/plugins/codec/eventjson/EventJsonInputOutputCodecTest.java

+        inputCodec = createInputCodec();
+        inputCodec.parse(inputStream, record -> { });
+        inputStream = new ByteArrayInputStream("[]".getBytes());
+        inputCodec.parse(inputStream, record -> { });


We should do something to assert this.

Probably:

Consumer<Record<Event>> consumer = mock(Consumer.class); inputCodec.parse(inputStream, consumer); verifyNoInteractions(consumer);

Signed-off-by: Krishna Kondaka <[email protected]>

dlvenable · 2024-04-19T15:01:14Z

.../src/main/java/org/opensearch/dataprepper/plugins/codec/event_json/EventJsonOutputCodec.java

@@ -44,22 +44,24 @@ public String getExtension() {
    public void start(OutputStream outputStream, Event event, OutputCodecContext context) throws IOException {
        Objects.requireNonNull(outputStream);
        generator = factory.createGenerator(outputStream, JsonEncoding.UTF8);
-        generator.writeStartObject();
+        generator.writeStartArray();


We should keep the writeStartObject.

This could allow us to also include metadata such as the version.

e.g.

{ "metadata" : {"version" : "2.8"}, "events" : [ {"data":...,"metadata"}, {"data":...,"metadata"}, {"data":...,"metadata"}, ] }

See this code:

data-prepper/data-prepper-plugins/parse-json-processor/src/main/java/org/opensearch/dataprepper/plugins/codec/json/JsonOutputCodec.java

Lines 55 to 57 in f9a3a60

generator.writeStartObject();

generator.writeFieldName(config.getKeyName());

generator.writeStartArray();

Though, I don't think we need to make the events key configurable.

Good point. But the following should be enough.

{ "version" : "2.8", "events" : [ {"data":...,"metadata"}, {"data":...,"metadata"}, {"data":...,"metadata"}, ] }

graytaylor0 · 2024-04-19T16:04:39Z

...s/src/main/java/org/opensearch/dataprepper/plugins/codec/event_json/EventJsonInputCodec.java

+        if (data == null) {
+            return null;
+        }
+        if (overrideTimeReceived) {


I think you want to run this part if override_time_received is false

You mean, the overrideTimeReceived flag "true" means it will override with the value from the data and "false" means it will take Instant.now()?

graytaylor0 · 2024-04-19T16:09:05Z

.../java/org/opensearch/dataprepper/plugins/codec/event_json/EventJsonInputOutputCodecTest.java

+    }
+
+    public EventJsonInputCodec createInputCodec() {
+        when(eventJsonInputCodecConfig.getOverrideTimeReceived()).thenReturn(false);


Let's move this to setup and then add a test for when the override_time_received is true

Modified as per the suggestion

Signed-off-by: Krishna Kondaka <[email protected]>

dlvenable · 2024-04-22T15:42:05Z

...s/src/main/java/org/opensearch/dataprepper/plugins/codec/event_json/EventJsonInputCodec.java

+                final Map<String, Object> innerJson = objectMapper.readValue(jsonParser, Map.class);
+                final List<Map<String, Object>> maps = (List<Map<String, Object>>)innerJson.get(EventJsonDefines.EVENTS);
+                final String version = (String)innerJson.get(EventJsonDefines.VERSION);
+                if (!version.equals(DataPrepperVersion.getCurrentVersion().toString())) {


Do we need strict matching here? One thought is that we could be strict on the major version and ignore the minor version.

dlvenable · 2024-04-22T15:42:51Z

...s/src/main/java/org/opensearch/dataprepper/plugins/codec/event_json/EventJsonInputCodec.java

+            if (jsonParser.getCurrentToken() == JsonToken.START_OBJECT) {
+                final Map<String, Object> innerJson = objectMapper.readValue(jsonParser, Map.class);
+                final List<Map<String, Object>> maps = (List<Map<String, Object>>)innerJson.get(EventJsonDefines.EVENTS);
+                final String version = (String)innerJson.get(EventJsonDefines.VERSION);


It may be good to move this check into a small method to help make the code more readable.

Signed-off-by: Krishna Kondaka <[email protected]>

dlvenable · 2024-04-22T20:17:46Z

...s/src/main/java/org/opensearch/dataprepper/plugins/codec/event_json/EventJsonInputCodec.java

@@ -48,6 +48,22 @@ public EventJsonInputCodec(final EventJsonInputCodecConfig config) {
        this.overrideTimeReceived = config.getOverrideTimeReceived();
    }

+    private boolean isCompatibleVersion(Map<String, Object> json) {
+        final String versionStr = (String)json.get(EventJsonDefines.VERSION);


The DataPrepperVersion class provides a lot of this.

final String versionStr = (String)json.get(EventJsonDefines.VERSION); final DataPrepperVersion definedVersion = DataPrepperVersion.parse(versionStr); if(definedVersion.getMajorVersion() != DataPrepperVersion.getCurrentVersion().getMajorVersion()) { LOG.error("Version mismatch! Current version {} Received data version {}", DataPrepperVersion.getCurrentVersion(), definedVersion); return false; }

Signed-off-by: Krishna Kondaka <[email protected]>

dlvenable

Thanks for adding this and making these improvements to the feature!

* Event Json input and output codecs Signed-off-by: Krishna Kondaka <[email protected]> * Modified test case to check for event metadata attributes Signed-off-by: Krishna Kondaka <[email protected]> * Modified the coverage to 0.9 Signed-off-by: Krishna Kondaka <[email protected]> * Addressed review comments Signed-off-by: Krishna Kondaka <[email protected]> * Fixes for failing coverage tests Signed-off-by: Krishna Kondaka <[email protected]> * Addressed review comments Signed-off-by: Krishna Kondaka <[email protected]> * Fixed test coverage Signed-off-by: Krishna Kondaka <[email protected]> * Added more tests Signed-off-by: Krishna Kondaka <[email protected]> * Added more tests for coverage Signed-off-by: Krishna Kondaka <[email protected]> * Addressed review comments Signed-off-by: Krishna Kondaka <[email protected]> * Fixed code coverage failure Signed-off-by: Krishna Kondaka <[email protected]> * Addressed review comments Signed-off-by: Krishna Kondaka <[email protected]> * Addressed review comments Signed-off-by: Krishna Kondaka <[email protected]> * Addressed review comments Signed-off-by: Krishna Kondaka <[email protected]> --------- Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

Event Json input and output codecs

8e8bdb9

Signed-off-by: Krishna Kondaka <[email protected]>

kkondaka requested review from chenqi0805, engechas, graytaylor0, dinujoh, asifsmohammed, KarstenSchnitter, dlvenable and oeyh as code owners April 17, 2024 23:49

Krishna Kondaka added 2 commits April 18, 2024 00:36

Modified test case to check for event metadata attributes

f255900

Signed-off-by: Krishna Kondaka <[email protected]>

Modified the coverage to 0.9

26e2674

Signed-off-by: Krishna Kondaka <[email protected]>

graytaylor0 reviewed Apr 18, 2024

View reviewed changes

Addressed review comments

1f74b6d

Signed-off-by: Krishna Kondaka <[email protected]>

dinujoh reviewed Apr 18, 2024

View reviewed changes

Fixes for failing coverage tests

b36af73

Signed-off-by: Krishna Kondaka <[email protected]>

dlvenable requested changes Apr 18, 2024

View reviewed changes

Krishna Kondaka added 4 commits April 19, 2024 05:26

Addressed review comments

c8c6d3d

Signed-off-by: Krishna Kondaka <[email protected]>

Fixed test coverage

6ce0c8b

Signed-off-by: Krishna Kondaka <[email protected]>

Added more tests

fe5c7fc

Signed-off-by: Krishna Kondaka <[email protected]>

Added more tests for coverage

24c6d79

Signed-off-by: Krishna Kondaka <[email protected]>

dlvenable requested changes Apr 19, 2024

View reviewed changes

graytaylor0 reviewed Apr 19, 2024

View reviewed changes

Krishna Kondaka added 2 commits April 20, 2024 01:59

Addressed review comments

8fb9d50

Signed-off-by: Krishna Kondaka <[email protected]>

Fixed code coverage failure

4f3b17e

Signed-off-by: Krishna Kondaka <[email protected]>

dlvenable requested changes Apr 22, 2024

View reviewed changes

graytaylor0 previously approved these changes Apr 22, 2024

View reviewed changes

Addressed review comments

cf1ae56

Signed-off-by: Krishna Kondaka <[email protected]>

kkondaka dismissed graytaylor0’s stale review via cf1ae56 April 22, 2024 17:50

dlvenable requested changes Apr 22, 2024

View reviewed changes

Krishna Kondaka added 2 commits April 22, 2024 20:32

Addressed review comments

8957ec6

Signed-off-by: Krishna Kondaka <[email protected]>

Addressed review comments

d47a923

Signed-off-by: Krishna Kondaka <[email protected]>

dlvenable approved these changes Apr 22, 2024

View reviewed changes

graytaylor0 approved these changes Apr 22, 2024

View reviewed changes

kkondaka merged commit df17cc5 into opensearch-project:main Apr 22, 2024
72 of 74 checks passed

kkondaka added this to the v2.8 milestone May 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for Event Json input and output codecs #4436

Support for Event Json input and output codecs #4436

kkondaka commented Apr 17, 2024

graytaylor0 Apr 18, 2024

dlvenable Apr 18, 2024

graytaylor0 Apr 18, 2024

kkondaka Apr 18, 2024

graytaylor0 Apr 18, 2024

graytaylor0 commented Apr 18, 2024

dinujoh Apr 18, 2024

dlvenable Apr 18, 2024

dlvenable Apr 18, 2024

dlvenable Apr 18, 2024

kkondaka Apr 19, 2024

dlvenable Apr 18, 2024

dlvenable Apr 18, 2024

dlvenable Apr 18, 2024

dlvenable Apr 18, 2024

kkondaka Apr 18, 2024

dlvenable Apr 18, 2024

dlvenable Apr 18, 2024

dlvenable Apr 19, 2024

kkondaka Apr 20, 2024

graytaylor0 Apr 19, 2024

kkondaka Apr 19, 2024

graytaylor0 Apr 19, 2024

kkondaka Apr 20, 2024

dlvenable Apr 22, 2024

dlvenable Apr 22, 2024

dlvenable Apr 22, 2024

dlvenable left a comment

	generator.writeStartObject();
	generator.writeFieldName(config.getKeyName());
	generator.writeStartArray();

Support for Event Json input and output codecs #4436

Support for Event Json input and output codecs #4436

Conversation

kkondaka commented Apr 17, 2024

Description

Issues Resolved

Check List

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

graytaylor0 commented Apr 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dlvenable left a comment

Choose a reason for hiding this comment