Experimental support for Log AnyValue body #5880

jack-berg · 2023-10-04T22:44:27Z

Resolves #5847.

This adds experimental support for recording AnyValue for log record bodies. A summary:

(almost) All changes are in the incubator package for now
Add a new interface called AnyValue which mirrors the proto AnyValue definition.
Add a overload of LogRecordBuilder#setBody(..) which accepts AnyValue as an argument.
In the SDK, extend Body.Type enum with a new value ANY_VALUE (other values are STRING and EMPTY). Unfortunately, while Body is aligned with the new AnyValue (it has a getType() method and could theoretically be extended to represent arbitrarily complex data) the name isn't right. It would be weird to have a body nested within a body nested within a body. So Body appears destined to be a wrapper around AnyValue. This was an API design miss in the log SDK. Still, the ergonomics shouldn't be too bad.
To use, exporters should check the Body#getType(). If it is AnyValue, they can cast it to AnyValueBody, access the AnyValue, and encode it however they see fit. If the exporter protocol doesn't understand how to export AnyValueBody, it can use Body#asString(), which will encode the AnyValue data into a String. The AnyValueBody#asString() implementation currently produces a java-toString style response, but we could substitute a JSON encoding to make it even more useful.

Will need to followup on this to update LogMarshaler to properly encode this to OTLP. Currently, it will just send to the asString() representation.

codecov · 2023-10-04T23:04:32Z

Codecov Report

Attention: 23 lines in your changes are missing coverage. Please review.

Files	Coverage Δ
...etry/exporter/internal/otlp/logs/LogMarshaler.java	`68.75% <ø> (ø)`
...entelemetry/extension/incubator/logs/AnyValue.java	`100.00% <100.00%> (ø)`
...lemetry/extension/incubator/logs/AnyValueType.java	`100.00% <100.00%> (ø)`
...elemetry/extension/incubator/logs/KeyAnyValue.java	`100.00% <100.00%> (ø)`
...etry/extension/incubator/logs/KeyAnyValueImpl.java	`100.00% <100.00%> (ø)`
...io/opentelemetry/sdk/logs/SdkLogRecordBuilder.java	`97.95% <100.00%> (+0.08%)`	⬆️
...main/java/io/opentelemetry/sdk/logs/data/Body.java	`100.00% <ø> (ø)`
.../opentelemetry/sdk/logs/internal/AnyValueBody.java	`75.00% <75.00%> (ø)`
...emetry/extension/incubator/logs/AnyValueArray.java	`80.00% <80.00%> (ø)`
...etry/extension/incubator/logs/AnyValueBoolean.java	`75.00% <75.00%> (ø)`
... and 5 more

... and 11 files with indirect coverage changes

📢 Thoughts on this report? Let us know!.

jackshirazi · 2023-10-12T11:58:52Z

Is this effectively a new serialization type? What is the advantage over declaring a new interface (like

interface LogBody { byte[] convertToBytes(); }

or even just allowing Externalizable objects or an equivalent interface?

jack-berg · 2023-10-12T15:02:49Z

@jackshirazi the log record body field is defined in the data model as type any. See proto definition for body, and AnyValue.

So far, we've only known about use cases where log record body is a string, so we've restricted it. However, new use cases have emerged which require the log record body to include structure:

Structured logs in slf4j / logback / log4j. We're currently mapping the key value pairs of the structured log to attributes, but this isn't quite right - they really represent the body of the log in the same way a string message. Also, mapping the structured log key value pairs to the body ensures there aren't any conflicts with MDC keys, which we map to attributes.
Events. We want to support events with complex types. When mapping events from other systems, its important to keep the structure as close as possible to the original, and AnyValue facilitates that.

So its not as simple as having a type that can be serializes to byte[] - we need to know about the structure / types of the fields so we can encode it to proto AnyValue.

…y-java into log-body-any-value

jack-berg · 2023-10-13T20:57:21Z

I haven't heard any criticism of this approach so I've gone ahead and tidied it up and marked it ready for review.

breedx-splk

Hey @jack-berg this is great! I had a few comments/questions, but overall this feels solid to me.

...mon/src/test/java/io/opentelemetry/exporter/internal/otlp/logs/LogsRequestMarshalerTest.java

extensions/incubator/src/main/java/io/opentelemetry/extension/incubator/logs/AnyValueBytes.java

...ubator/src/main/java/io/opentelemetry/extension/incubator/logs/ExtendedLogRecordBuilder.java

...sions/incubator/src/main/java/io/opentelemetry/extension/incubator/logs/KeyAnyValueList.java

extensions/incubator/src/test/java/io/opentelemetry/extension/incubator/logs/AnyValueTest.java

sdk/logs/src/test/java/io/opentelemetry/sdk/logs/AnyValueBodyTest.java

breedx-splk · 2023-10-19T19:17:37Z

extensions/incubator/src/main/java/io/opentelemetry/extension/incubator/logs/AnyValueBytes.java

-  public byte[] getValue() {
-    return value;
+  public ByteBuffer getValue() {
+    return ByteBuffer.wrap(raw).asReadOnlyBuffer();


breedx-splk · 2023-10-19T19:18:55Z

extensions/incubator/src/test/java/io/opentelemetry/extension/incubator/logs/AnyValueTest.java

-              assertThat(anyValue.getValue()).isEqualTo(new byte[] {'a', 'b'});
+              ByteBuffer value = anyValue.getValue();
+              // AnyValueBytes returns read only view of ByteBuffer
+              assertThatThrownBy(value::array).isInstanceOf(ReadOnlyBufferException.class);


jack-berg · 2023-10-24T20:57:14Z

@open-telemetry/java-approvers PTAL when you have a chance. I've got a followup PR drafted in #5938 which extends the OTLP marshalers to be able to handle AnyValue. Would be good to bundle this and #5938 in the same release.

breedx-splk · 2023-10-24T23:11:47Z

This also very much relates to the ongoing discussions around event.data vs. this style of event body. See open-telemetry/opentelemetry-android#122 and the working group open-telemetry/community#1702 and https://github.com/orgs/open-telemetry/projects/65/views/1

jack-berg · 2023-10-30T16:33:10Z

Will merge tomorrow if there are no additional comments. Need to leave time for the followup #5938 before the next release.

jack-berg · 2023-10-31T18:32:59Z

In the interest of being very conservative, I've removed the new Body#ANY_VALUE enum option. There is now no new public API surface area.

Anyone wishing to access the AnyValue in an LogRecordExporter should check if the if(logRecord.getBody() instanceof AnyValueBody).

Log AnyValue body prototype

363cd27

Improve API ergonomics

9b81423

LikeTheSalad mentioned this pull request Oct 11, 2023

Add mobile crash data model spec open-telemetry/semantic-conventions#362

Closed

jack-berg added 3 commits October 13, 2023 15:54

Clean up to prepare for review

057ebbc

Merge branch 'main' of https://github.com/open-telemetry/opentelemetr…

465377d

…y-java into log-body-any-value

japicmp

943c3fa

jack-berg marked this pull request as ready for review October 13, 2023 20:57

jack-berg requested a review from a team October 13, 2023 20:57

jack-berg changed the title ~~Log AnyValue body prototype~~ Experimental support for Log AnyValue body Oct 13, 2023

breedx-splk reviewed Oct 16, 2023

View reviewed changes

PR feedback

07a51c2

breedx-splk reviewed Oct 19, 2023

View reviewed changes

breedx-splk approved these changes Oct 20, 2023

View reviewed changes

This was referenced Oct 23, 2023

Update minimum android version #5936

Closed

Serialize log body any value #5938

Merged

jack-berg mentioned this pull request Oct 30, 2023

Remove mdc attributes prefix open-telemetry/opentelemetry-java-instrumentation#9536

Merged

Remove Body#ANY_VALUE

72daf32

jack-berg merged commit efa46a5 into open-telemetry:main Oct 31, 2023
18 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Experimental support for Log AnyValue body #5880

Experimental support for Log AnyValue body #5880

jack-berg commented Oct 4, 2023 •

edited

Loading

codecov bot commented Oct 4, 2023 •

edited

Loading

jackshirazi commented Oct 12, 2023 •

edited

Loading

jack-berg commented Oct 12, 2023

jack-berg commented Oct 13, 2023

breedx-splk left a comment

breedx-splk Oct 19, 2023

breedx-splk Oct 19, 2023

jack-berg commented Oct 24, 2023

breedx-splk commented Oct 24, 2023

jack-berg commented Oct 30, 2023

jack-berg commented Oct 31, 2023

Experimental support for Log AnyValue body #5880

Experimental support for Log AnyValue body #5880

Conversation

jack-berg commented Oct 4, 2023 • edited Loading

codecov bot commented Oct 4, 2023 • edited Loading

Codecov Report

jackshirazi commented Oct 12, 2023 • edited Loading

jack-berg commented Oct 12, 2023

jack-berg commented Oct 13, 2023

breedx-splk left a comment

Choose a reason for hiding this comment

breedx-splk Oct 19, 2023

Choose a reason for hiding this comment

breedx-splk Oct 19, 2023

Choose a reason for hiding this comment

jack-berg commented Oct 24, 2023

breedx-splk commented Oct 24, 2023

jack-berg commented Oct 30, 2023

jack-berg commented Oct 31, 2023

jack-berg commented Oct 4, 2023 •

edited

Loading

codecov bot commented Oct 4, 2023 •

edited

Loading

jackshirazi commented Oct 12, 2023 •

edited

Loading