[CORE][SB][OTEL] Serialization operation failed due to unsupported type System.Byte[]

[macieyng]

• Package Name: azure-servicebus, azure-core, azure-core-tracing-opentelemetry
• Package Version: azure-core==1.26.4, azure-servicebus==7.10.0b1, azure-core-tracing-opentelemetry==1.0.0b9
• Operating System: Debian (irrelevant I guess)
• Python Version: python3.8.16

Describe the bug
We're implementing tracing with OpenTelemetry for our FastAPI application. We've added SDK tracing via azure-core-tracing-opentelemetry. Part of the application is to sent messages to ServiceBus. I was working fine before we merged that to our dev environment. The error we're getting comes from the Service Bus end (I guess), because description is unlike something Python would produce.
MessageSendFailed Error condition: ErrorCondition.LinkMessageSizeExceeded Error Description: Serialization operation failed due to unsupported type System.Byte[]. TrackingId:d19c43280000039d0001e2e8644b645d_G0S1_B9S1, SystemTracker:REDACTED:Topic:REDACTED, Timestamp:2023-04-28T06:14:54

Here is an object (redacted a little) that failed to be sent to Service Bus:
<Message(header=None, delivery_annotations=None, message_annotations=None, properties="<Properties(message_id='42fcf47e-8779-482f-9b32-b6b5b7538c70', user_id=None, to=None, subject=None, reply_to=None, correlation_id=None, content_type=None, content_encoding=None, absolute_expiry_time=None, creation_time=None, group_id=None, group_sequence=None, reply_to_group_id=None)>", application_properties={'event': <EventType.EVENT_TYPE: 'EVENT_TYPE'>, "b'Diagnostic-Id'": "b'00-4276d6cffb60f24aa3c47c1310b9c82e-68796d813923c983-01'"}, data=['b\'{"event": "EVENT_TYPE", "id": "fc90751b-8c85-42d6-bfde-a5138a4a809d", "created_at": "2023-04-28T06:14:46.663387Z", "sender": "sender", "data": {"id": "1c5d0205-4e7f-4d8d-9838-38293f86ca4c", "code": "62b96b41-1122-48bf-9888-50262a6569ab", "status": "OPEN", "entity": 1, "other_id": "16510526-a3cd-43d8-a424-efb02dbd3491", "role": "customer", "user_id": "8b97fb90-92cf-4b2d-b7c4-22f726e10f9a", "schema": "Schema", "value": 1500}}\''], sequence=None, value=None, footer=None)>

Error LinkMessageSizeExceeded suggests that message is too big, but the limit is 256kb afaik and no way this message is this big, so I would focus more on why serialization would fail and how we can treat that.

I would guess that there is something not right with application_properties and Diagnostic-Id (or traceparent), but maybe there's something wrong with our data structure that we can't spot.

To Reproduce
(to update)

Expected behavior
Message is sent properly to Service Bus.

Additional context
Add any other context about the problem here.

[macieyng]

Possibly related to sdk/servicebus/azure-servicebus/README.md
https://github.com/Azure/azure-sdk-for-python/blob/393e3c6c5cfdaa72d0c2ae12c481715fbcf0fed9/sdk/servicebus/azure-servicebus/README.md#pure-python-amqp-transport-and-backward-compatibility-support

[xiangyan99]

Thanks for the feedback, we’ll investigate asap.

[pvaneck]

Hey, @macieyng. Thanks for the detailed report. I actually just recently opened a PR to update ServiceBus tracing which should mitigate this issue, as I remember running into this myself when testing.

The issue seems to stem from adding a key-value pair to the message application properties when the value is bytes instead of str. In my PR, I adjusted it to just add the property as a string instead of encoding it (see here), and that worked fine.

Strangely enough, adding bytes as a value to message application properties seems to work fine in eventhubs from my testing. Any idea what could cause the discrepancy between the amqp implementations of servicebus and eventhubs?

/cc @kashifkhan @swathipil

[kashifkhan]

Thanks @pvaneck , seems like we have a fix for the situation already :)

to answer your question about the discrepancy -> EventHub is a little more forgiving with the amqp standard and allows adding in bytes while ServiceBus adhereing strictly. Per the standard application properties can only be simple types such as ints, bools and strings. Arrays are not included.

[swathipil]

Hi @macieyng - I'm currently running an opentelemetry sample with azure-servicebus==7.10.0b1 and not seeing this error. Would you be able to update here with a sample that reproduces this issue? Thanks!

[swathipil]

@macieyng - Updated the sample and am able to reproduce the issue now. Feel free to disregard the above - thanks!

[macieyng]

@kashifkhan happy to hear that the issue was resolved! When can we expect release with the fix?

[kashifkhan]

@macieyng that PR was merged last week and we are planning a new release for service bus this week. Ill update this thread once its out on pypi

[kashifkhan]

@macieyng we released azure-servicebus 7.10.0 today which should fix the problem :)

[github-actions]

Hi @macieyng. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.

[macieyng]

Thanks @kashifkhan! I'll let you know in couple of weeks how's that working for us. Few other things have priority over this for us. I believe that the issue can be closed. In case of any issues, I'll reopen it.