-
Notifications
You must be signed in to change notification settings - Fork 349
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support writing large json base64 and string values #2846
Conversation
test/FunctionalTests/Microsoft.OData.Core.Tests/Json/ODataUtf8JsonWriterTests.cs
Outdated
Show resolved
Hide resolved
This PR has Quantification details
Why proper sizing of changes matters
Optimal pull request sizes drive a better predictable PR flow as they strike a
What can I do to optimize my changes
How to interpret the change counts in git diff output
Was this comment helpful? 👍 :ok_hand: :thumbsdown: (Email) |
This PR has Quantification details
Why proper sizing of changes matters
Optimal pull request sizes drive a better predictable PR flow as they strike a
What can I do to optimize my changes
How to interpret the change counts in git diff output
Was this comment helpful? 👍 :ok_hand: :thumbsdown: (Email) |
This PR has Quantification details
Why proper sizing of changes matters
Optimal pull request sizes drive a better predictable PR flow as they strike a
What can I do to optimize my changes
How to interpret the change counts in git diff output
Was this comment helpful? 👍 :ok_hand: :thumbsdown: (Email) |
This PR has Quantification details
Why proper sizing of changes matters
Optimal pull request sizes drive a better predictable PR flow as they strike a
What can I do to optimize my changes
How to interpret the change counts in git diff output
Was this comment helpful? 👍 :ok_hand: :thumbsdown: (Email) |
aea2d9d
to
38d01e3
Compare
This PR has Quantification details
Why proper sizing of changes matters
Optimal pull request sizes drive a better predictable PR flow as they strike a
What can I do to optimize my changes
How to interpret the change counts in git diff output
Was this comment helpful? 👍 :ok_hand: :thumbsdown: (Email) |
This PR has Quantification details
Why proper sizing of changes matters
Optimal pull request sizes drive a better predictable PR flow as they strike a
What can I do to optimize my changes
How to interpret the change counts in git diff output
Was this comment helpful? 👍 :ok_hand: :thumbsdown: (Email) |
test/FunctionalTests/Microsoft.OData.Core.Tests/Json/ODataUtf8JsonWriterTests.cs
Show resolved
Hide resolved
This PR has Quantification details
Why proper sizing of changes matters
Optimal pull request sizes drive a better predictable PR flow as they strike a
What can I do to optimize my changes
How to interpret the change counts in git diff output
Was this comment helpful? 👍 :ok_hand: :thumbsdown: (Email) |
This PR has Quantification details
Why proper sizing of changes matters
Optimal pull request sizes drive a better predictable PR flow as they strike a
What can I do to optimize my changes
How to interpret the change counts in git diff output
Was this comment helpful? 👍 :ok_hand: :thumbsdown: (Email) |
This PR has Quantification details
Why proper sizing of changes matters
Optimal pull request sizes drive a better predictable PR flow as they strike a
What can I do to optimize my changes
How to interpret the change counts in git diff output
Was this comment helpful? 👍 :ok_hand: :thumbsdown: (Email) |
This PR has Quantification details
Why proper sizing of changes matters
Optimal pull request sizes drive a better predictable PR flow as they strike a
What can I do to optimize my changes
How to interpret the change counts in git diff output
Was this comment helpful? 👍 :ok_hand: :thumbsdown: (Email) |
This PR has Quantification details
Why proper sizing of changes matters
Optimal pull request sizes drive a better predictable PR flow as they strike a
What can I do to optimize my changes
How to interpret the change counts in git diff output
Was this comment helpful? 👍 :ok_hand: :thumbsdown: (Email) |
* support writing large string values in utf8jsonwriter * update based on review comments * update based on review comments * update based on review comments * update based on review comments * update based on review comments. * update based on review comments
* support writing large string values in utf8jsonwriter * update based on review comments * update based on review comments * update based on review comments * update based on review comments * update based on review comments. * update based on review comments
…ta#2873) * support writing large string values in utf8jsonwriter * update based on review comments * update based on review comments * update based on review comments * update based on review comments * update based on review comments. * update based on review comments
Issues
This pull request fixes #2845.
Description
This pull request introduces functionality to handle the writing of large string and byte array values. It accomplishes this by breaking down oversized strings and bytes into smaller chunks and directly writing them to the buffer.
The process involves first checking if the size of the string or byte array exceeds the buffer's capacity. If it does, the program iterates through the large string, dividing it into chunks of a predefined size. Each chunk is then written to the buffer until the entire string is successfully written.
Furthermore, special characters within the string are properly escaped to ensure accurate representation during the writing process.
Methods added:
LOH allocations before these changes:
char[] in LOH:
byte[] in LOH from growing the writer buffer when calling WriteStringValue
LOH allocations after these changes:
No byte[] or char[] in the LOH that comes from ODataUtf8JsonWriter
Latency Before:
Bombarding https://localhost:7120/customers/ODataMessageWriter-Utf8JsonWriter-Async?count=10&largeFields=true for 30s using 30 connection(s)
[=================================================================================================================] 30s
Done!
Statistics Avg Stdev Max
Reqs/sec 23.70 69.93 1492.98
Latency 1.71s 0.87s 7.36s
Latency Distribution
50% 1.60s
75% 1.73s
90% 1.87s
95% 2.29s
99% 6.58s
HTTP codes:
1xx - 0, 2xx - 534, 3xx - 0, 4xx - 0, 5xx - 0
others - 2
Errors:
tls handshake timed out - 2
Throughput: 533.38MB/s
Latency After:
Bombarding https://localhost:7120/customers/ODataMessageWriter-Utf8JsonWriter-Async?count=10&largeFields=true for 30s using 30 connection(s)
[=================================================================================================================] 30s
Done!
Statistics Avg Stdev Max
Reqs/sec 26.07 44.40 429.41
Latency 1.41s 578.31ms 5.48s
Latency Distribution
50% 1.33s
75% 1.47s
90% 1.64s
95% 1.86s
99% 4.79s
HTTP codes:
1xx - 0, 2xx - 648, 3xx - 0, 4xx - 0, 5xx - 0
others - 0
Throughput: 651.77MB/s
With this changes, there is an improvement in latency and there are no allocations in the LOH.
Checklist (Uncheck if it is not completed)
Additional work necessary
If documentation update is needed, please add "Docs Needed" label to the issue and provide details about the required document change in the issue.