Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Option to JSON encode Decimal types as numbers, not strings #440

Closed
jcrist opened this issue Jun 12, 2023 · 2 comments · Fixed by #465
Closed

Option to JSON encode Decimal types as numbers, not strings #440

jcrist opened this issue Jun 12, 2023 · 2 comments · Fixed by #465

Comments

@jcrist
Copy link
Owner

jcrist commented Jun 12, 2023

We like that msgspec handles decimal however they are embedded as strings. Here is some finding from another discussion around handling decimals.

meltano/sdk#1046 (comment)

Do we think that msgspec might output decimals like simplejson without the double quotes in the future?

Originally posted by @s7clarke10 in #349 (comment)

@jcrist
Copy link
Owner Author

jcrist commented Jun 12, 2023

In the future I'd expect to make the encoding of Decimal types configurable, but currently that's not possible. However, I'd argue that encoding Decimal types as strings is the optimal format, and would encourage you to rethink using numbers for these objects.

Encoding decimals as strings makes it easier for downstream consumers of the JSON to properly load decimal objects without losing precision. Most JSON libraries (both in Python and in other languages) don't support hooking into the parsing process for numbers (123, 123.45, ...) - they almost always get converted to a float64 before being passed off to downstream code. This means that encoding decimals as numbers would prevent most libraries from reading this data with full precision.

For example, using JSON.parse in javascript, then converting the output into a decimal from decimal.js only accurately roundtrips the decimals if they're serialized as JSON strings, not JSON numbers. This is because there's no way to grab the string representation of the number before it's parsed into a float64 (where it loses precision). You can see this in this JSFiddle.

@s7clarke10
Copy link

s7clarke10 commented Jun 12, 2023

Brilliant, thank you for looking into the use case for this and raising a new issue.

I agree with you that outputting as a string can avoid downstream issues including truncation / rounding, we have certainly seen this issue. In some situations we wish to treat decimals as decimals, so the option to make this configurable sounds like a good path forward. Thank you for you efforts in producing this package.

@jcrist jcrist changed the title JSON encode Decimal types as numbers, not strings Option to JSON encode Decimal types as numbers, not strings Jun 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants