-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(spanner): use json.Number for decoding unknown values from spanner #9054
Conversation
# Conflicts: # spanner/value.go
fa2b944
to
9608e01
Compare
be1e957
to
2133585
Compare
spanner/value.go
Outdated
jsonProvider jsoniter.API | ||
|
||
once sync.Once |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If I understand this correctly, it means that jsonProvider
is set once as a global variable the first time a client is created. That means that if you create a new client later with a different JSON decoding configuration, it will have no effect. Although it is not a very common use case, it is also very confusing. If we can only provide this as a global variable, then we should model it as such in the API as well, and clearly call out how it should be used.
spanner/client.go
Outdated
// UseNumber causes the Decoder to unmarshal a number into an interface{} as a | ||
// Number instead of as a float64. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm only able to understand this comment because I know the context of this PR. Without that context, this comment contains too little information. Can we add:
- Additional context that explains that this is for the JSON decoder. (
the Decoder
is not a clear reference to JSON decoding) - What it means that something is being unmarshalled as a
Number
instead of afloat64
? So what are the pros/cons of each?
So for example:
UseNumber specifies whether number values inside a Cloud Spanner JSON value should be decoded as a Number or a float64. Decoding to a Number guarantees that the precision used by Cloud Spanner is preserved. Decoding to a float64 can cause loss of precision. The default JSON decode function in Go uses float64. This is therefore also the default used by this client library. Change this value to true
to prevent loss of precision.
spanner/client.go
Outdated
|
||
once.Do(func() { | ||
// Initialize json provider. | ||
jsonProvider = jsoniter.Config{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See also my comment below: This will be confusing for anyone who calls NewClientWithConfig
first with UseNumber: false
and then later again with UseNumber: true
. The configuration appears to be for a single client, but is in reality a global variable. We need to change that so it is either clearer, or it is actually a client config. Possible solutions for that could be:
- Add a global method that sets the preferred decode method. Clearly document that the value that is set will be used by all clients, including both clients that have already been created and clients that will be created in the future. Based on where the configuration option is needed, this is probably the best/only possible solution.
- OR: Add overloaded functions that accept an argument that determines which decoding method should be used wherever possible (this would not have my preference).
- OR (I don't think it is possible, but for completeness): Pass a reference to the config on to all objects that need to know, and use the value there.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I pushed changes for approach 1.
5291a3a
to
bb44f9b
Compare
60903bb
to
a190f53
Compare
a190f53
to
0c5fe88
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should make it possible for customers to both enable and disable the feature, as it is possible that they want to test their code with both options.
spanner/integration_test.go
Outdated
for idx := 0; idx < 2; idx++ { | ||
if idx == 1 { | ||
UseNumberWithJSONDecoderEncoder() | ||
defer testSetJSONProviderNumberConfig(false) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: can we change this into
- First get the current value of
UseNumberWithJSONDecoderEncoder
- A loop that loops through two boolean value true and false
- Then calls
UseNumberWithJSONDecoderEncoder(boolValue)
- Then after the loop set the
UseNumberWithJSONDecoderEncoder
to the original value.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also: Are we actually using this to check whether we get the expected value? In other words: I don't see any tests that are conditional on whether idx == 1
or idx == 0
, and that has a different expected outcome depending on that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is condition on whether idx is 0/1, https://github.com/googleapis/google-cloud-go/pull/9054/files#diff-3303adad40b1fa3c6b0a92ea1f4ad28c32bc84c52000e7a5e9fc8b37e05eaa85R2226-R2229
spanner/value.go
Outdated
// as Number (preserving precision) or float64 (risking loss). | ||
// Defaults to float64, call this method for lossless precision. | ||
// NOTE: This change affects all clients created by this library, both existing and future ones. | ||
func UseNumberWithJSONDecoderEncoder() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This API means that you can never disable it after having enabled it. I don't think we have a good reason for that restriction, or? I can imagine that making it possible to both enable and disable the option will make it easier for users to use it in their own tests to determine what the behavior is.
So I think that we should add an argument to the method (true/false), or add a separate method for disabling the option.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Allowing users to enable/disable the feature means they can run into situation where Client 1(supposed to run without Number) running in thread1 get impacted by some thread2 which uses another Client2 and call UseNumberWithJSONDecoderEncoder(), are we ok with it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That was already the case, however only in the direction default behavior => new behavior
. Now we allow this to happen both ways.
What I mean is that the original implementation also allowed the following:
- A client is created. This client is assumed to use the default Go encoding/decoding.
- The flag is set to true and a new client is created.
- Both the old client and the new client now use the new behavior.
- It was however not possible to go back to the default behavior.
5724768
to
a6d1e54
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks for being patient with my many small requests :-)
Fixes: #8669
Go client library uses native json library(encoding/json) for marshalling/unmarshalling for decoding JSON spanner types.
Since native package maps values to float64 it automatically rounds off the values.
Example
will decode the value as
0.39240506
.Similarly when reading
145688415796432520
will be decoded as145688415796432500
OR1.456884157964325e+17
With this change Go client library will parse the JSON number values to same precision as stored in Spanner database using jsoniter library using json.Number when UseNumberWithJSONDecoderEncoder() is called from the application
NOTE