Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: add precision overwritten to 9 digit if the passed in JSON type is FLOAT or DOUBLE #1932

Merged
merged 61 commits into from
Jan 18, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
5a63d95
feat: Split writer into connection worker and wrapper, this is a
GaoleMeng Sep 9, 2022
5a13302
feat: add connection worker pool skeleton, used for multiplexing client
GaoleMeng Sep 13, 2022
0297204
Merge branch 'main' into main
GaoleMeng Sep 14, 2022
8a81ad3
feat: add Load api for connection worker for multiplexing client
GaoleMeng Sep 14, 2022
68fd040
Merge remote-tracking branch 'upstream/main'
GaoleMeng Sep 14, 2022
3106dae
Merge remote-tracking branch 'upstream/main'
GaoleMeng Sep 15, 2022
5bf04e5
Merge branch 'googleapis:main' into main
GaoleMeng Sep 15, 2022
2fc7551
Merge branch 'main' of https://github.com/GaoleMeng/java-bigquerystorage
GaoleMeng Sep 15, 2022
7a6d919
feat: add multiplexing support to connection worker. We will treat every
GaoleMeng Sep 15, 2022
3ba7659
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Sep 16, 2022
f379a78
Updates from OwlBot post-processor
gcf-owl-bot[bot] Sep 16, 2022
9307776
Merge branch 'main' of https://github.com/GaoleMeng/java-bigquerystorage
GaoleMeng Sep 16, 2022
de73013
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Sep 16, 2022
19005a1
feat: port the multiplexing client core algorithm and basic tests
GaoleMeng Sep 19, 2022
c5d14ba
Merge branch 'googleapis:main' into main
GaoleMeng Sep 19, 2022
644360a
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Sep 20, 2022
3099d82
Merge branch 'googleapis:main' into main
GaoleMeng Sep 20, 2022
e707dd6
Merge branch 'googleapis:main' into main
GaoleMeng Sep 20, 2022
9e7a8fa
Merge branch 'main' of https://github.com/GaoleMeng/java-bigquerystorage
GaoleMeng Sep 20, 2022
31f1755
Merge branch 'googleapis:main' into main
GaoleMeng Sep 20, 2022
44c36fc
feat: wire multiplexing connection pool to stream writer
GaoleMeng Sep 20, 2022
87a4036
feat: some fixes for multiplexing client
GaoleMeng Sep 23, 2022
c92ea1b
Merge remote-tracking branch 'upstream/main'
GaoleMeng Sep 23, 2022
019520c
Merge branch 'googleapis:main' into main
GaoleMeng Sep 26, 2022
47893df
feat: fix some todos, and reject the mixed behavior of passed in clie…
GaoleMeng Sep 27, 2022
8bd4e6a
Merge remote-tracking branch 'upstream/main'
GaoleMeng Sep 27, 2022
83409b0
Merge remote-tracking branch 'upstream/main'
GaoleMeng Sep 27, 2022
f7dd72d
Merge branch 'googleapis:main' into main
GaoleMeng Sep 27, 2022
a48399f
Merge branch 'googleapis:main' into main
GaoleMeng Sep 29, 2022
6789bc9
feat: fix the bug that we may peek into the write_stream field but it's
GaoleMeng Sep 29, 2022
46b4e6c
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Sep 29, 2022
dfd4dd9
Merge branch 'googleapis:main' into main
GaoleMeng Sep 29, 2022
d68ae70
feat: fix the bug that we may peek into the write_stream field but it's
GaoleMeng Sep 29, 2022
2983fe9
Merge branch 'main' of https://github.com/GaoleMeng/java-bigquerystorage
GaoleMeng Sep 29, 2022
d406256
Merge branch 'googleapis:main' into main
GaoleMeng Oct 13, 2022
22e9e07
feat: add getInflightWaitSeconds implementation
GaoleMeng Oct 13, 2022
fdb4e1c
Merge branch 'googleapis:main' into main
GaoleMeng Oct 21, 2022
0469474
Merge branch 'googleapis:main' into main
GaoleMeng Nov 2, 2022
d1b7740
feat: Add schema comparision in connection loop to ensure schema upda…
GaoleMeng Nov 3, 2022
e4cd529
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Nov 4, 2022
74ff1c4
Merge branch 'googleapis:main' into main
GaoleMeng Nov 4, 2022
762f49e
feat: add schema update support to multiplexing
GaoleMeng Nov 5, 2022
de456c2
Merge branch 'googleapis:main' into main
GaoleMeng Nov 11, 2022
c2f6edc
Merge branch 'googleapis:main' into main
GaoleMeng Nov 15, 2022
2487227
fix: fix windows build bug: windows Instant resolution is different with
GaoleMeng Nov 15, 2022
084d6d1
fix: fix another failing tests for windows build
GaoleMeng Nov 16, 2022
89c9701
Merge branch 'main' of https://github.com/GaoleMeng/java-bigquerystorage
GaoleMeng Nov 16, 2022
8441518
fix: fix another test failure for Windows build
GaoleMeng Nov 16, 2022
d249add
Merge branch 'googleapis:main' into main
GaoleMeng Nov 30, 2022
83aa7ff
feat: Change new thread for each retry to be a thread pool to avoid
GaoleMeng Nov 30, 2022
92a9c36
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Nov 30, 2022
a713a52
Merge branch 'googleapis:main' into main
GaoleMeng Nov 30, 2022
a042d5c
fix: add back the background executor provider that's accidentally
GaoleMeng Nov 30, 2022
53f4ec8
feat: throw error when use connection pool for explicit stream
GaoleMeng Dec 2, 2022
c494d8b
Merge branch 'googleapis:main' into main
GaoleMeng Dec 20, 2022
14b0c12
fix: Add precision truncation to the passed in value from JSON float and
GaoleMeng Jan 17, 2023
0da0e4b
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Jan 17, 2023
33d23ac
Merge branch 'googleapis:main' into main
GaoleMeng Jan 17, 2023
d2ee46e
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Jan 17, 2023
be6646e
modify the bom version
GaoleMeng Jan 17, 2023
62d8c41
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Jan 17, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ If you are using Maven without BOM, add this to your dependencies:
If you are using Gradle 5.x or later, add this to your dependencies:

```Groovy
implementation platform('com.google.cloud:libraries-bom:26.3.0')
implementation platform('com.google.cloud:libraries-bom:26.4.0')

implementation 'com.google.cloud:google-cloud-bigquerystorage'
```
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
import com.google.protobuf.Message;
import com.google.protobuf.UninitializedMessageException;
import java.math.BigDecimal;
import java.math.RoundingMode;
import java.time.LocalDate;
import java.util.List;
import java.util.logging.Logger;
Expand All @@ -49,6 +50,7 @@
*/
public class JsonToProtoMessage {
private static final Logger LOG = Logger.getLogger(JsonToProtoMessage.class.getName());
private static int NUMERIC_SCALE = 9;
private static ImmutableMap<FieldDescriptor.Type, String> FieldTypeToDebugMessage =
new ImmutableMap.Builder<FieldDescriptor.Type, String>()
.put(FieldDescriptor.Type.BOOL, "boolean")
Expand Down Expand Up @@ -315,10 +317,15 @@ private static void fillField(
new BigDecimal(((Number) val).longValue())));
return;
} else if (val instanceof Float || val instanceof Double) {
// In JSON, the precision passed in is machine dependent. We should round the number
// before passing to backend.
BigDecimal bigDecimal = new BigDecimal(String.valueOf(val));
if (bigDecimal.scale() > 9) {
bigDecimal = bigDecimal.setScale(NUMERIC_SCALE, RoundingMode.HALF_UP);
}
protoMsg.setField(
fieldDescriptor,
BigDecimalByteStringEncoder.encodeToNumericByteString(
new BigDecimal(String.valueOf(val))));
BigDecimalByteStringEncoder.encodeToNumericByteString(bigDecimal));
return;
} else if (val instanceof BigDecimal) {
protoMsg.setField(
Expand Down Expand Up @@ -559,10 +566,13 @@ private static void fillRepeatedField(
new BigDecimal(((Number) val).longValue())));
added = true;
} else if (val instanceof Float || val instanceof Double) {
BigDecimal bigDecimal = new BigDecimal(String.valueOf(val));
if (bigDecimal.scale() > 9) {
bigDecimal = bigDecimal.setScale(NUMERIC_SCALE, RoundingMode.HALF_UP);
}
protoMsg.addRepeatedField(
fieldDescriptor,
BigDecimalByteStringEncoder.encodeToNumericByteString(
new BigDecimal(String.valueOf(val))));
BigDecimalByteStringEncoder.encodeToNumericByteString(bigDecimal));
added = true;
} else if (val instanceof BigDecimal) {
protoMsg.addRepeatedField(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@

import com.google.cloud.bigquery.storage.test.JsonTest.*;
import com.google.cloud.bigquery.storage.test.SchemaTest.*;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import com.google.protobuf.ByteString;
import com.google.protobuf.Descriptors.Descriptor;
Expand Down Expand Up @@ -685,6 +686,60 @@ public void testDouble() throws Exception {
assertEquals(expectedProto, protoMsg);
}

@Test
public void testDoubleHighPrecision() throws Exception {
TableSchema tableSchema =
TableSchema.newBuilder()
.addFields(
TableFieldSchema.newBuilder()
.setName("numeric")
.setType(TableFieldSchema.Type.NUMERIC)
.build())
.build();
TestNumeric expectedProto =
TestNumeric.newBuilder()
.setNumeric(
BigDecimalByteStringEncoder.encodeToNumericByteString(
new BigDecimal("3.400500513")))
.build();
JSONObject json = new JSONObject();
json.put("numeric", 3.400500512978076);
DynamicMessage protoMsg =
JsonToProtoMessage.convertJsonToProtoMessage(
TestNumeric.getDescriptor(), tableSchema, json);
assertEquals(expectedProto, protoMsg);
}

@Test
public void testDoubleHighPrecision_RepeatedField() throws Exception {
TableSchema tableSchema =
TableSchema.newBuilder()
.addFields(
0,
TableFieldSchema.newBuilder()
.setName("bignumeric")
.setType(TableFieldSchema.Type.NUMERIC)
.setMode(TableFieldSchema.Mode.REPEATED)
.build())
.build();
TestBignumeric expectedProto =
TestBignumeric.newBuilder()
.addBignumeric(
BigDecimalByteStringEncoder.encodeToNumericByteString(
new BigDecimal("3.400500513")))
.addBignumeric(
BigDecimalByteStringEncoder.encodeToNumericByteString(new BigDecimal("0.1")))
.addBignumeric(
BigDecimalByteStringEncoder.encodeToNumericByteString(new BigDecimal("0.12")))
.build();
JSONObject json = new JSONObject();
json.put("bignumeric", ImmutableList.of(3.400500512978076, 0.10000000000055, 0.12));
DynamicMessage protoMsg =
JsonToProtoMessage.convertJsonToProtoMessage(
TestBignumeric.getDescriptor(), tableSchema, json);
assertEquals(expectedProto, protoMsg);
}

@Test
public void testTimestamp() throws Exception {
TableSchema tableSchema =
Expand Down