Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: fix windows build failure by using nanoSeconds instead of Instant for better accuracy. #1887

Merged
merged 48 commits into from
Nov 16, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
5a63d95
feat: Split writer into connection worker and wrapper, this is a
GaoleMeng Sep 9, 2022
5a13302
feat: add connection worker pool skeleton, used for multiplexing client
GaoleMeng Sep 13, 2022
0297204
Merge branch 'main' into main
GaoleMeng Sep 14, 2022
8a81ad3
feat: add Load api for connection worker for multiplexing client
GaoleMeng Sep 14, 2022
68fd040
Merge remote-tracking branch 'upstream/main'
GaoleMeng Sep 14, 2022
3106dae
Merge remote-tracking branch 'upstream/main'
GaoleMeng Sep 15, 2022
5bf04e5
Merge branch 'googleapis:main' into main
GaoleMeng Sep 15, 2022
2fc7551
Merge branch 'main' of https://github.com/GaoleMeng/java-bigquerystorage
GaoleMeng Sep 15, 2022
7a6d919
feat: add multiplexing support to connection worker. We will treat every
GaoleMeng Sep 15, 2022
3ba7659
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Sep 16, 2022
f379a78
Updates from OwlBot post-processor
gcf-owl-bot[bot] Sep 16, 2022
9307776
Merge branch 'main' of https://github.com/GaoleMeng/java-bigquerystorage
GaoleMeng Sep 16, 2022
de73013
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Sep 16, 2022
19005a1
feat: port the multiplexing client core algorithm and basic tests
GaoleMeng Sep 19, 2022
c5d14ba
Merge branch 'googleapis:main' into main
GaoleMeng Sep 19, 2022
644360a
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Sep 20, 2022
3099d82
Merge branch 'googleapis:main' into main
GaoleMeng Sep 20, 2022
e707dd6
Merge branch 'googleapis:main' into main
GaoleMeng Sep 20, 2022
9e7a8fa
Merge branch 'main' of https://github.com/GaoleMeng/java-bigquerystorage
GaoleMeng Sep 20, 2022
31f1755
Merge branch 'googleapis:main' into main
GaoleMeng Sep 20, 2022
44c36fc
feat: wire multiplexing connection pool to stream writer
GaoleMeng Sep 20, 2022
87a4036
feat: some fixes for multiplexing client
GaoleMeng Sep 23, 2022
c92ea1b
Merge remote-tracking branch 'upstream/main'
GaoleMeng Sep 23, 2022
019520c
Merge branch 'googleapis:main' into main
GaoleMeng Sep 26, 2022
47893df
feat: fix some todos, and reject the mixed behavior of passed in clie…
GaoleMeng Sep 27, 2022
8bd4e6a
Merge remote-tracking branch 'upstream/main'
GaoleMeng Sep 27, 2022
83409b0
Merge remote-tracking branch 'upstream/main'
GaoleMeng Sep 27, 2022
f7dd72d
Merge branch 'googleapis:main' into main
GaoleMeng Sep 27, 2022
a48399f
Merge branch 'googleapis:main' into main
GaoleMeng Sep 29, 2022
6789bc9
feat: fix the bug that we may peek into the write_stream field but it's
GaoleMeng Sep 29, 2022
46b4e6c
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Sep 29, 2022
dfd4dd9
Merge branch 'googleapis:main' into main
GaoleMeng Sep 29, 2022
d68ae70
feat: fix the bug that we may peek into the write_stream field but it's
GaoleMeng Sep 29, 2022
2983fe9
Merge branch 'main' of https://github.com/GaoleMeng/java-bigquerystorage
GaoleMeng Sep 29, 2022
d406256
Merge branch 'googleapis:main' into main
GaoleMeng Oct 13, 2022
22e9e07
feat: add getInflightWaitSeconds implementation
GaoleMeng Oct 13, 2022
fdb4e1c
Merge branch 'googleapis:main' into main
GaoleMeng Oct 21, 2022
0469474
Merge branch 'googleapis:main' into main
GaoleMeng Nov 2, 2022
d1b7740
feat: Add schema comparision in connection loop to ensure schema upda…
GaoleMeng Nov 3, 2022
e4cd529
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Nov 4, 2022
74ff1c4
Merge branch 'googleapis:main' into main
GaoleMeng Nov 4, 2022
762f49e
feat: add schema update support to multiplexing
GaoleMeng Nov 5, 2022
de456c2
Merge branch 'googleapis:main' into main
GaoleMeng Nov 11, 2022
c2f6edc
Merge branch 'googleapis:main' into main
GaoleMeng Nov 15, 2022
2487227
fix: fix windows build bug: windows Instant resolution is different with
GaoleMeng Nov 15, 2022
084d6d1
fix: fix another failing tests for windows build
GaoleMeng Nov 16, 2022
89c9701
Merge branch 'main' of https://github.com/GaoleMeng/java-bigquerystorage
GaoleMeng Nov 16, 2022
8441518
fix: fix another test failure for Windows build
GaoleMeng Nov 16, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,6 @@
import io.grpc.Status.Code;
import io.grpc.StatusRuntimeException;
import java.io.IOException;
import java.time.Instant;
import java.util.Comparator;
import java.util.Deque;
import java.util.HashMap;
Expand Down Expand Up @@ -610,7 +609,7 @@ private void requestCallback(AppendRowsResponse response) {
this.lock.lock();
if (response.hasUpdatedSchema()) {
this.updatedSchema =
TableSchemaAndTimestamp.create(Instant.now(), response.getUpdatedSchema());
TableSchemaAndTimestamp.create(System.nanoTime(), response.getUpdatedSchema());
}
try {
// Had a successful connection with at least one result, reset retries.
Expand Down Expand Up @@ -824,12 +823,12 @@ public static void setOverwhelmedCountsThreshold(double newThreshold) {
@AutoValue
abstract static class TableSchemaAndTimestamp {
// Shows the timestamp updated schema is reported from response
abstract Instant updateTimeStamp();
abstract long updateTimeStamp();

// The updated schema returned from server.
abstract TableSchema updatedSchema();

static TableSchemaAndTimestamp create(Instant updateTimeStamp, TableSchema updatedSchema) {
static TableSchemaAndTimestamp create(long updateTimeStamp, TableSchema updatedSchema) {
return new AutoValue_ConnectionWorker_TableSchemaAndTimestamp(updateTimeStamp, updatedSchema);
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,6 @@
import com.google.common.collect.ImmutableList;
import com.google.common.util.concurrent.MoreExecutors;
import java.io.IOException;
import java.time.Instant;
import java.util.Collections;
import java.util.Comparator;
import java.util.HashSet;
Expand Down Expand Up @@ -268,7 +267,7 @@ public ApiFuture<AppendRowsResponse> append(
if (response.getWriteStream() != "" && response.hasUpdatedSchema()) {
tableNameToUpdatedSchema.put(
response.getWriteStream(),
TableSchemaAndTimestamp.create(Instant.now(), response.getUpdatedSchema()));
TableSchemaAndTimestamp.create(System.nanoTime(), response.getUpdatedSchema()));
}
return response;
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,6 @@
import io.grpc.Status.Code;
import io.grpc.StatusRuntimeException;
import java.io.IOException;
import java.time.Instant;
import java.util.Map;
import java.util.Objects;
import java.util.UUID;
Expand Down Expand Up @@ -88,7 +87,7 @@ public class StreamWriter implements AutoCloseable {
new ConcurrentHashMap<>();

/** Creation timestamp of this streamwriter */
private final Instant creationTimestamp;
private final long creationTimestamp;

/** The maximum size of one request. Defined by the API. */
public static long getApiMaxRequestBytes() {
Expand Down Expand Up @@ -260,7 +259,7 @@ private StreamWriter(Builder builder) throws IOException {
client.close();
}
}
this.creationTimestamp = Instant.now();
this.creationTimestamp = System.nanoTime();
}

@VisibleForTesting
Expand Down Expand Up @@ -414,12 +413,12 @@ public synchronized TableSchema getUpdatedSchema() {
if (tableSchemaAndTimestamp == null) {
return null;
}
return creationTimestamp.compareTo(tableSchemaAndTimestamp.updateTimeStamp()) < 0
return creationTimestamp < tableSchemaAndTimestamp.updateTimeStamp()
? tableSchemaAndTimestamp.updatedSchema()
: null;
}

Instant getCreationTimestamp() {
long getCreationTimestamp() {
return creationTimestamp;
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -530,8 +530,6 @@ public void run() throws Throwable {
public void testSimpleSchemaUpdate() throws Exception {
try (JsonStreamWriter writer =
getTestJsonStreamWriterBuilder(TEST_STREAM, TABLE_SCHEMA).build()) {
// Sleep for a short period to make sure the creation timestamp is older.
Sleeper.DEFAULT.sleep(200);
testBigQueryWrite.addResponse(
AppendRowsResponse.newBuilder()
.setAppendResult(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@
import static org.junit.Assert.assertThrows;
import static org.junit.Assert.assertTrue;

import com.google.api.client.util.Sleeper;
import com.google.api.core.ApiFuture;
import com.google.api.gax.batching.FlowController;
import com.google.api.gax.core.NoCredentialsProvider;
Expand Down Expand Up @@ -310,8 +309,6 @@ private void testUpdatedSchemaFetch(boolean enableMultiplexing)
AppendRowsResponse response =
writer.append(createProtoRows(new String[] {String.valueOf(0)}), 0).get();
assertEquals(writer.getUpdatedSchema(), UPDATED_TABLE_SCHEMA);
// Sleep for a short period to make sure the creation timestamp is older.
Sleeper.DEFAULT.sleep(200);

// Create another writer, although it's the same stream name but the time stamp is newer, thus
// the old updated schema won't get returned.
Expand Down