feat: wire connection pool to stream writer without implementing updated schema #1790

GaoleMeng · 2022-09-20T21:45:29Z

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
Ensure the tests and linter pass
Code coverage does not decrease (if any source code was changed)
Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> ☕️

If you write sample code, please follow the samples format.

prerequisite for multiplexing client

new stream name as a switch of destinationt

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

also fixed a tiny bug inside fake bigquery write impl for getting thre response from offset

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

yirutang · 2022-09-20T22:20:49Z

...oud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1/StreamWriterTest.java

    return StreamWriter.newBuilder(TEST_STREAM, client)
        .setWriterSchema(createProtoSchema())
        .setTraceId(TEST_TRACE_ID)
+        .setConnectionMode(connectionMode)


Let's call it enableConnectionPool.

yirutang · 2022-09-20T22:25:12Z

...oud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1/StreamWriterTest.java

@@ -703,6 +710,16 @@ public void testWriterId()
    Assert.assertNotEquals(writer1.getWriterId(), writer2.getWriterId());
  }

+  @Test
+  public void testInitialization_operationKind() throws Exception {


Do we need to duplicate all the tests into these two dimensions?

Let's do this in a separate CL, my plan is to duplicate some of them

yirutang · 2022-09-20T22:26:04Z

...e-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/StreamWriter.java

+   * Static map from {@link ConnectionPoolKey} to connection pool. Note this map is static to be
+   * shared by every stream writer in the same process.
+   */
+  private static final Map<ConnectionPoolKey, ConnectionWorkerPool> connectionPoolMap =


I thought we talked about moving this pool map onto ConnectionWorkerPool?

I think putting here will make the structure a lot easier to understand, otherwise we need to create a separate class e.g. PoolCollection to store the mapping from poolkey to connection pool

yirutang · 2022-09-20T22:27:00Z

...e-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/StreamWriter.java

+   */
+  @AutoValue
+  abstract static class ConnectionPoolKey {
+    abstract BigQueryWriteClient client();


Why the pool would contain only the BigQueryWriteClient? Should it be keyed by location(region)?

Changed to location

yirutang · 2022-09-20T22:30:22Z

...e-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/StreamWriter.java

+                          builder.traceId,
+                          client,
+                          ownsBigQueryWriteClient)));
+      if (!Objects.equals(


Let's also check limitExceedBehavior and ownsBigQueryWriteClient.

Done, I think for ownsBigQueryWriteClient let's check the client instead

GaoleMeng and others added 20 commits September 13, 2022 01:58

feat: Split writer into connection worker and wrapper, this is a

5a63d95

prerequisite for multiplexing client

feat: add connection worker pool skeleton, used for multiplexing client

5a13302

Merge branch 'main' into main

0297204

feat: add Load api for connection worker for multiplexing client

8a81ad3

Merge remote-tracking branch 'upstream/main'

68fd040

Merge remote-tracking branch 'upstream/main'

3106dae

Merge branch 'googleapis:main' into main

5bf04e5

Merge branch 'main' of https://github.com/GaoleMeng/java-bigquerystorage

2fc7551

feat: add multiplexing support to connection worker. We will treat every

7a6d919

new stream name as a switch of destinationt

🦉 Updates from OwlBot post-processor

3ba7659

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Updates from OwlBot post-processor

f379a78

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Merge branch 'main' of https://github.com/GaoleMeng/java-bigquerystorage

9307776

🦉 Updates from OwlBot post-processor

de73013

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

feat: port the multiplexing client core algorithm and basic tests

19005a1

also fixed a tiny bug inside fake bigquery write impl for getting thre response from offset

Merge branch 'googleapis:main' into main

c5d14ba

🦉 Updates from OwlBot post-processor

644360a

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Merge branch 'googleapis:main' into main

3099d82

Merge branch 'googleapis:main' into main

e707dd6

Merge branch 'main' of https://github.com/GaoleMeng/java-bigquerystorage

9e7a8fa

Merge branch 'googleapis:main' into main

31f1755

GaoleMeng requested review from a team and alvarowolfx September 20, 2022 21:45

product-auto-label bot added size: m Pull request size is medium. api: bigquerystorage Issues related to the googleapis/java-bigquerystorage API. labels Sep 20, 2022

GaoleMeng requested a review from yirutang September 20, 2022 21:46

product-auto-label bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Sep 20, 2022

yirutang reviewed Sep 20, 2022

View reviewed changes

GaoleMeng added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 20, 2022

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 20, 2022

feat: wire multiplexing connection pool to stream writer

44c36fc

yirutang approved these changes Sep 21, 2022

View reviewed changes

GaoleMeng added the owlbot:run Add this label to trigger the Owlbot post processor. label Sep 21, 2022

gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Sep 21, 2022

GaoleMeng merged commit 3eb1475 into googleapis:main Sep 21, 2022

release-please bot mentioned this pull request Sep 21, 2022

chore(main): release 2.22.0 #1785

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: wire connection pool to stream writer without implementing updated schema #1790

feat: wire connection pool to stream writer without implementing updated schema #1790

GaoleMeng commented Sep 20, 2022

yirutang Sep 20, 2022

GaoleMeng Sep 20, 2022

yirutang Sep 20, 2022

GaoleMeng Sep 20, 2022

yirutang Sep 20, 2022

GaoleMeng Sep 20, 2022

yirutang Sep 20, 2022

GaoleMeng Sep 20, 2022

yirutang Sep 20, 2022

GaoleMeng Sep 20, 2022

feat: wire connection pool to stream writer without implementing updated schema #1790

feat: wire connection pool to stream writer without implementing updated schema #1790

Conversation

GaoleMeng commented Sep 20, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment