fix: remove unrecoverable connection from connection pool during multiplexing #1967

GaoleMeng · 2023-01-31T19:36:19Z

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
Ensure the tests and linter pass
Code coverage does not decrease (if any source code was changed)
Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> ☕️

If you write sample code, please follow the samples format.

prerequisite for multiplexing client

new stream name as a switch of destinationt

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

also fixed a tiny bug inside fake bigquery write impl for getting thre response from offset

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

…nt or not

possible the proto schema does not contain this field

case

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

…tream name

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

...oud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/ConnectionWorker.java

yirutang · 2023-01-31T19:58:14Z

...bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/ConnectionWorkerPool.java

+                }
+                // Before search for the next connection to attach, clear the finalized connections
+                // first so that they will not be selected.
+                clearFinalizedConnectionWorker();


Why do we need a global remove, is it OK just to have a local remove?

The reason is we want to prevent the search algorithm to hit the connection with unrecoverable state?

The global remove is super quick as it won't need close

…iplexing (googleapis#1967) * feat: Split writer into connection worker and wrapper, this is a prerequisite for multiplexing client * feat: add connection worker pool skeleton, used for multiplexing client * feat: add Load api for connection worker for multiplexing client * feat: add multiplexing support to connection worker. We will treat every new stream name as a switch of destinationt * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: port the multiplexing client core algorithm and basic tests also fixed a tiny bug inside fake bigquery write impl for getting thre response from offset * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: wire multiplexing connection pool to stream writer * feat: some fixes for multiplexing client * feat: fix some todos, and reject the mixed behavior of passed in client or not * feat: fix the bug that we may peek into the write_stream field but it's possible the proto schema does not contain this field * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: fix the bug that we may peek into the write_stream field but it's possible the proto schema does not contain this field * feat: add getInflightWaitSeconds implementation * feat: Add schema comparision in connection loop to ensure schema update for the same stream name can be notified * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: add schema update support to multiplexing * fix: fix windows build bug: windows Instant resolution is different with linux * fix: fix another failing tests for windows build * fix: fix another test failure for Windows build * feat: Change new thread for each retry to be a thread pool to avoid create/tear down too much threads if lots of retries happens * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * fix: add back the background executor provider that's accidentally removed * feat: throw error when use connection pool for explicit stream * fix: Add precision truncation to the passed in value from JSON float and double type. * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * modify the bom version * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * fix deadlockissue in ConnectionWorkerPool * fix: fix deadlock issue during close + append for multiplexing * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * fix: fix one potential root cause of deadlock issue for non-multiplexing case * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Add timeout to inflight queue waiting, and also add some extra log * feat: allow java client lib handle switch table schema for the same stream name * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

* fix: at connection level, retry for internal errors * . * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * . * . * . * . * . * feat: allow java client to handle schema change during same stream name (#1964) * feat: Split writer into connection worker and wrapper, this is a prerequisite for multiplexing client * feat: add connection worker pool skeleton, used for multiplexing client * feat: add Load api for connection worker for multiplexing client * feat: add multiplexing support to connection worker. We will treat every new stream name as a switch of destinationt * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: port the multiplexing client core algorithm and basic tests also fixed a tiny bug inside fake bigquery write impl for getting thre response from offset * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: wire multiplexing connection pool to stream writer * feat: some fixes for multiplexing client * feat: fix some todos, and reject the mixed behavior of passed in client or not * feat: fix the bug that we may peek into the write_stream field but it's possible the proto schema does not contain this field * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: fix the bug that we may peek into the write_stream field but it's possible the proto schema does not contain this field * feat: add getInflightWaitSeconds implementation * feat: Add schema comparision in connection loop to ensure schema update for the same stream name can be notified * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: add schema update support to multiplexing * fix: fix windows build bug: windows Instant resolution is different with linux * fix: fix another failing tests for windows build * fix: fix another test failure for Windows build * feat: Change new thread for each retry to be a thread pool to avoid create/tear down too much threads if lots of retries happens * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * fix: add back the background executor provider that's accidentally removed * feat: throw error when use connection pool for explicit stream * fix: Add precision truncation to the passed in value from JSON float and double type. * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * modify the bom version * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * fix deadlockissue in ConnectionWorkerPool * fix: fix deadlock issue during close + append for multiplexing * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * fix: fix one potential root cause of deadlock issue for non-multiplexing case * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Add timeout to inflight queue waiting, and also add some extra log * feat: allow java client lib handle switch table schema for the same stream name * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> * fix: remove unrecoverable connection from connection pool during multiplexing (#1967) * feat: Split writer into connection worker and wrapper, this is a prerequisite for multiplexing client * feat: add connection worker pool skeleton, used for multiplexing client * feat: add Load api for connection worker for multiplexing client * feat: add multiplexing support to connection worker. We will treat every new stream name as a switch of destinationt * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: port the multiplexing client core algorithm and basic tests also fixed a tiny bug inside fake bigquery write impl for getting thre response from offset * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: wire multiplexing connection pool to stream writer * feat: some fixes for multiplexing client * feat: fix some todos, and reject the mixed behavior of passed in client or not * feat: fix the bug that we may peek into the write_stream field but it's possible the proto schema does not contain this field * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: fix the bug that we may peek into the write_stream field but it's possible the proto schema does not contain this field * feat: add getInflightWaitSeconds implementation * feat: Add schema comparision in connection loop to ensure schema update for the same stream name can be notified * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: add schema update support to multiplexing * fix: fix windows build bug: windows Instant resolution is different with linux * fix: fix another failing tests for windows build * fix: fix another test failure for Windows build * feat: Change new thread for each retry to be a thread pool to avoid create/tear down too much threads if lots of retries happens * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * fix: add back the background executor provider that's accidentally removed * feat: throw error when use connection pool for explicit stream * fix: Add precision truncation to the passed in value from JSON float and double type. * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * modify the bom version * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * fix deadlockissue in ConnectionWorkerPool * fix: fix deadlock issue during close + append for multiplexing * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * fix: fix one potential root cause of deadlock issue for non-multiplexing case * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Add timeout to inflight queue waiting, and also add some extra log * feat: allow java client lib handle switch table schema for the same stream name * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Gaole Meng <[email protected]>

GaoleMeng and others added 30 commits September 13, 2022 01:58

feat: Split writer into connection worker and wrapper, this is a

5a63d95

prerequisite for multiplexing client

feat: add connection worker pool skeleton, used for multiplexing client

5a13302

Merge branch 'main' into main

0297204

feat: add Load api for connection worker for multiplexing client

8a81ad3

Merge remote-tracking branch 'upstream/main'

68fd040

Merge remote-tracking branch 'upstream/main'

3106dae

Merge branch 'googleapis:main' into main

5bf04e5

Merge branch 'main' of https://github.com/GaoleMeng/java-bigquerystorage

2fc7551

feat: add multiplexing support to connection worker. We will treat every

7a6d919

new stream name as a switch of destinationt

🦉 Updates from OwlBot post-processor

3ba7659

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Updates from OwlBot post-processor

f379a78

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Merge branch 'main' of https://github.com/GaoleMeng/java-bigquerystorage

9307776

🦉 Updates from OwlBot post-processor

de73013

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

feat: port the multiplexing client core algorithm and basic tests

19005a1

also fixed a tiny bug inside fake bigquery write impl for getting thre response from offset

Merge branch 'googleapis:main' into main

c5d14ba

🦉 Updates from OwlBot post-processor

644360a

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Merge branch 'googleapis:main' into main

3099d82

Merge branch 'googleapis:main' into main

e707dd6

Merge branch 'main' of https://github.com/GaoleMeng/java-bigquerystorage

9e7a8fa

Merge branch 'googleapis:main' into main

31f1755

feat: wire multiplexing connection pool to stream writer

44c36fc

feat: some fixes for multiplexing client

87a4036

Merge remote-tracking branch 'upstream/main'

c92ea1b

Merge branch 'googleapis:main' into main

019520c

feat: fix some todos, and reject the mixed behavior of passed in clie…

47893df

…nt or not

Merge remote-tracking branch 'upstream/main'

8bd4e6a

Merge remote-tracking branch 'upstream/main'

83409b0

Merge branch 'googleapis:main' into main

f7dd72d

Merge branch 'googleapis:main' into main

a48399f

feat: fix the bug that we may peek into the write_stream field but it's

6789bc9

possible the proto schema does not contain this field

GaoleMeng and others added 12 commits January 20, 2023 11:15

Merge branch 'googleapis:main' into main

7d3da74

Merge branch 'googleapis:main' into main

ecf6807

Merge branch 'googleapis:main' into main

057dab9

fix: fix one potential root cause of deadlock issue for non-multiplexing

5db46a2

case

🦉 Updates from OwlBot post-processor

32e9d33

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Merge branch 'googleapis:main' into main

183941a

Add timeout to inflight queue waiting, and also add some extra log

f93f89e

Merge branch 'googleapis:main' into main

e2ec7e7

feat: allow java client lib handle switch table schema for the same s…

1be6ab4

…tream name

🦉 Updates from OwlBot post-processor

544f063

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Merge branch 'googleapis:main' into main

3cf5482

Merge branch 'googleapis:main' into main

d9da007

GaoleMeng requested review from a team and shollyman January 31, 2023 19:36

product-auto-label bot added the size: m Pull request size is medium. label Jan 31, 2023

GaoleMeng requested a review from yirutang January 31, 2023 19:36

product-auto-label bot added the api: bigquerystorage Issues related to the googleapis/java-bigquerystorage API. label Jan 31, 2023

GaoleMeng requested a review from a team as a code owner January 31, 2023 19:40

yirutang reviewed Jan 31, 2023

View reviewed changes

Merge branch 'main' of https://github.com/GaoleMeng/java-bigquerystorage

4626857

shollyman requested a review from Neenu1995 January 31, 2023 20:39

yirutang approved these changes Jan 31, 2023

View reviewed changes

GaoleMeng added the owlbot:run Add this label to trigger the Owlbot post processor. label Jan 31, 2023

gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Jan 31, 2023

Neenu1995 approved these changes Jan 31, 2023

View reviewed changes

GaoleMeng merged commit 091dddb into googleapis:main Jan 31, 2023

release-please bot mentioned this pull request Feb 1, 2023

chore(main): release 2.29.0 #1969

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: remove unrecoverable connection from connection pool during multiplexing #1967

fix: remove unrecoverable connection from connection pool during multiplexing #1967

GaoleMeng commented Jan 31, 2023

yirutang Jan 31, 2023

GaoleMeng Jan 31, 2023

GaoleMeng Jan 31, 2023

fix: remove unrecoverable connection from connection pool during multiplexing #1967

fix: remove unrecoverable connection from connection pool during multiplexing #1967

Conversation

GaoleMeng commented Jan 31, 2023

yirutang Jan 31, 2023

Choose a reason for hiding this comment

GaoleMeng Jan 31, 2023

Choose a reason for hiding this comment

GaoleMeng Jan 31, 2023

Choose a reason for hiding this comment