fix: fix deadlock issue in ConnectionWorkerPool #1938

GaoleMeng · 2023-01-18T19:47:08Z

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
Ensure the tests and linter pass
Code coverage does not decrease (if any source code was changed)
Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> ☕️

If you write sample code, please follow the samples format.

prerequisite for multiplexing client

new stream name as a switch of destinationt

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

also fixed a tiny bug inside fake bigquery write impl for getting thre response from offset

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

…nt or not

possible the proto schema does not contain this field

double type.

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

...uerystorage/src/test/java/com/google/cloud/bigquery/storage/v1/ConnectionWorkerPoolTest.java

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

yirutang · 2023-01-20T01:10:11Z

...oud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/ConnectionWorker.java

+        Instant insertTime = requestWrapper.insertTime;
+        Instant currentTime = Instant.now();
+        long latencyToAdd = currentTime.toEpochMilli() - insertTime.toEpochMilli();
+        updateLatencyDistribution(latencyToAdd);


Let's add the error message for now from client lib at L686. So in case there is still a blockage on customer's case, we will have more info to investigate.

Do you mean the error message in response? I think that's already printed by Pablo's current code... I was planning to revert the change in this file once Pablo copy over this part

Yes, we want it to be in the release we hand over to customers, so if anything is going on, we have some logs.

let me clean this file up, Pablo has already copied the stats log, let's add error log directly

But won't that kind of log overwhelmed the log console?

yirutang · 2023-01-20T01:11:52Z

...oud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/ConnectionWorker.java

+    this.reportingThread.start();
+  }
+
+  private void reportSingleCurrentStatus() {


Is this too much logging? Is it going to be in the final code?

Not going to be in the final code, it's reported every 5 seconds

GaoleMeng and others added 30 commits September 13, 2022 01:58

feat: Split writer into connection worker and wrapper, this is a

5a63d95

prerequisite for multiplexing client

feat: add connection worker pool skeleton, used for multiplexing client

5a13302

Merge branch 'main' into main

0297204

feat: add Load api for connection worker for multiplexing client

8a81ad3

Merge remote-tracking branch 'upstream/main'

68fd040

Merge remote-tracking branch 'upstream/main'

3106dae

Merge branch 'googleapis:main' into main

5bf04e5

Merge branch 'main' of https://github.com/GaoleMeng/java-bigquerystorage

2fc7551

feat: add multiplexing support to connection worker. We will treat every

7a6d919

new stream name as a switch of destinationt

🦉 Updates from OwlBot post-processor

3ba7659

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Updates from OwlBot post-processor

f379a78

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Merge branch 'main' of https://github.com/GaoleMeng/java-bigquerystorage

9307776

🦉 Updates from OwlBot post-processor

de73013

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

feat: port the multiplexing client core algorithm and basic tests

19005a1

also fixed a tiny bug inside fake bigquery write impl for getting thre response from offset

Merge branch 'googleapis:main' into main

c5d14ba

🦉 Updates from OwlBot post-processor

644360a

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Merge branch 'googleapis:main' into main

3099d82

Merge branch 'googleapis:main' into main

e707dd6

Merge branch 'main' of https://github.com/GaoleMeng/java-bigquerystorage

9e7a8fa

Merge branch 'googleapis:main' into main

31f1755

feat: wire multiplexing connection pool to stream writer

44c36fc

feat: some fixes for multiplexing client

87a4036

Merge remote-tracking branch 'upstream/main'

c92ea1b

Merge branch 'googleapis:main' into main

019520c

feat: fix some todos, and reject the mixed behavior of passed in clie…

47893df

…nt or not

Merge remote-tracking branch 'upstream/main'

8bd4e6a

Merge remote-tracking branch 'upstream/main'

83409b0

Merge branch 'googleapis:main' into main

f7dd72d

Merge branch 'googleapis:main' into main

a48399f

feat: fix the bug that we may peek into the write_stream field but it's

6789bc9

possible the proto schema does not contain this field

GaoleMeng and others added 7 commits January 17, 2023 10:33

fix: Add precision truncation to the passed in value from JSON float and

14b0c12

double type.

🦉 Updates from OwlBot post-processor

0da0e4b

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Merge branch 'googleapis:main' into main

33d23ac

🦉 Updates from OwlBot post-processor

d2ee46e

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

modify the bom version

be6646e

🦉 Updates from OwlBot post-processor

62d8c41

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

fix deadlockissue in ConnectionWorkerPool

adf5f3f

GaoleMeng requested a review from a team January 18, 2023 19:47

GaoleMeng requested a review from a team as a code owner January 18, 2023 19:47

GaoleMeng requested a review from Neenu1995 January 18, 2023 19:47

product-auto-label bot added size: m Pull request size is medium. api: bigquerystorage Issues related to the googleapis/java-bigquerystorage API. labels Jan 18, 2023

Merge branch 'googleapis:main' into main

c1970ff

yirutang approved these changes Jan 19, 2023

View reviewed changes

...uerystorage/src/test/java/com/google/cloud/bigquery/storage/v1/ConnectionWorkerPoolTest.java Outdated Show resolved Hide resolved

GaoleMeng changed the title ~~[[preview only] fix deadlock issue in ConnectionWorkerPool~~ fix: fix deadlock issue in ConnectionWorkerPool Jan 19, 2023

fix: fix deadlock issue during close + append for multiplexing

3488df8

GaoleMeng added the owlbot:run Add this label to trigger the Owlbot post processor. label Jan 20, 2023

gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Jan 20, 2023

🦉 Updates from OwlBot post-processor

6a512e8

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

yirutang reviewed Jan 20, 2023

View reviewed changes

Merge branch 'googleapis:main' into main

05edc2f

GaoleMeng added the kokoro:run Add this label to force Kokoro to re-run the tests. label Jan 20, 2023

yoshi-kokoro removed the kokoro:run Add this label to force Kokoro to re-run the tests. label Jan 20, 2023

Merge branch 'googleapis:main' into main

7d3da74

GaoleMeng added the owlbot:run Add this label to trigger the Owlbot post processor. label Jan 20, 2023

gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Jan 20, 2023

GaoleMeng merged commit caf1e76 into googleapis:main Jan 20, 2023

release-please bot mentioned this pull request Jan 20, 2023

chore(main): release 2.28.3 #1950

Merged

Neenu1995 mentioned this pull request Jan 23, 2023

[DO-NOT-MERGE]fix: fix deadlock issue in ConnectionWorkerPool #1953

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: fix deadlock issue in ConnectionWorkerPool #1938

fix: fix deadlock issue in ConnectionWorkerPool #1938

GaoleMeng commented Jan 18, 2023

yirutang Jan 20, 2023

GaoleMeng Jan 20, 2023

yirutang Jan 20, 2023

GaoleMeng Jan 20, 2023

GaoleMeng Jan 20, 2023

yirutang Jan 20, 2023

GaoleMeng Jan 20, 2023

fix: fix deadlock issue in ConnectionWorkerPool #1938

fix: fix deadlock issue in ConnectionWorkerPool #1938

Conversation

GaoleMeng commented Jan 18, 2023

yirutang Jan 20, 2023

Choose a reason for hiding this comment

GaoleMeng Jan 20, 2023

Choose a reason for hiding this comment

yirutang Jan 20, 2023

Choose a reason for hiding this comment

GaoleMeng Jan 20, 2023

Choose a reason for hiding this comment

GaoleMeng Jan 20, 2023

Choose a reason for hiding this comment

yirutang Jan 20, 2023

Choose a reason for hiding this comment

GaoleMeng Jan 20, 2023

Choose a reason for hiding this comment