Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Add unit test for concurrent issues we worried about, and fix some locking issues #854

Merged
merged 40 commits into from
Mar 2, 2021

Conversation

yirutang
Copy link
Contributor

@yirutang yirutang commented Feb 19, 2021

  • Improve FakeBigQueryService so that we can remove wait code in test.
  • Add unit tests for:
    Have something in inflight queue and we failed and then shutdown.
    Shutdown while append is going on.
    Shutdown with stuff both in batch leftover and inflight queue.
    Additional Batch test to increase code coverage.
  • Fix the locks in:
    Merge AppendAndRefreshAppendLock and MessageBatchLock into one.
    Release lock while we are waiting on waiter.
    Change shutdown and setAlarm to be normal variable since they are all protected by appendAndRefreshAppend lock.
    Make sure inflight requests are flushed after a stream error (on shutdown)

@yirutang yirutang requested review from a team and steffnay February 19, 2021 01:54
@product-auto-label product-auto-label bot added the api: bigquerystorage Issues related to the googleapis/java-bigquerystorage API. label Feb 19, 2021
@google-cla google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Feb 19, 2021
@yirutang yirutang marked this pull request as draft February 19, 2021 01:59
@yirutang yirutang marked this pull request as ready for review February 19, 2021 10:50
@yirutang yirutang changed the title fix: Add unit test for concurrent issues we worried about, and fix some locking issues. fix: Add unit test for concurrent issues we worried about, and fix some locking issues Feb 19, 2021
@yirutang yirutang marked this pull request as draft February 19, 2021 10:56
@yirutang yirutang requested a review from yayi-google February 19, 2021 17:08
@yirutang yirutang marked this pull request as ready for review February 19, 2021 17:40
@stephaniewang526 stephaniewang526 added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Feb 19, 2021
@yirutang
Copy link
Contributor Author

yirutang commented Feb 22, 2021

Having issues running Yi's stress test.

@stephaniewang526 stephaniewang526 added kokoro:force-run Add this label to force Kokoro to re-run the tests. and removed kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Feb 22, 2021
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Feb 22, 2021
@yayi-google
Copy link
Contributor

yayi-google commented Feb 22, 2021 via email

@yirutang yirutang requested a review from a team February 23, 2021 01:39
@yirutang yirutang requested a review from a team as a code owner February 23, 2021 01:39
@generated-files-bot
Copy link

Warning: This pull request is touching the following templated files:

  • samples/install-without-bom/pom.xml
  • samples/snippets/pom.xml

@snippet-bot
Copy link

snippet-bot bot commented Feb 23, 2021

Here is the summary of changes.

You are about to add 1 region tag.

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

  • Refresh this comment

@codecov
Copy link

codecov bot commented Feb 23, 2021

Codecov Report

Merging #854 (37db867) into master (7c01f45) will increase coverage by 0.36%.
The diff coverage is 83.13%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master     #854      +/-   ##
============================================
+ Coverage     80.28%   80.65%   +0.36%     
- Complexity      961     1016      +55     
============================================
  Files            76       76              
  Lines          5311     5552     +241     
  Branches        405      432      +27     
============================================
+ Hits           4264     4478     +214     
- Misses          875      900      +25     
- Partials        172      174       +2     
Impacted Files Coverage Δ Complexity Δ
...e/cloud/bigquery/storage/v1beta2/StreamWriter.java 79.20% <64.93%> (-3.54%) 31.00 <6.00> (-1.00)
...cloud/bigquery/storage/v1beta2/StreamWriterV2.java 96.13% <97.84%> (+1.17%) 35.00 <15.00> (+12.00)
.../google/cloud/bigquery/storage/v1beta2/Waiter.java 56.04% <100.00%> (+2.11%) 12.00 <0.00> (ø)
...d/bigquery/storage/v1beta2/JsonToProtoMessage.java 96.85% <0.00%> (-0.77%) 44.00% <0.00%> (ø%)
.../bigquery/storage/v1alpha2/JsonToProtoMessage.java 96.96% <0.00%> (-0.75%) 47.00% <0.00%> (ø%)
.../cloud/bigquery/storage/v1alpha2/StreamWriter.java 84.31% <0.00%> (-0.09%) 37.00% <0.00%> (ø%)
...torage/v1beta2/BQTableSchemaToProtoDescriptor.java 98.48% <0.00%> (+0.07%) 8.00% <0.00%> (ø%)
...orage/v1alpha2/BQTableSchemaToProtoDescriptor.java 98.48% <0.00%> (+0.07%) 8.00% <0.00%> (ø%)
... and 26 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7c01f45...31af7be. Read the comment docs.

@yirutang
Copy link
Contributor Author

Having issues running Yi's stress test.

@yirutang yirutang closed this Feb 23, 2021
@yirutang yirutang reopened this Feb 23, 2021
@stephaniewang526 stephaniewang526 added the status: investigating The issue is under investigation, which is determined to be non-trivial. label Feb 24, 2021
@stephaniewang526 stephaniewang526 removed the status: investigating The issue is under investigation, which is determined to be non-trivial. label Feb 26, 2021
@stephaniewang526
Copy link
Contributor

This test keeps failing:

StreamWriterTest.testFlowControlBehaviorBlock:574 Call got exception: java.lang.IllegalStateException: Cannot append on a shut-down writer.

Full trace

@yirutang
Copy link
Contributor Author

yirutang commented Feb 26, 2021 via email

@stephaniewang526 stephaniewang526 merged commit 0870797 into googleapis:master Mar 2, 2021
gcf-merge-on-green bot pushed a commit that referenced this pull request Mar 4, 2021
🤖 I have created a release \*beep\* \*boop\*
---
## [1.14.0](https://www.github.com/googleapis/java-bigquerystorage/compare/v1.13.0...v1.14.0) (2021-03-04)


### Features

* update StreamWriterV2 to support trace id ([#895](https://www.github.com/googleapis/java-bigquerystorage/issues/895)) ([2e49ce8](https://www.github.com/googleapis/java-bigquerystorage/commit/2e49ce8c79cb059840c3307898ba16980f6892fa))


### Bug Fixes

* add schema update back to json writer ([#905](https://www.github.com/googleapis/java-bigquerystorage/issues/905)) ([a2adbf8](https://www.github.com/googleapis/java-bigquerystorage/commit/a2adbf80753161cbddd23d5a7db75e9250db58fa))
* Add unit test for concurrent issues we worried about, and fix some locking issues ([#854](https://www.github.com/googleapis/java-bigquerystorage/issues/854)) ([0870797](https://www.github.com/googleapis/java-bigquerystorage/commit/087079728195e20f93701e8d5e1e59ba29a7d21b))
* test failure testAppendWhileShutdownSuccess ([#904](https://www.github.com/googleapis/java-bigquerystorage/issues/904)) ([b80183e](https://www.github.com/googleapis/java-bigquerystorage/commit/b80183ea23c8b78611a42d22d8c62a4ba4904a80))
* testAppendWhileShutdownSuccess race ([#907](https://www.github.com/googleapis/java-bigquerystorage/issues/907)) ([d39443d](https://www.github.com/googleapis/java-bigquerystorage/commit/d39443d51d2625e4b3aee59d1e593229e9e449d3))


### Dependencies

* update dependency com.google.cloud:google-cloud-bigquery to v1.127.6 ([#909](https://www.github.com/googleapis/java-bigquerystorage/issues/909)) ([505938b](https://www.github.com/googleapis/java-bigquerystorage/commit/505938bcba5a4a7af9e618572bbc41f365702f47))
---


This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
@yirutang yirutang deleted the fixfix branch April 30, 2021 19:00
shubhwip pushed a commit to shubhwip/java-bigquerystorage that referenced this pull request Oct 7, 2023
…0.123.24 (googleapis#854)

[![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [com.google.cloud:google-cloud-nio](https://togithub.com/googleapis/java-storage-nio) | `0.123.23` -> `0.123.24` | [![age](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-nio/0.123.24/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-nio/0.123.24/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-nio/0.123.24/compatibility-slim/0.123.23)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-nio/0.123.24/confidence-slim/0.123.23)](https://docs.renovatebot.com/merge-confidence/) |

---

### Release Notes

<details>
<summary>googleapis/java-storage-nio</summary>

### [`v0.123.24`](https://togithub.com/googleapis/java-storage-nio/blob/HEAD/CHANGELOG.md#&#8203;012324-httpsgithubcomgoogleapisjava-storage-niocomparev012323v012324-2022-03-09)

[Compare Source](https://togithub.com/googleapis/java-storage-nio/compare/v0.123.23...v0.123.24)

</details>

---

### Configuration

📅 **Schedule**: At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, click this checkbox.

---

This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/java-storage-nio).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquerystorage Issues related to the googleapis/java-bigquerystorage API. cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants