Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 #2295

Closed
Anviking opened this issue Nov 4, 2020 · 2 comments
Closed

Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 #2295

Anviking opened this issue Nov 4, 2020 · 2 comments
Labels
Test failure A flaky test or nightly CI failure

Comments

@Anviking
Copy link
Member

Anviking commented Nov 4, 2020 •
edited
Loading

Context

#2191 (comment)
#2191 (comment)

Only seen in #2191 (parallel) so far.

Test Case

https://github.com/input-output-hk/cardano-wallet/blob/74ebfcaec9b170f9a8dc303b679b00f570537464/lib/core-integration/src/Test/Integration/Scenario/API/Shelley/Transactions.hs#L604-L665

Failure / Counter-example

Failures:
--
  |  
  | src/Test/Integration/Scenario/API/Shelley/Transactions.hs:641:9:
  | 1) API Specifications, SHELLEY_TRANSACTIONS, TRANS_TTL_01 - Pending transaction expiry
  | predicate failed on: SlotNo 80
  |  
  | To rerun use: --match "/API Specifications/SHELLEY_TRANSACTIONS/TRANS_TTL_01 - Pending transaction expiry/"
  |  
  | src/Test/Integration/Scenario/API/Shelley/Transactions.hs:673:9:
  | 2) API Specifications, SHELLEY_TRANSACTIONS, TRANS_TTL_02 - Custom transaction expiry
  | predicate failed on: SlotNo 70
  |  
  | To rerun use: --match "/API Specifications/SHELLEY_TRANSACTIONS/TRANS_TTL_02 - Custom transaction expiry/"

Resolution

QA
@Anviking Anviking added Bug Test failure A flaky test or nightly CI failure labels Nov 4, 2020
@Anviking
Copy link
Member Author

Anviking commented Nov 4, 2020

Points from discussion with Rodney:

  • it shouldn't be due to rollback moving the slotNo back, since we get it from the POST /transaction request itself
  • 80 slots is 16s, which seems a bit high
  • could be due to "some kind of resource or lock contention which prevents the transaction being submitted to the node in good time."
    • I did see tx submission to the node take 0.5s-1s recently, which is pretty slow. Could explain it.

This was referenced Nov 5, 2020
Merged
Merged
Merged
Closed
@piotr-iohk piotr-iohk mentioned this issue Nov 12, 2020
Merged
This was referenced Nov 13, 2020
Merged
Merged
@Anviking Anviking removed the Bug label Nov 19, 2020
iohk-bors bot added a commit that referenced this issue Nov 23, 2020
@iohk-bors @Anviking
Merge #2338
483f21c 
2338: Mark TRANS_TTL_{01,02} and STAKE_POOLS_JOIN_05 pending r=Anviking a=Anviking

# Issue Number

None. Addressing CI failures.

# Overview


- [x] Add new `flakyBecauseOf ticketOrReason` helper that calls `pendingWith` unless `RUN_FLAKY_TESTS` is set.
- [x] Mark TRANS_TTL_{01,02} and STAKE_POOLS_JOIN_05 pending/flaky.
- [x] Add manual test calling for running flaky tests

# Comments

- Should lower the failure rate by 21% of runs, from 59% to 38%. 
- Next candidate for marking pending would be #2224, but with a relatively low failure rate of 3.6%, and being important, I think it would be a bad idea.
- Maybe we should have flaky tests run per default, unless setting `DONT_RUN_FLAKY_TESTS` in CI, to maximise the times we run them locally.

Recent bors failures:
```
succeded: 19, failed: 37 (66%), total: 56
excluding #expected failures

Broken down by tags/issues:
10 times #2292 Flaky test - various DB properties causing timeout | #2292
7 times #2295 Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 | #2295
6 times
3 times #2311 Flaky test - integration test timeout after/related to STAKE_POOLS_LIST_01 | #2311
3 times #2230 Flaky  STAKE_POOLS_JOIN_05 - Can join when stake key already exists | #2230
2 times #2224 Flaky STAKE_POOLS_LIST_01 - List stake pools, has non-zero saturation & stake | #2224
1 times #another-integration-timeout  |
1 times #2337 STAKE_POOLS_GARBAGE_COLLECTION_01 timed out | #2337
1 times #2320 Flaky test - The node backend is unreachable at the moment. STAKE_POOLS_QUIT_02 | #2320
1 times #2295, #2331 Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 | #2295
1 times #2207 Flaky SHELLEY_MIGRATE_01_big_wallet | #2207
1 times #2118 Property `prop_rebalanceSelection` occasionally fails. | #2118
```


<!-- Additional comments or screenshots to attach if any -->

<!-- 
Don't forget to:

 ✓ Self-review your changes to make sure nothing unexpected slipped through
 ✓ Assign yourself to the PR
 ✓ Assign one or several reviewer(s)
 ✓ Once created, link this PR to its corresponding ticket
 ✓ Assign the PR to a corresponding milestone
 ✓ Acknowledge any changes required to the Wiki
-->


Co-authored-by: Johannes Lund <[email protected]>
iohk-bors bot added a commit that referenced this issue Nov 23, 2020
@iohk-bors @Anviking
Merge #2338
0bd8c21 
2338: Mark TRANS_TTL_{01,02} and STAKE_POOLS_JOIN_05 pending r=Anviking a=Anviking

# Issue Number

None. Addressing CI failures.

# Overview


- [x] Add new `flakyBecauseOf ticketOrReason` helper that calls `pendingWith` unless `RUN_FLAKY_TESTS` is set.
- [x] Mark TRANS_TTL_{01,02} and STAKE_POOLS_JOIN_05 pending/flaky.
- [x] Add manual test calling for running flaky tests

# Comments

- Should lower the failure rate by 21% of runs, from 59% to 38%. 
- Next candidate for marking pending would be #2224, but with a relatively low failure rate of 3.6%, and being important, I think it would be a bad idea.
- Maybe we should have flaky tests run per default, unless setting `DONT_RUN_FLAKY_TESTS` in CI, to maximise the times we run them locally.

Recent bors failures:
```
succeded: 19, failed: 37 (66%), total: 56
excluding #expected failures

Broken down by tags/issues:
10 times #2292 Flaky test - various DB properties causing timeout | #2292
7 times #2295 Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 | #2295
6 times
3 times #2311 Flaky test - integration test timeout after/related to STAKE_POOLS_LIST_01 | #2311
3 times #2230 Flaky  STAKE_POOLS_JOIN_05 - Can join when stake key already exists | #2230
2 times #2224 Flaky STAKE_POOLS_LIST_01 - List stake pools, has non-zero saturation & stake | #2224
1 times #another-integration-timeout  |
1 times #2337 STAKE_POOLS_GARBAGE_COLLECTION_01 timed out | #2337
1 times #2320 Flaky test - The node backend is unreachable at the moment. STAKE_POOLS_QUIT_02 | #2320
1 times #2295, #2331 Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 | #2295
1 times #2207 Flaky SHELLEY_MIGRATE_01_big_wallet | #2207
1 times #2118 Property `prop_rebalanceSelection` occasionally fails. | #2118
```


<!-- Additional comments or screenshots to attach if any -->

<!-- 
Don't forget to:

 ✓ Self-review your changes to make sure nothing unexpected slipped through
 ✓ Assign yourself to the PR
 ✓ Assign one or several reviewer(s)
 ✓ Once created, link this PR to its corresponding ticket
 ✓ Assign the PR to a corresponding milestone
 ✓ Acknowledge any changes required to the Wiki
-->


Co-authored-by: Johannes Lund <[email protected]>
iohk-bors bot added a commit that referenced this issue Nov 23, 2020
@iohk-bors @Anviking
Merge #2338
f92a8fa 
2338: Mark TRANS_TTL_{01,02} and STAKE_POOLS_JOIN_05 pending r=Anviking a=Anviking

# Issue Number

None. Addressing CI failures.

# Overview


- [x] Add new `flakyBecauseOf ticketOrReason` helper that calls `pendingWith` unless `RUN_FLAKY_TESTS` is set.
- [x] Mark TRANS_TTL_{01,02} and STAKE_POOLS_JOIN_05 pending/flaky.
- [x] Add manual test calling for running flaky tests

# Comments

- Should lower the failure rate by 21% of runs, from 59% to 38%. 
- Next candidate for marking pending would be #2224, but with a relatively low failure rate of 3.6%, and being important, I think it would be a bad idea.
- Maybe we should have flaky tests run per default, unless setting `DONT_RUN_FLAKY_TESTS` in CI, to maximise the times we run them locally.

Recent bors failures:
```
succeded: 19, failed: 37 (66%), total: 56
excluding #expected failures

Broken down by tags/issues:
10 times #2292 Flaky test - various DB properties causing timeout | #2292
7 times #2295 Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 | #2295
6 times
3 times #2311 Flaky test - integration test timeout after/related to STAKE_POOLS_LIST_01 | #2311
3 times #2230 Flaky  STAKE_POOLS_JOIN_05 - Can join when stake key already exists | #2230
2 times #2224 Flaky STAKE_POOLS_LIST_01 - List stake pools, has non-zero saturation & stake | #2224
1 times #another-integration-timeout  |
1 times #2337 STAKE_POOLS_GARBAGE_COLLECTION_01 timed out | #2337
1 times #2320 Flaky test - The node backend is unreachable at the moment. STAKE_POOLS_QUIT_02 | #2320
1 times #2295, #2331 Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 | #2295
1 times #2207 Flaky SHELLEY_MIGRATE_01_big_wallet | #2207
1 times #2118 Property `prop_rebalanceSelection` occasionally fails. | #2118
```


<!-- Additional comments or screenshots to attach if any -->

<!-- 
Don't forget to:

 ✓ Self-review your changes to make sure nothing unexpected slipped through
 ✓ Assign yourself to the PR
 ✓ Assign one or several reviewer(s)
 ✓ Once created, link this PR to its corresponding ticket
 ✓ Assign the PR to a corresponding milestone
 ✓ Acknowledge any changes required to the Wiki
-->


Co-authored-by: Johannes Lund <[email protected]>
iohk-bors bot added a commit that referenced this issue Nov 23, 2020
@iohk-bors @Anviking
Merge #2338
12f6b1e 
2338: Mark TRANS_TTL_{01,02} and STAKE_POOLS_JOIN_05 pending r=Anviking a=Anviking

# Issue Number

None. Addressing CI failures.

# Overview


- [x] Add new `flakyBecauseOf ticketOrReason` helper that calls `pendingWith` unless `RUN_FLAKY_TESTS` is set.
- [x] Mark TRANS_TTL_{01,02} and STAKE_POOLS_JOIN_05 pending/flaky.
- [x] Add manual test calling for running flaky tests

# Comments

- Should lower the failure rate by 21% of runs, from 59% to 38%. 
- Next candidate for marking pending would be #2224, but with a relatively low failure rate of 3.6%, and being important, I think it would be a bad idea.
- Maybe we should have flaky tests run per default, unless setting `DONT_RUN_FLAKY_TESTS` in CI, to maximise the times we run them locally.

Recent bors failures:
```
succeded: 19, failed: 37 (66%), total: 56
excluding #expected failures

Broken down by tags/issues:
10 times #2292 Flaky test - various DB properties causing timeout | #2292
7 times #2295 Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 | #2295
6 times
3 times #2311 Flaky test - integration test timeout after/related to STAKE_POOLS_LIST_01 | #2311
3 times #2230 Flaky  STAKE_POOLS_JOIN_05 - Can join when stake key already exists | #2230
2 times #2224 Flaky STAKE_POOLS_LIST_01 - List stake pools, has non-zero saturation & stake | #2224
1 times #another-integration-timeout  |
1 times #2337 STAKE_POOLS_GARBAGE_COLLECTION_01 timed out | #2337
1 times #2320 Flaky test - The node backend is unreachable at the moment. STAKE_POOLS_QUIT_02 | #2320
1 times #2295, #2331 Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 | #2295
1 times #2207 Flaky SHELLEY_MIGRATE_01_big_wallet | #2207
1 times #2118 Property `prop_rebalanceSelection` occasionally fails. | #2118
```


<!-- Additional comments or screenshots to attach if any -->

<!-- 
Don't forget to:

 ✓ Self-review your changes to make sure nothing unexpected slipped through
 ✓ Assign yourself to the PR
 ✓ Assign one or several reviewer(s)
 ✓ Once created, link this PR to its corresponding ticket
 ✓ Assign the PR to a corresponding milestone
 ✓ Acknowledge any changes required to the Wiki
-->


Co-authored-by: Johannes Lund <[email protected]>
iohk-bors bot added a commit that referenced this issue Nov 23, 2020
@iohk-bors @Anviking
Merge #2338
1b11b2e 
2338: Mark TRANS_TTL_{01,02} and STAKE_POOLS_JOIN_05 pending r=Anviking a=Anviking

# Issue Number

None. Addressing CI failures.

# Overview


- [x] Add new `flakyBecauseOf ticketOrReason` helper that calls `pendingWith` unless `RUN_FLAKY_TESTS` is set.
- [x] Mark TRANS_TTL_{01,02} and STAKE_POOLS_JOIN_05 pending/flaky.
- [x] Add manual test calling for running flaky tests

# Comments

- Should lower the failure rate by 21% of runs, from 59% to 38%. 
- Next candidate for marking pending would be #2224, but with a relatively low failure rate of 3.6%, and being important, I think it would be a bad idea.
- Maybe we should have flaky tests run per default, unless setting `DONT_RUN_FLAKY_TESTS` in CI, to maximise the times we run them locally.

Recent bors failures:
```
succeded: 19, failed: 37 (66%), total: 56
excluding #expected failures

Broken down by tags/issues:
10 times #2292 Flaky test - various DB properties causing timeout | #2292
7 times #2295 Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 | #2295
6 times
3 times #2311 Flaky test - integration test timeout after/related to STAKE_POOLS_LIST_01 | #2311
3 times #2230 Flaky  STAKE_POOLS_JOIN_05 - Can join when stake key already exists | #2230
2 times #2224 Flaky STAKE_POOLS_LIST_01 - List stake pools, has non-zero saturation & stake | #2224
1 times #another-integration-timeout  |
1 times #2337 STAKE_POOLS_GARBAGE_COLLECTION_01 timed out | #2337
1 times #2320 Flaky test - The node backend is unreachable at the moment. STAKE_POOLS_QUIT_02 | #2320
1 times #2295, #2331 Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 | #2295
1 times #2207 Flaky SHELLEY_MIGRATE_01_big_wallet | #2207
1 times #2118 Property `prop_rebalanceSelection` occasionally fails. | #2118
```


<!-- Additional comments or screenshots to attach if any -->

<!-- 
Don't forget to:

 ✓ Self-review your changes to make sure nothing unexpected slipped through
 ✓ Assign yourself to the PR
 ✓ Assign one or several reviewer(s)
 ✓ Once created, link this PR to its corresponding ticket
 ✓ Assign the PR to a corresponding milestone
 ✓ Acknowledge any changes required to the Wiki
-->


Co-authored-by: Johannes Lund <[email protected]>
iohk-bors bot added a commit that referenced this issue Nov 24, 2020
@iohk-bors @Anviking
Merge #2338
d4d55e0 
2338: Mark TRANS_TTL_{01,02} and STAKE_POOLS_JOIN_05 pending r=Anviking a=Anviking

# Issue Number

None. Addressing CI failures.

# Overview


- [x] Add new `flakyBecauseOf ticketOrReason` helper that calls `pendingWith` unless `RUN_FLAKY_TESTS` is set.
- [x] Mark TRANS_TTL_{01,02} and STAKE_POOLS_JOIN_05 pending/flaky.
- [x] Add manual test calling for running flaky tests

# Comments

- Should lower the failure rate by 21% of runs, from 59% to 38%. 
- Next candidate for marking pending would be #2224, but with a relatively low failure rate of 3.6%, and being important, I think it would be a bad idea.
- Maybe we should have flaky tests run per default, unless setting `DONT_RUN_FLAKY_TESTS` in CI, to maximise the times we run them locally.

Recent bors failures:
```
succeded: 19, failed: 37 (66%), total: 56
excluding #expected failures

Broken down by tags/issues:
10 times #2292 Flaky test - various DB properties causing timeout | #2292
7 times #2295 Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 | #2295
6 times
3 times #2311 Flaky test - integration test timeout after/related to STAKE_POOLS_LIST_01 | #2311
3 times #2230 Flaky  STAKE_POOLS_JOIN_05 - Can join when stake key already exists | #2230
2 times #2224 Flaky STAKE_POOLS_LIST_01 - List stake pools, has non-zero saturation & stake | #2224
1 times #another-integration-timeout  |
1 times #2337 STAKE_POOLS_GARBAGE_COLLECTION_01 timed out | #2337
1 times #2320 Flaky test - The node backend is unreachable at the moment. STAKE_POOLS_QUIT_02 | #2320
1 times #2295, #2331 Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 | #2295
1 times #2207 Flaky SHELLEY_MIGRATE_01_big_wallet | #2207
1 times #2118 Property `prop_rebalanceSelection` occasionally fails. | #2118
```


<!-- Additional comments or screenshots to attach if any -->

<!-- 
Don't forget to:

 ✓ Self-review your changes to make sure nothing unexpected slipped through
 ✓ Assign yourself to the PR
 ✓ Assign one or several reviewer(s)
 ✓ Once created, link this PR to its corresponding ticket
 ✓ Assign the PR to a corresponding milestone
 ✓ Acknowledge any changes required to the Wiki
-->


Co-authored-by: Johannes Lund <[email protected]>
iohk-bors bot added a commit that referenced this issue Nov 24, 2020
@iohk-bors @Anviking
Merge #2338
58f2bbc 
2338: Mark TRANS_TTL_{01,02} and STAKE_POOLS_JOIN_05 pending r=Anviking a=Anviking

# Issue Number

None. Addressing CI failures.

# Overview


- [x] Add new `flakyBecauseOf ticketOrReason` helper that calls `pendingWith` unless `RUN_FLAKY_TESTS` is set.
- [x] Mark TRANS_TTL_{01,02} and STAKE_POOLS_JOIN_05 pending/flaky.
- [x] Add manual test calling for running flaky tests

# Comments

- Should lower the failure rate by 21% of runs, from 59% to 38%. 
- Next candidate for marking pending would be #2224, but with a relatively low failure rate of 3.6%, and being important, I think it would be a bad idea.
- Maybe we should have flaky tests run per default, unless setting `DONT_RUN_FLAKY_TESTS` in CI, to maximise the times we run them locally.

Recent bors failures:
```
succeded: 19, failed: 37 (66%), total: 56
excluding #expected failures

Broken down by tags/issues:
10 times #2292 Flaky test - various DB properties causing timeout | #2292
7 times #2295 Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 | #2295
6 times
3 times #2311 Flaky test - integration test timeout after/related to STAKE_POOLS_LIST_01 | #2311
3 times #2230 Flaky  STAKE_POOLS_JOIN_05 - Can join when stake key already exists | #2230
2 times #2224 Flaky STAKE_POOLS_LIST_01 - List stake pools, has non-zero saturation & stake | #2224
1 times #another-integration-timeout  |
1 times #2337 STAKE_POOLS_GARBAGE_COLLECTION_01 timed out | #2337
1 times #2320 Flaky test - The node backend is unreachable at the moment. STAKE_POOLS_QUIT_02 | #2320
1 times #2295, #2331 Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 | #2295
1 times #2207 Flaky SHELLEY_MIGRATE_01_big_wallet | #2207
1 times #2118 Property `prop_rebalanceSelection` occasionally fails. | #2118
```


<!-- Additional comments or screenshots to attach if any -->

<!-- 
Don't forget to:

 ✓ Self-review your changes to make sure nothing unexpected slipped through
 ✓ Assign yourself to the PR
 ✓ Assign one or several reviewer(s)
 ✓ Once created, link this PR to its corresponding ticket
 ✓ Assign the PR to a corresponding milestone
 ✓ Acknowledge any changes required to the Wiki
-->


Co-authored-by: Johannes Lund <[email protected]>
iohk-bors bot added a commit that referenced this issue Nov 24, 2020
@iohk-bors @Anviking
Merge #2338
18b7313 
2338: Mark TRANS_TTL_{01,02} and STAKE_POOLS_JOIN_05 pending r=jonathanknowles a=Anviking

# Issue Number

None. Addressing CI failures.

# Overview


- [x] Add new `flakyBecauseOf ticketOrReason` helper that calls `pendingWith` unless `RUN_FLAKY_TESTS` is set.
- [x] Mark TRANS_TTL_{01,02} and STAKE_POOLS_JOIN_05 pending/flaky.
- [x] Add manual test calling for running flaky tests

# Comments

- Should lower the failure rate by 21% of runs, from 59% to 38%. 
- Next candidate for marking pending would be #2224, but with a relatively low failure rate of 3.6%, and being important, I think it would be a bad idea.
- Maybe we should have flaky tests run per default, unless setting `DONT_RUN_FLAKY_TESTS` in CI, to maximise the times we run them locally.

Recent bors failures:
```
succeded: 19, failed: 37 (66%), total: 56
excluding #expected failures

Broken down by tags/issues:
10 times #2292 Flaky test - various DB properties causing timeout | #2292
7 times #2295 Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 | #2295
6 times
3 times #2311 Flaky test - integration test timeout after/related to STAKE_POOLS_LIST_01 | #2311
3 times #2230 Flaky  STAKE_POOLS_JOIN_05 - Can join when stake key already exists | #2230
2 times #2224 Flaky STAKE_POOLS_LIST_01 - List stake pools, has non-zero saturation & stake | #2224
1 times #another-integration-timeout  |
1 times #2337 STAKE_POOLS_GARBAGE_COLLECTION_01 timed out | #2337
1 times #2320 Flaky test - The node backend is unreachable at the moment. STAKE_POOLS_QUIT_02 | #2320
1 times #2295, #2331 Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 | #2295
1 times #2207 Flaky SHELLEY_MIGRATE_01_big_wallet | #2207
1 times #2118 Property `prop_rebalanceSelection` occasionally fails. | #2118
```


<!-- Additional comments or screenshots to attach if any -->

<!-- 
Don't forget to:

 ✓ Self-review your changes to make sure nothing unexpected slipped through
 ✓ Assign yourself to the PR
 ✓ Assign one or several reviewer(s)
 ✓ Once created, link this PR to its corresponding ticket
 ✓ Assign the PR to a corresponding milestone
 ✓ Acknowledge any changes required to the Wiki
-->


Co-authored-by: Johannes Lund <[email protected]>
iohk-bors bot added a commit that referenced this issue Nov 24, 2020
@iohk-bors @Anviking
Merge #2338
2e6e7a0 
2338: Mark TRANS_TTL_{01,02} and STAKE_POOLS_JOIN_05 pending r=jonathanknowles a=Anviking

# Issue Number

None. Addressing CI failures.

# Overview


- [x] Add new `flakyBecauseOf ticketOrReason` helper that calls `pendingWith` unless `RUN_FLAKY_TESTS` is set.
- [x] Mark TRANS_TTL_{01,02} and STAKE_POOLS_JOIN_05 pending/flaky.
- [x] Add manual test calling for running flaky tests

# Comments

- Should lower the failure rate by 21% of runs, from 59% to 38%. 
- Next candidate for marking pending would be #2224, but with a relatively low failure rate of 3.6%, and being important, I think it would be a bad idea.
- Maybe we should have flaky tests run per default, unless setting `DONT_RUN_FLAKY_TESTS` in CI, to maximise the times we run them locally.

Recent bors failures:
```
succeded: 19, failed: 37 (66%), total: 56
excluding #expected failures

Broken down by tags/issues:
10 times #2292 Flaky test - various DB properties causing timeout | #2292
7 times #2295 Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 | #2295
6 times
3 times #2311 Flaky test - integration test timeout after/related to STAKE_POOLS_LIST_01 | #2311
3 times #2230 Flaky  STAKE_POOLS_JOIN_05 - Can join when stake key already exists | #2230
2 times #2224 Flaky STAKE_POOLS_LIST_01 - List stake pools, has non-zero saturation & stake | #2224
1 times #another-integration-timeout  |
1 times #2337 STAKE_POOLS_GARBAGE_COLLECTION_01 timed out | #2337
1 times #2320 Flaky test - The node backend is unreachable at the moment. STAKE_POOLS_QUIT_02 | #2320
1 times #2295, #2331 Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 | #2295
1 times #2207 Flaky SHELLEY_MIGRATE_01_big_wallet | #2207
1 times #2118 Property `prop_rebalanceSelection` occasionally fails. | #2118
```


<!-- Additional comments or screenshots to attach if any -->

<!-- 
Don't forget to:

 ✓ Self-review your changes to make sure nothing unexpected slipped through
 ✓ Assign yourself to the PR
 ✓ Assign one or several reviewer(s)
 ✓ Once created, link this PR to its corresponding ticket
 ✓ Assign the PR to a corresponding milestone
 ✓ Acknowledge any changes required to the Wiki
-->


Co-authored-by: Johannes Lund <[email protected]>
iohk-bors bot added a commit that referenced this issue Nov 24, 2020
@iohk-bors @Anviking
Merge #2338
3be544c 
2338: Mark TRANS_TTL_{01,02} and STAKE_POOLS_JOIN_05 pending r=jonathanknowles a=Anviking

# Issue Number

None. Addressing CI failures.

# Overview


- [x] Add new `flakyBecauseOf ticketOrReason` helper that calls `pendingWith` unless `RUN_FLAKY_TESTS` is set.
- [x] Mark TRANS_TTL_{01,02} and STAKE_POOLS_JOIN_05 pending/flaky.
- [x] Add manual test calling for running flaky tests

# Comments

- Should lower the failure rate by 21% of runs, from 59% to 38%. 
- Next candidate for marking pending would be #2224, but with a relatively low failure rate of 3.6%, and being important, I think it would be a bad idea.
- Maybe we should have flaky tests run per default, unless setting `DONT_RUN_FLAKY_TESTS` in CI, to maximise the times we run them locally.

Recent bors failures:
```
succeded: 19, failed: 37 (66%), total: 56
excluding #expected failures

Broken down by tags/issues:
10 times #2292 Flaky test - various DB properties causing timeout | #2292
7 times #2295 Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 | #2295
6 times
3 times #2311 Flaky test - integration test timeout after/related to STAKE_POOLS_LIST_01 | #2311
3 times #2230 Flaky  STAKE_POOLS_JOIN_05 - Can join when stake key already exists | #2230
2 times #2224 Flaky STAKE_POOLS_LIST_01 - List stake pools, has non-zero saturation & stake | #2224
1 times #another-integration-timeout  |
1 times #2337 STAKE_POOLS_GARBAGE_COLLECTION_01 timed out | #2337
1 times #2320 Flaky test - The node backend is unreachable at the moment. STAKE_POOLS_QUIT_02 | #2320
1 times #2295, #2331 Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 | #2295
1 times #2207 Flaky SHELLEY_MIGRATE_01_big_wallet | #2207
1 times #2118 Property `prop_rebalanceSelection` occasionally fails. | #2118
```


<!-- Additional comments or screenshots to attach if any -->

<!-- 
Don't forget to:

 ✓ Self-review your changes to make sure nothing unexpected slipped through
 ✓ Assign yourself to the PR
 ✓ Assign one or several reviewer(s)
 ✓ Once created, link this PR to its corresponding ticket
 ✓ Assign the PR to a corresponding milestone
 ✓ Acknowledge any changes required to the Wiki
-->


Co-authored-by: Johannes Lund <[email protected]>
iohk-bors bot added a commit that referenced this issue Nov 25, 2020
@iohk-bors @Anviking
Merge #2338
fca0086 
2338: Mark TRANS_TTL_{01,02}, STAKE_POOLS_JOIN_05, and STAKE_POOLS_SMASH_01 pending r=Anviking a=Anviking

# Issue Number

None. Addressing CI failures.

# Overview


- [x] Add new `flakyBecauseOf ticketOrReason` helper that calls `pendingWith` unless `RUN_FLAKY_TESTS` is set.
- [x] Mark TRANS_TTL_{01,02} and STAKE_POOLS_JOIN_05 pending/flaky.
- [x] Also mark STAKE_POOLS_SMASH_01 pending
- [x] Add manual test calling for running flaky tests

# Comments

- Should lower the failure rate by 21% of runs, from 59% to 38%. 
- Next candidate for marking pending would be #2224, but with a relatively low failure rate of 3.6%, and being important, I think it would be a bad idea.
- Maybe we should have flaky tests run per default, unless setting `DONT_RUN_FLAKY_TESTS` in CI, to maximise the times we run them locally.

Recent bors failures:
```
succeded: 19, failed: 37 (66%), total: 56
excluding #expected failures

Broken down by tags/issues:
10 times #2292 Flaky test - various DB properties causing timeout | #2292
7 times #2295 Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 | #2295
6 times
3 times #2311 Flaky test - integration test timeout after/related to STAKE_POOLS_LIST_01 | #2311
3 times #2230 Flaky  STAKE_POOLS_JOIN_05 - Can join when stake key already exists | #2230
2 times #2224 Flaky STAKE_POOLS_LIST_01 - List stake pools, has non-zero saturation & stake | #2224
1 times #another-integration-timeout  |
1 times #2337 STAKE_POOLS_GARBAGE_COLLECTION_01 timed out | #2337
1 times #2320 Flaky test - The node backend is unreachable at the moment. STAKE_POOLS_QUIT_02 | #2320
1 times #2295, #2331 Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 | #2295
1 times #2207 Flaky SHELLEY_MIGRATE_01_big_wallet | #2207
1 times #2118 Property `prop_rebalanceSelection` occasionally fails. | #2118
```


<!-- Additional comments or screenshots to attach if any -->

<!-- 
Don't forget to:

 ✓ Self-review your changes to make sure nothing unexpected slipped through
 ✓ Assign yourself to the PR
 ✓ Assign one or several reviewer(s)
 ✓ Once created, link this PR to its corresponding ticket
 ✓ Assign the PR to a corresponding milestone
 ✓ Acknowledge any changes required to the Wiki
-->


Co-authored-by: Johannes Lund <[email protected]>
iohk-bors bot added a commit that referenced this issue Nov 25, 2020
@iohk-bors @Anviking @KtorZ
Merge #2338 #2346
2b7ca4a 
2338: Mark TRANS_TTL_{01,02}, STAKE_POOLS_JOIN_05, and STAKE_POOLS_SMASH_01 pending r=Anviking a=Anviking

# Issue Number

None. Addressing CI failures.

# Overview


- [x] Add new `flakyBecauseOf ticketOrReason` helper that calls `pendingWith` unless `RUN_FLAKY_TESTS` is set.
- [x] Mark TRANS_TTL_{01,02} and STAKE_POOLS_JOIN_05 pending/flaky.
- [x] Also mark STAKE_POOLS_SMASH_01 pending
- [x] Add manual test calling for running flaky tests

# Comments

- Should lower the failure rate by 21% of runs, from 59% to 38%. 
- Next candidate for marking pending would be #2224, but with a relatively low failure rate of 3.6%, and being important, I think it would be a bad idea.
- Maybe we should have flaky tests run per default, unless setting `DONT_RUN_FLAKY_TESTS` in CI, to maximise the times we run them locally.

Recent bors failures:
```
succeded: 19, failed: 37 (66%), total: 56
excluding #expected failures

Broken down by tags/issues:
10 times #2292 Flaky test - various DB properties causing timeout | #2292
7 times #2295 Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 | #2295
6 times
3 times #2311 Flaky test - integration test timeout after/related to STAKE_POOLS_LIST_01 | #2311
3 times #2230 Flaky  STAKE_POOLS_JOIN_05 - Can join when stake key already exists | #2230
2 times #2224 Flaky STAKE_POOLS_LIST_01 - List stake pools, has non-zero saturation & stake | #2224
1 times #another-integration-timeout  |
1 times #2337 STAKE_POOLS_GARBAGE_COLLECTION_01 timed out | #2337
1 times #2320 Flaky test - The node backend is unreachable at the moment. STAKE_POOLS_QUIT_02 | #2320
1 times #2295, #2331 Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 | #2295
1 times #2207 Flaky SHELLEY_MIGRATE_01_big_wallet | #2207
1 times #2118 Property `prop_rebalanceSelection` occasionally fails. | #2118
```


<!-- Additional comments or screenshots to attach if any -->

<!-- 
Don't forget to:

 ✓ Self-review your changes to make sure nothing unexpected slipped through
 ✓ Assign yourself to the PR
 ✓ Assign one or several reviewer(s)
 ✓ Once created, link this PR to its corresponding ticket
 ✓ Assign the PR to a corresponding milestone
 ✓ Acknowledge any changes required to the Wiki
-->


2346: get rid of 'OnDanglingChange' option for fee balancing r=KtorZ a=KtorZ

# Issue Number

<!-- Put here a reference to the issue this PR relates to and which requirements it tackles -->

ADP-568

# Overview

<!-- Detail in a few bullet points the work accomplished in this PR -->

- [ ] I have removed 'OnDanglingChange' option for fee balancing.

# Comments

<!-- Additional comments or screenshots to attach if any -->

  This option allowed choosing between two modes: SaveMoney and
  PayAndBalance. The former was used with cardano-node, and the latter
  used with jormungandr. The reason for having a difference is was
  because of discrepency in the minimum transaction expected by both
  ledger. On jormungandr, transactions have to be _exactly_ balanced and
  leave exactly the expected fees required by the network. On the
  counterpart, the fee calculation was only a function of the number of
  inputs and outputs... therefore much easier to satisfy than on
  cardano-node. Now that we've removed jormungandr, this extra
  indirection / complexity is just harmful. Since the 'PayAndBalance'
  mode is never used, I've removed the option entirely and made code
  assume 'SaveMoney' everywhere it used to choose between both
  alternatives.

  This also seemingly remove the 'allowUnbalancedTx' field from the
  transaction layer which was directly related to this option.

<!-- 
Don't forget to:

 ✓ Self-review your changes to make sure nothing unexpected slipped through
 ✓ Assign yourself to the PR
 ✓ Assign one or several reviewer(s)
 ✓ Once created, link this PR to its corresponding ticket
 ✓ Assign the PR to a corresponding milestone
 ✓ Acknowledge any changes required to the Wiki
-->


Co-authored-by: Johannes Lund <[email protected]>
Co-authored-by: KtorZ <[email protected]>
iohk-bors bot added a commit that referenced this issue Nov 25, 2020
@iohk-bors @Anviking
Merge #2338
3528105 
2338: Mark TRANS_TTL_{01,02}, STAKE_POOLS_JOIN_05, and STAKE_POOLS_SMASH_01 pending r=Anviking a=Anviking

# Issue Number

None. Addressing CI failures.

# Overview


- [x] Add new `flakyBecauseOf ticketOrReason` helper that calls `pendingWith` unless `RUN_FLAKY_TESTS` is set.
- [x] Mark TRANS_TTL_{01,02} and STAKE_POOLS_JOIN_05 pending/flaky.
- [x] Also mark STAKE_POOLS_SMASH_01 pending
- [x] Add manual test calling for running flaky tests

# Comments

- Should lower the failure rate by 21% of runs, from 59% to 38%. 
- Next candidate for marking pending would be #2224, but with a relatively low failure rate of 3.6%, and being important, I think it would be a bad idea.
- Maybe we should have flaky tests run per default, unless setting `DONT_RUN_FLAKY_TESTS` in CI, to maximise the times we run them locally.

Recent bors failures:
```
succeded: 19, failed: 37 (66%), total: 56
excluding #expected failures

Broken down by tags/issues:
10 times #2292 Flaky test - various DB properties causing timeout | #2292
7 times #2295 Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 | #2295
6 times
3 times #2311 Flaky test - integration test timeout after/related to STAKE_POOLS_LIST_01 | #2311
3 times #2230 Flaky  STAKE_POOLS_JOIN_05 - Can join when stake key already exists | #2230
2 times #2224 Flaky STAKE_POOLS_LIST_01 - List stake pools, has non-zero saturation & stake | #2224
1 times #another-integration-timeout  |
1 times #2337 STAKE_POOLS_GARBAGE_COLLECTION_01 timed out | #2337
1 times #2320 Flaky test - The node backend is unreachable at the moment. STAKE_POOLS_QUIT_02 | #2320
1 times #2295, #2331 Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 | #2295
1 times #2207 Flaky SHELLEY_MIGRATE_01_big_wallet | #2207
1 times #2118 Property `prop_rebalanceSelection` occasionally fails. | #2118
```


<!-- Additional comments or screenshots to attach if any -->

<!-- 
Don't forget to:

 ✓ Self-review your changes to make sure nothing unexpected slipped through
 ✓ Assign yourself to the PR
 ✓ Assign one or several reviewer(s)
 ✓ Once created, link this PR to its corresponding ticket
 ✓ Assign the PR to a corresponding milestone
 ✓ Acknowledge any changes required to the Wiki
-->


Co-authored-by: Johannes Lund <[email protected]>
iohk-bors bot added a commit that referenced this issue Nov 25, 2020
@iohk-bors @Anviking
Merge #2338
426f744 
2338: Mark TRANS_TTL_{01,02}, STAKE_POOLS_JOIN_05, and STAKE_POOLS_SMASH_01 pending r=Anviking a=Anviking

# Issue Number

None. Addressing CI failures.

# Overview


- [x] Add new `flakyBecauseOf ticketOrReason` helper that calls `pendingWith` unless `RUN_FLAKY_TESTS` is set.
- [x] Mark TRANS_TTL_{01,02} and STAKE_POOLS_JOIN_05 pending/flaky.
- [x] Also mark STAKE_POOLS_SMASH_01 pending
- [x] Add manual test calling for running flaky tests

# Comments

- Should lower the failure rate by 21% of runs, from 59% to 38%. 
- Next candidate for marking pending would be #2224, but with a relatively low failure rate of 3.6%, and being important, I think it would be a bad idea.
- Maybe we should have flaky tests run per default, unless setting `DONT_RUN_FLAKY_TESTS` in CI, to maximise the times we run them locally.

Recent bors failures:
```
succeded: 19, failed: 37 (66%), total: 56
excluding #expected failures

Broken down by tags/issues:
10 times #2292 Flaky test - various DB properties causing timeout | #2292
7 times #2295 Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 | #2295
6 times
3 times #2311 Flaky test - integration test timeout after/related to STAKE_POOLS_LIST_01 | #2311
3 times #2230 Flaky  STAKE_POOLS_JOIN_05 - Can join when stake key already exists | #2230
2 times #2224 Flaky STAKE_POOLS_LIST_01 - List stake pools, has non-zero saturation & stake | #2224
1 times #another-integration-timeout  |
1 times #2337 STAKE_POOLS_GARBAGE_COLLECTION_01 timed out | #2337
1 times #2320 Flaky test - The node backend is unreachable at the moment. STAKE_POOLS_QUIT_02 | #2320
1 times #2295, #2331 Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 | #2295
1 times #2207 Flaky SHELLEY_MIGRATE_01_big_wallet | #2207
1 times #2118 Property `prop_rebalanceSelection` occasionally fails. | #2118
```


<!-- Additional comments or screenshots to attach if any -->

<!-- 
Don't forget to:

 ✓ Self-review your changes to make sure nothing unexpected slipped through
 ✓ Assign yourself to the PR
 ✓ Assign one or several reviewer(s)
 ✓ Once created, link this PR to its corresponding ticket
 ✓ Assign the PR to a corresponding milestone
 ✓ Acknowledge any changes required to the Wiki
-->


Co-authored-by: Johannes Lund <[email protected]>
@Anviking
Copy link
Member Author

Stale, closing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Test failure A flaky test or nightly CI failure
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Anviking