Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DB migration tests failing intermittently #1649

Closed
piotr-iohk opened this issue May 11, 2020 · 4 comments · Fixed by #1681
Closed

DB migration tests failing intermittently #1649

piotr-iohk opened this issue May 11, 2020 · 4 comments · Fixed by #1681
Assignees

Comments

@piotr-iohk
Copy link
Contributor

Context

DB migration test seem to be failing intermittently. In last 3 nightly builds (all of them being run on the same revision 655a459) they failed twice, i.e.:

NB 509 - passed (:heavy_check_mark:)
NB 510 - failed (:x:)
NB 511 - failed (:x:)

Test Case

DB migration test

Failure / Counter-example

NB 510
https://buildkite.com/input-output-hk/cardano-wallet-nightly/builds/510#60af6023-b991-4dcb-8715-572668288d9f

Seems that Wallet stopped working (?) after migrating from v2020-04-28 => v2020-05-06


[migration-test:Notice:4] [2020-05-09 01:13:33.74 UTC] Starting process cardano-wallet-jormungandr
--
  | launch
  | --state-dir ./state-migration-test-v2020-04-28
  | --genesis-block /nix/store/8b6in2dygz6hk29p9zpyxzhrs17cb3ni-cardano-wallet-src/lib/jormungandr/test/data/jormungandr/block0.bin
  | --
  | --secret /nix/store/8b6in2dygz6hk29p9zpyxzhrs17cb3ni-cardano-wallet-src/lib/jormungandr/test/data/jormungandr/secret.yaml
  | --config /nix/store/8b6in2dygz6hk29p9zpyxzhrs17cb3ni-cardano-wallet-src/lib/jormungandr/test/data/jormungandr/config.yaml
  |  
  | [migration-test:Info:4] [2020-05-09 01:13:33.74 UTC] [cardano-wallet-jormungandr.128152] Process started
  | [migration-test:Debug:13] [2020-05-09 01:13:33.74 UTC] [cardano-wallet-jormungandr.128152] About to waitForProcess
  | [migration-test:Debug:12] [2020-05-09 01:13:33.74 UTC] [cardano-wallet-jormungandr.128152] Running withBackend action
  | [migration-test:Notice:12] [2020-05-09 01:13:33.74 UTC] Waiting for sync
  | [cardano-wallet.network:Notice:13] [2020-05-09 01:13:33.80 UTC] [jormungandr.128078] Terminating child process
  | [cardano-wallet.main:Debug:4] [2020-05-09 01:13:33.80 UTC] Logging shutdown.
  | [migration-test:Notice:12] [2020-05-09 01:13:33.80 UTC] [cardano-wallet-jormungandr.128152] Terminating child process
  | migration-test: HttpExceptionRequest Request {
  | host                 = "localhost"
  | port                 = 8090
  | secure               = False
  | requestHeaders       = [("User-Agent","haskell wreq-0.5.3.2")]
  | path                 = "/v2/network/information"
  | queryString          = ""
  | method               = "GET"
  | proxy                = Nothing
  | rawBody              = False
  | redirectCount        = 10
  | responseTimeout      = ResponseTimeoutDefault
  | requestVersion       = HTTP/1.1

NB 511
https://buildkite.com/input-output-hk/cardano-wallet-nightly/builds/511#e16f28bd-fab4-4d1b-a3a4-51ad7737a67f

This time looks like wallet stopped working at some point on migration v2020-04-07 => v2020-05-06


cardano-wallet-jormungandr: ErrUnexpectedNetworkFailure (Link {_segments = ["api","v0","tip"], _queryParams = []}) (FailureResponse (Request {requestPath = (BaseUrl {baseUrlScheme = Http, baseUrlHost = "127.0.0.1", baseUrlPort = 33759, baseUrlPath = ""},"/api/v0/tip"), requestQueryString = fromList [], requestBody = Nothing, requestAccept = fromList [text/plain;charset=utf-8], requestHeaders = fromList []), requestHttpVersion = HTTP/1.1, requestMethod = "GET"} (Response {responseStatusCode = Status {statusCode = 500, statusMessage = "Internal Server Error"}, responseHeaders = fromList [("content-length","38"),("content-type","text/plain; charset=utf-8"),("date","Sun, 10 May 2020 01:20:01 GMT")], responseHttpVersion = HTTP/1.1, responseBody = "Blockchain tip not set in REST context"}))
--
  | [migration-test:Debug:13] [2020-05-10 01:20:01.06 UTC] [cardano-wallet-jormungandr.105008] waitForProcess returned 1
  | [migration-test:Notice:12] [2020-05-10 01:20:01.06 UTC] [cardano-wallet-jormungandr.105008] Terminating child process
  | [migration-test:Error:4] [2020-05-10 01:20:01.06 UTC] Child process cardano-wallet-jormungandr exited with status 1
  | [migration-test:Error:4] [2020-05-10 01:20:01.06 UTC] Process exited with status ExitFailure 1



Resolution


QA

@rvl
Copy link
Contributor

rvl commented May 11, 2020

It's due to port 8090 being in use by another running build I think. The migration test should pick a free port to use rather than 8090.

@KtorZ KtorZ added BUG:CONFIRMED and removed BUG? labels May 11, 2020
@KtorZ KtorZ added BUG? and removed BUG:CONFIRMED labels May 11, 2020
iohk-bors bot added a commit that referenced this issue May 14, 2020
1662: Use port 9090 on migration tests not to interfere with any wallet tests that run on default port r=piotr-iohk a=piotr-iohk



# Issue Number

#1649 (comment)

# Overview

- 9faa90b
  Use port 9090 on migration tests not to interfere with any wallet tests that run on default port
  


# Comments

<!-- Additional comments or screenshots to attach if any -->

<!-- 
Don't forget to:

 ✓ Self-review your changes to make sure nothing unexpected slipped through
 ✓ Assign yourself to the PR
 ✓ Assign one or several reviewer(s)
 ✓ Once created, link this PR to its corresponding ticket
 ✓ Assign the PR to a corresponding milestone
 ✓ Acknowledge any changes required to the Wiki
-->


Co-authored-by: Piotr Stachyra <[email protected]>
@piotr-iohk
Copy link
Contributor Author

Closing #1662 should improve stability.

@piotr-iohk
Copy link
Contributor Author

It seems the same/similar issue occured again despite changing port to non-default 9090...
https://buildkite.com/input-output-hk/cardano-wallet-nightly/builds/516#39f40217-30d4-47c9-b7da-71a3761998bb

[cardano-wallet.main:Debug:4] [2020-05-18 06:37:26.53 UTC] Logging shutdown.
--
  | cardano-wallet-jormungandr: ErrUnexpectedNetworkFailure (Link {_segments = ["api","v0","tip"], _queryParams = []}) (FailureResponse (Request {requestPath = (BaseUrl {baseUrlScheme = Http, baseUrlHost = "127.0.0.1", baseUrlPort = 44359, baseUrlPath = ""},"/api/v0/tip"), requestQueryString = fromList [], requestBody = Nothing, requestAccept = fromList [text/plain;charset=utf-8], requestHeaders = fromList []), requestHttpVersion = HTTP/1.1, requestMethod = "GET"} (Response {responseStatusCode = Status {statusCode = 500, statusMessage = "Internal Server Error"}, responseHeaders = fromList [("content-length","38"),("content-type","text/plain; charset=utf-8"),("date","Mon, 18 May 2020 06:37:26 GMT")], responseHttpVersion = HTTP/1.1, responseBody = "Blockchain tip not set in REST context"}))
  | [migration-test:Debug:13] [2020-05-18 06:37:26.53 UTC] [cardano-wallet-jormungandr.82473] waitForProcess returned 1
  | [migration-test:Notice:12] [2020-05-18 06:37:26.53 UTC] [cardano-wallet-jormungandr.82473] Terminating child process
  | [migration-test:Error:4] [2020-05-18 06:37:26.53 UTC] Child process cardano-wallet-jormungandr exited with status 1
  | [migration-test:Error:4] [2020-05-18 06:37:26.53 UTC] Process exited with status ExitFailure 1


@rvl
Copy link
Contributor

rvl commented May 19, 2020

@piotr-iohk The port 8090 conflict was still a problem, so changing the port would have fixed that. This other error is due to #1647 I think.

@rvl rvl added BUG:CONFIRMED and removed BUG? labels May 21, 2020
@rvl rvl self-assigned this May 21, 2020
iohk-bors bot added a commit that referenced this issue May 22, 2020
1676: Bump cardano-node to pioneer-3 r=rvl a=Anviking

# Issue
None, or #1657
# Overview

- [x] I have bumped cardano-node to the `pioneer-3` tag, such that the new friends-and-family testnet works

# Comments

- `nix-shell` not working yet

<!-- Additional comments or screenshots to attach if any -->

<!-- 
Don't forget to:

 ✓ Self-review your changes to make sure nothing unexpected slipped through
 ✓ Assign yourself to the PR
 ✓ Assign one or several reviewer(s)
 ✓ Once created, link this PR to its corresponding ticket
 ✓ Assign the PR to a corresponding milestone
 ✓ Acknowledge any changes required to the Wiki
-->


1679: Reduce running time of fee estimation endpoint r=rvl a=rvl

### Issue Number

Relates to ADP-283 / #1648.

### Overview

The nightly api endpoint latency benchmark got 100× slower for `postTransactionFee`. This fixes it by moving common setup for coin selection outside of the loop.

### Comments

```
    Latencies for 10 fixture wallets scenario
        postTransactionFee - before - 102.2 ms
        postTransactionFee  - after - 4.8 ms
```

1680: Fix port selection in the db migration tests r=rvl a=rvl

### Issue number

Relates to #1649

### Overview

Each migration test running in turn was using the same port.

However due to the `SO_LINGER` TCP socket option, the port remains in use for a short time after the wallet server process exits.

So use a different port for each test.

### Comments

Tested with:

    nix-build -A migration-tests -o migration-tests && ./migration-tests/runall.sh


Co-authored-by: Johannes Lund <[email protected]>
Co-authored-by: Rodney Lorrimar <[email protected]>
@iohk-bors iohk-bors bot closed this as completed in 6a23e94 May 22, 2020
@iohk-bors iohk-bors bot closed this as completed in #1681 May 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants