Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Weeder: Command exited with code -9 #2695

Closed
Anviking opened this issue Jun 8, 2021 · 7 comments · Fixed by #2765
Closed

Weeder: Command exited with code -9 #2695

Anviking opened this issue Jun 8, 2021 · 7 comments · Fixed by #2765
Labels
Test failure A flaky test or nightly CI failure

Comments

@Anviking
Copy link
Member

Anviking commented Jun 8, 2021 •
edited
Loading

Context

Test Case

Weeder check

Failure / Counter-example

--- Weeder
weeder 
error: Command exited with code -9!
Continuing...
./buildkite-script-15be0fbf-9d48-4997-b874-643c35bebbf3: line 5: 30308 Killed                  ./sr/bin/rebuild --build-dir=$BUILD_DIR --cache-dir=$CACHE_DIR

Resolution

QA
@Anviking Anviking added the Test failure A flaky test or nightly CI failure label Jun 8, 2021
This was referenced Jun 8, 2021
Merged
Merged
@rvl
Copy link
Contributor

rvl commented Jun 9, 2021

On Linux, if a process exits with -9, it's because it's been killed with SIGKILL.

Normally builds or processes are stopped with SIGTERM or SIGINT.

So I believe this build failure is probably due to the Linux OOM killer running on the Buildkite build agent.

It may have been another project's build running at the same time which used all the memory and caused our build to be zapped.

This would hopefully be a pretty rare situation.

@Anviking If you provide a link to the buildkite job, we examine the grafana host monitoring charts at the time of the build, and perhaps see the cause.

@Anviking Anviking mentioned this issue Jun 10, 2021
Merged
1 task
@Anviking
Copy link
Member Author

If you provide a link to the buildkite job, we examine the grafana host monitoring charts at the time of the build, and perhaps see the cause.

@rvl FYI they are linked indirectly through github:

Skärmavbild 2021-06-10 kl 14 22 06

But e.g. https://buildkite.com/input-output-hk/cardano-wallet/builds/15044

@rvl
Copy link
Contributor

rvl commented Jun 11, 2021

OK I see. The monitoring shows something gobbled all the memory at the time the build failed:

grafana-packet-ipxe-2

@rvl
Copy link
Contributor

rvl commented Jun 11, 2021

I wonder if it's a problem with weeder itself. Because after looking at the logs for packet-ipxe-2, the only jobs running around the same time were:

@Anviking
Copy link
Member Author

OK I see. The monitoring shows something gobbled all the memory at the time the build failed:

😮 nice finding!

I wonder if it's a problem with weeder itself.

✔️ 🤔

@Anviking Anviking mentioned this issue Jun 14, 2021
Merged
@Anviking Anviking mentioned this issue Jul 13, 2021
Merged
2 tasks
@rvl
Copy link
Contributor

rvl commented Jul 19, 2021

Happened again in https://buildkite.com/input-output-hk/cardano-wallet/builds/15563#d800ef36-7911-4a15-aced-2e28621653ce/10267-10269. The weeder step was running for 27 minutes before it was killed.

This was referenced Jul 19, 2021
Merged
Merged
@rvl
Copy link
Contributor

rvl commented Jul 19, 2021

After the weeder config fixes, I found this error on Buildkite:

incompatible hie file: /build/cardano-wallet/lib/text-class/.stack-work/dist/x86_64-linux/Cabal-3.2.1.0/build/Test/Text/Roundtrip.hie
    expected .hie file version 8104 but got 8105
    weeder must be built with the same GHC version as the project it is used on

That is due to loading a cached .stack-work built with a previous compiler version.

Perhaps the weeder problem is due to corrupted .hie files or something like that?

@rvl rvl mentioned this issue Jul 20, 2021
Merged
5 tasks
iohk-bors bot added a commit that referenced this issue Jul 20, 2021
@iohk-bors @rvl @jonathanknowles
Merge #2765
f366405 
2765: Fix weeder 2.x config and remove weeds r=rvl a=rvl

### Issue Number

This could fix #2695, I really hope it does.

### Overview

I was looking into #2695, running `weeder` locally to see what it could be doing.

But it turns out that Weeder has been doing nothing at all since upgrading `weeder` to 2.1.3, which happened when we upgraded to ghc-8.10.4.

It needed to have a ghc option added to generate `.hie` files, which I didn't set before. If Weeder can't find `.hie` files, it just reports zero weeds (ocharles/weeder#19).

So I have:
1. Fixed the Weeder config for Stack and Cabal.
2. Removed weeds which would have been introduced while weeder wasn't working.
3. Removed weeds found by Weeder 2.x, which were never found by Weeder 1.0.9.

### Comments

```
 49 files changed, 109 insertions(+), 1681 deletions(-)
```


Co-authored-by: Rodney Lorrimar <[email protected]>
Co-authored-by: Jonathan Knowles <[email protected]>
@iohk-bors iohk-bors bot closed this as completed in f8455ff Jul 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Test failure A flaky test or nightly CI failure
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix weeder 2.x config and remove weeds cardano-foundation/cardano-wallet
2 participants
@Anviking @rvl