"unexpected EOF" causes Google Cloud workflows to fail

[teor2345]

Motivation

Sometimes Google Cloud CI workflows fail with:

unexpected EOF
Error: Process completed with exit code 1.

https://github.com/ZcashFoundation/zebra/runs/5728525849?check_suite_focus=true

This seems to be happening a few times a day. It might be a Google Cloud change or infrastructure bug.

Suggested Fix

If the ssh command fails, wait a few seconds, then re-launch the command. Fail if the ssh command has failed more than 5-10 times.

This needs to be fixed in every workflow that follows Google Cloud logs.

[dconnolly]

This is still happening, I don't think disk size is an issue, we have 100GB disks

[teor2345]

I added a suggested fix to this ticket:

If the ssh command fails, wait a few seconds, then re-launch the command. Fail if the ssh command has failed more than 5-10 times.

[teor2345]

This seems to be happening a lot, we might want to make it a high priority for the next sprint.

[dconnolly]

Just seen again on a boring dependency update

[gustavovalverde]

I've been researching a bit about this, there's no error in the infrastructure side. I found some disperse Rust reports with similar issues, some of those ended up finding an issue with the logs parser. Maybe this could be related to a change I required to @jvff, to enable coloring in the terminal (?). I could temporarily disable the feature and see if this still happens.

[teor2345]

@gustavovalverde can we try repeating the ssh command if it fails?

That might be a simple workaround until we can diagnose and fix the underlying issue.

[teor2345]

This is a blocker for #4155, because these errors are failing every full sync test I try to run.