Shutdown gracefully on SIGPIPE, SIGHUP

[michaelsproul]

Description

When running Lighthouse in a shell pipeline it fails to exit gracefully because of the lack of signal handler for SIGPIPE. I noticed that my Lighthouse nodes wouldn't shutdown gracefully from my runner script, and it took a bit of digging to find the cause.

This command reproduces the issue (bash assumed):

$ lighthouse bn --network pyrmont |& tee -a pyrmont_bn.log

Or equivalently:

$ lighthouse bn --network pyrmont 2>&1 | tee -a pyrmont_bn.log

Both of these commands redirect lighthouse's stderr (2) to its stdout (&1), and then send stdout to tee for writing to the terminal and a file simultaneously (a common UNIX pattern). If you hit ^C while running either of these commands Lighthouse will exit immediately, without running the shutdown handlers, running drop on the BeaconChain, or printing these messages:

Dec 22 18:09:01.939 INFO Shutting down..
Dec 22 18:09:07.463 INFO Saved beacon chain to disk              service: beacon

Version

Lighthouse v1.0.4

Steps to resolve

Currently we use the ctrlc crate to register a signal handler for SIGINT and SIGTERM. Unfortunately it seems this crate doesn't allow us to register handlers for other signals.

lighthouse/lighthouse/environment/src/lib.rs

Lines 382 to 392 in 1abc70e

	ctrlc::set_handler(move || {
	if let Some(ctrlc_send) = ctrlc_send_c.try_borrow_mut().unwrap().take() {
	if let Err(e) = ctrlc_send.send(()) {
	error!(
	log,
	"Error sending ctrl-c message";
	"error" => e
	);
	}
	}
	})

Using another crate (or maybe tokio::signal), we should ensure that this handler runs for all of the signals we can reasonably handle. Based on the list here I think that should just be SIGINT, SIGTERM, plus:

• SIGPIPE: The SIGPIPE signal is sent to a process when it attempts to write to a pipe without a process connected to the other end.
• SIGHUP: The SIGHUP signal is sent to a process when its controlling terminal is closed.

[mstallmo]

Would love to take a stab at resolving this if no one is working on it already!

[michaelsproul]

@mstallmo That would be awesome! There's nobody working on it currently, it's all yours!

[michaelsproul]

Finally got to the bottom of this here: #2486 (review)

For anyone wanting to see the shutdown output in tee, run with tee -i to stop tee from dying on interrupt