FASSERT, flux_assfail(), flux_panic() need review

[garlick]

Current FASSERT macro calls flux_assfail(), which logs its assertion failure as a LOG_CRIT, sleeps 5 seconds, then rasises sigabort. A lot can happen in 5s that may obscure the original error. The session could even initiate normal shutdown (and be successful due to the way modules are not unloaded gracefully during shutdown)

flux_panic() uses a separate mechanism that causes the local broker to print a message on stderr and exit.

There should be a way to panic a comms session that is abrupt, final, and reliably reports the first cause.

[garlick]

As noted in PR #215, we need a way to panic an application when a handle error, ENOMEM, or other fatal error occurs, that does not depend on a healthy channel to the broker as flux_panic() currently does. Errors of this sort in thread-based comms modules could be redirected to stderr and thus captured by the enclosing instance. Maybe flux_panic() needs some additional hints either via some state in the handle, or a flag passed in, to determine the best course of action. Or maybe flux_log() errors of a certain severity should be Cc'ed to stderr to ensure they get out?

[garlick]

#321 is a slightly more modern issue that discusses making a pass through these functions. Closing this one.