rc3: exits immediately on command failure, resulting in many errors

[grondo]

Since rc3 is run with -e, it exits immediately on the first command failure. When flux-sched is installed along with flux-core, the qmanager-start rc1 script removes sched-simple in order to load the qmanager module, but this results in flux module remove sched-simple failing for every instance. Since rc3 then exits before unloading all other modules.

E.g. on fluke:

 grondo@fluke108:~$ flux start flux getattr size
1
2020-01-24T21:17:30.234811Z broker.err[0]: rc3: flux-module: cmb.rmmod sched-simple: No such file or directory
2020-01-24T21:17:30.235136Z broker.err[0]: Run level 3 Exited with non-zero status (rc=1) 0.0s
flux-broker: module 'cron' was not cleanly shutdown
flux-broker: module 'kvs-watch' was not cleanly shutdown
flux-broker: module 'job-exec' was not cleanly shutdown
flux-broker: module 'job-manager' was not cleanly shutdown
flux-broker: module 'userdb' was not cleanly shutdown
flux-broker: module 'job-ingest' was not cleanly shutdown
flux-broker: module 'aggregator' was not cleanly shutdown
flux-broker: module 'content-sqlite' was not cleanly shutdown
flux-broker: module 'job-info' was not cleanly shutdown
flux-broker: module 'kvs' was not cleanly shutdown
flux-broker: module 'barrier' was not cleanly shutdown

I can't recollect the rationale for running rc3 under bash -e. If there isn't a good reason, I propose we remove that flag. I think we want rc3 to try harder to run to completion.

This may be fixed by upcoming initialization changes. However, I would consider the current flux RPMs broken, so perhaps we should throw in a fix for this issue and generate some new packages.

[garlick]

Sorry to come into this late. This was my bad - the behavior of flux module remove changed in e42826e when mrpc was dropped. Before that, it reported errors on stderr but still exited with success. I stumbled across that and "fixed" it without thinking through these ramifications. Sorry about that.

The benefit of -e is to make the instance fail if something important like flushing the content cache to disk fails. Exiting immediately when anything goes wrong is not a great way to handle that, but ignoring all errors may be going too far. I'm fine with dropping the -e now and restoring some error handling as part of resolving #2650. Does that make sense?

[grondo]

Yes, that seems reasonable!