flux_rpc_then() callbacks must consume message or infinite loop

[garlick]

In the new Flux RPC design, a continuation callback that never calls flux_rpc_get() will trigger an infinite loop. This may be asking for nasty bugs down the line. Other alternatives are to make the get call and pass the results to the callback, or to destroy unconsumed responses after each callback.

This needs another look before we finalize the public API.

[grondo]

If nothing else, perhaps some kind of watchdog you could enable to at least detect this condition and spit out a warning. (Embarrassing to say how many times I've had these bugs in poll() loop based software -- So though it is possible in all event loop architectures to hit this kind of bug, it would be pretty nice if we could at least provide a mechanism to detect it, i.e. something like kernel hung_task_timeout.)

[garlick]

This is no longer the case.

A continuation callback that doesn't call flux_rpc_get() simply misses that message.