Flux API: replace zmsg_t with new opaque type

[garlick]

The Flux API should not include parameters of type zmsg_t as this forces a czmq/zmq dependency on all users and forgoes some opportunities for optimization of common Flux use cases (such as matching topic sub-strings) within the message class. Instead introduce a new opaque type flux_msg_t (or similar) and revise API to use that instead.

This is an opportunity to refine the message API and so let's agree on interfaces here before proceeding. The low-level headers in question are

src/common/libflux/message.h
src/common/libflux/request.h
src/common/libflux/handle.h
src/common/libflux/event.h
src/common/libflux/reactor.h

Other interfaces built on these may have to change too but let's tackle the low level first.

[garlick]

See issue #124 for some discussion on reactor API revision

[garlick]

The fact that zmsg_send() takes a (zmsg_t **) and destroys it on success has proliferated though the API. @grondo and I had discussed changing this so that messages are not destroyed by the send function.

[grondo]

Ah yes. This tripped me up because at one point I was trying to save a zmsg_t so that I could send multiple replies to the same sender. If we had a more natural way to connect two endpoints for a multiple message conversation, that issue at least would be resolved. (It is possible, though that in this case i was just doing it wrong) 😦

[garlick]

I'm for losing that idiom. There are few places where we do things like send a response if the zmsg hasn't been destroyed in a request handler and this just gets confusing.

[trws]

I agree, that particular idiom is rather obtuse. How best to deal with it I'm not sure though. Duplicating the message every time it gets passed to CZMQ doesn't feel like the right answer, but I'm not sure there's another one.

I'm putting together a meta-issue for the public API issues at #170.

[garlick]

Thanks @trws for your work on the trackers and milestones.

zmsg_send() can be avoided by building on a lower level function like:

int zmq_send (void *socket, void *buf, size_t len, int flags);

I think that'd be the way to squash the behavior without having to zmsg_dup() messages.

[garlick]

PR #219 solved this for libflux-core. Closing.