build: verify that zeromq/czmq built with security, epgm

[garlick]

from: Mark Grondona [email protected]
to: Jim Garlick [email protected]
date: Sun, Sep 28, 2014 at 11:12 AM

On my laptop running Ubuntu 14.04.1 with

libzmq3-dev:amd64 4.0.4+dfsg-2
libmunge-dev 0.5.11-1ubuntu1

And czmq built from git and installed in /usr/local.. I'm getting
zeroed keys for curve, which seems to be causing an assert
when starting cmbd. I assume I'm missing some dependency
or didn't build something right, but I'm not sure where to start
looking...

$ ./flux keygen
Saving /home/grondo/.flux/curve/client
Saving /home/grondo/.flux/curve/client_private
Saving /home/grondo/.flux/curve/server
Saving /home/grondo/.flux/curve/server_private

grondo@moron:~/git/flux-core/src/cmd$ cat /home/grondo/.flux/curve/server
#   ****  Generated on 2014-09-28 11:09:06 by CZMQ  ****
#   ZeroMQ CURVE Public Certificate
#   Exchange securely, or use a secure mechanism to verify the contents
#   of this file after exchange. Store public certificates in your home
#   directory, in the .curve subdirectory.

metadata
    time = "2014-09-28T11:09:06"
    role = "server"
curve
    public-key = "0000000000000000000000000000000000000000"

$ ./flux start --size=1
cmbd: zcertstore.c:173: zcertstore_insert: Assertion `rc == 0' failed.
flux-start: 0 (pid 12491) Aborted

from: Jim Garlick [email protected]
to: Mark Grondona [email protected]
date: Sun, Sep 28, 2014 at 12:27 PM

The security stuff came in zeromq4 so you'll need to build that too.
I think the steps are:
-build and install libsodium to /usr/local (me: 0.5.0)
-build and install libzmq (me: 4.0.4), I think you need --with-pgm
(but it is now included in the source tree)
-build and install czmq (me: 2.2.0)

Sounds like we might want to build in checks for functioning pgm and
curve to our configure.ac.
(Maybe open an issue on that?)

from: Mark Grondona [email protected]
to: Jim Garlick [email protected]
date: Sun, Sep 28, 2014 at 12:44 PM

The libzmq3 in Ubuntu is confusingly zeromq 4.0.4 (see version of package). Maybe not built with libsodium? Would runtime checks be possible so that keygen doesn't silently create zero keys?

[grondo]

My guess is we might be hitting something like zeromq/czmq #325

which was fixed in this commit (At least zcert_new() would return NULL)

zeromq/czmq@6009411

However, that behavior has since changed, which is why I'm still getting zeroed keys from
czmq git master.

As a workaround (though I admit an unsatisfying one), we could check for a cert with zero keys
as I did here:

grondo@76601b5

Or, as I just noticed is used in CZMQ, judicious use of zsys_has_curve() might be enough
to detect at runtime if we HAVE_ZAUTH but don't have Curve support.

(Sorry, a bit new at this code and missed the obvious the first time)

[garlick]

Possibly we could call this and look for a return of -1 and errno == ENOTSUPP?

http://api.zeromq.org/4-0:zmq-curve-keypair

Does that make sense as an autoconf test?

[grondo]

The autoconf test would be a definite improvement.

Since you're building with CZMQ anyway, zsys_has_curve() might be easier to use,
but I'm not sure if it is reliable yet against libzmq without zmq_has().

There is still a chance of generating zero keys silently at runtime though, since
one could configure against working libzmq and run against non-working libzmq.
Therefore, best case would be to detect this at runtime. Best best case would be
if zcert_new() failed instead of creating zero keys (which cause an assert at runtime
in other parts of the code)

[garlick]

Good point, agreed we need a runtime check.

I think zsys_has_curve() is pretty new (newer than czmq 2.2.0).
Maybe AC_CHECK_FUNCS for that and test it at runtime if available?
And if not fall back to your null cred check code?

[grondo]

I have a version that uses zsys_has_curve() if available and falls back to
the null cred check if not, but now it occurs to me -- why don't we just use
zmq_curve_keypair() and zcert_new_from() in gencurve() instead of
zcert_new()? This would be much more straightforward, unless you had a
reason to avoid zmq_curve_keypair()?

[garlick]

No, I don't think I did. Most likely I was looking at the czmq manual nd it just worked so I never got to the underlying stuff. Sorry I didn't think of that!

[grondo]

Ok, I tested this out in grondo@8970b23

I'll try it on my laptop tonight and generate a pull request if it works. Please comment if you have the chance.

[garlick]

I commented on the commit but I'll say it here too: looks good and thank you!

[grondo]

Build time check might be overkill? But I propose one above....

[garlick]

Yes let's do both! Nice.

[garlick]

I tried regenerating my keys (flux keygen -f) after applying these changes and although they superficially look OK, zmq connections can no longer be made. No problem if keys were generated with the old code. Will have a look tomorrow. This is on my ubuntu test box.

[garlick]

The code for zcert_new() calls crypto_box_keypair () (I guess a libsodium call) and passes the result to zcert_new_from(). Maybe zcert_new_from() wants keys unarmored?

[grondo]

Oh, sorry! I did try to test my branch, but obviously failed miserably.

I think you are right about z85 armor. Now it seems obvious why zcert_new_from() takes byte *..
:-(

[grondo]

Test with branch grondo/flux-core:zcert_curve_new-fixes to make sure I've resolved the issue with curve keys. I tested with

$ cat t.sh
#!/bin/sh
sleep 1
./flux up

$ ./flux keygen --force && ./flux start --size=2 sh t.sh 2>/dev/null | grep -q '^ok:.*\[0-1\]' && echo "Success" || echo "Failed."

Saving /g/g0/grondo/.flux/curve/client
Saving /g/g0/grondo/.flux/curve/client_private
Saving /g/g0/grondo/.flux/curve/server
Saving /g/g0/grondo/.flux/curve/server_private
Success

[grondo]

Closing this one as the work has been done and merged I think.