check_zookeeper_znode.pl unable to talk to a secure cluster.

[hcoyote]

This is a feature request, as opposed to a specific bug. I haven't dug into trying to fix this yet.

Right now, this tool seems to work great if you're not using Kerberos or SASL-enabled zookeeper cluster. If you are, the zk client created isn't able to hit the particular znodes for things like mrzkfc, zkfc, or hbase because that data ends up being protected and ACL'd by default to particular users.

It would be great if this could be enabled so these monitoring tools would work on Kerberos-enabled clusters.

[hcoyote]

So, I've started looking at this. Net::ZooKeeper already has the support for accessing things that are acl'd as of 0.35 so this looks reasonably easy to deal with.

The issue that I'm trying to understand next is that there appear to be two different mechanisms for dealing with zookeeper acls. The first one is using digest mode, which is what's documented for use with zkfc/mrzkfc (at least on Cloudera). The second way appears to utilize appropriately configuring JAAS and Kerberos so that zookeeper can auth the accessing user against the kerberos domain. HBase appears to utilize this method on a cluster where security is enabled.

In both cases, I think can set up a read-only acl for a user that uses the digest mode to read the ACL'd znodes. This seems like the easiest way to get monitoring access for using check_zookeeper_znode.pl in a secured environment. The downside is that when using this method, it appears that the digest info is sent over the clear (but I think it would be easier than trying to figure out how to get perl to deal with KRB at this point).

Any thoughts on this approach?

[HariSekhon]

Sorry I haven't gotten around to properly looking in to this, can you confirm the platform you're testing this on and the error?

I'm hoping I'll get some time over Christmas to test and work on this. On some platforms this actually does work... but that may be because their zookeepers aren't secured the same way (or perhaps not properly at all).

I think the issue you're describing is simply an ACL issue in ZooKeeper?

I did look at this briefly before but I don't think the Net::ZooKeeper Perl API supports Kerberos from what I could tell since there was no mention of it on the CPAN page, although it does appear to have acl support.

Also steps to reproduce, ie the plugin command line you're running, which znode you're targeting etc would be helpful so I can test this.

[HariSekhon]

apache/zookeeper#1243