Skip to content

Secure ZooKeeper HowTo (copied from ekoontz zookeeper wiki)

Jump to bottom Edit New page
Andor Molnár edited this page Jul 9, 2019 · 1 revision

Up and Running with Secure Zookeeper

Introduction

The following assumes a blank Linux installation of a yum-based distribution such as Redhat or CentOS; specifically, I wrote this using the Amazon Linux AMI instance type.

Install Packages

[ec2-user@domU-12-31-39-0E-7D-D1 ~]$ sudo yum -y install java-1.6.0-openjdk emacs git ant krb5-server

Note that we would use the Sun JDK (as opposed to OpenJDK as above) in production.

Also feel free to substitute your favorite editor for emacs in the above.

If you are a developer, add to the install line:

ant-junit

.. so that you can run ant test. In fact, you will not be able to run the complete ant test unless you install a lot of other C++-related packages; see Running Zookeeper's 'ant test' on AWS EC2 Linux.

Clone Zookeeper

Starting with release 3.4, Zookeeper has built-in the security features described here.

[ec2-user@domU-12-31-39-0E-7D-D1 ~]$ git clone git://github.com/apache/zookeeper.git

Build

[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ export JAVA_HOME=/usr/lib/jvm/java-openjdk
[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ export PATH=$JAVA_HOME/bin:$PATH
[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ ant clean compile

Setup Kerberos KDC 

[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ sudo kdb5_util create -s
Loading random data
Initializing database '/var/kerberos/krb5kdc/principal' for realm 'EXAMPLE.COM',
master key name 'K/[email protected]'
You will be prompted for the database Master Password.
It is important that you NOT FORGET this password.
Enter KDC database master key: 
Re-enter KDC database master key to verify:

Start Kerberos KDC Daemon

[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ sudo /etc/init.d/krb5kdc start

Starting Kerberos 5 KDC: [  OK  ]

Add Zookeeper Server principal

[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ sudo kadmin.local
Authenticating as principal root/[email protected] with password.
kadmin.local:  addprinc -randkey zookeeper/localhost
WARNING: no policy specified for zookeeper/[email protected]; defaulting to no policy
Principal "zookeeper/[email protected]" created.
kadmin.local:  ktadd -k /home/ec2-user/zookeeper/conf/zookeeper.keytab zookeeper/localhost

Add a Zookeeper client principal

kadmin.local:  addprinc -randkey myzkclient
WARNING: no policy specified for [email protected]; defaulting to no policy
Principal "[email protected]" created.
kadmin.local:  ktadd -k /home/ec2-user/zookeeper/conf/myzkclient.keytab myzkclient

Set keytab files ownership

..so that Zookeeper client and principal (running as ec2-user) can read them.

[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ sudo chown ec2-user /home/ec2-user/zookeeper/conf/*.keytab

Configure Clients to use localhost as KDC 

[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ cat /etc/krb5.conf 
[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 default_realm = EXAMPLE.COM
 dns_lookup_realm = false
 dns_lookup_kdc = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true

[realms]
 EXAMPLE.COM = {
  kdc = localhost
  admin_server = localhost
 }

[domain_realm]
 .example.com = EXAMPLE.COM
 example.com = EXAMPLE.COM

Test Kerberos Authentication with KDC : Zookeeper Server

[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ kinit -k -t conf/zookeeper.keytab zookeeper/localhost
[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ klist
Ticket cache: FILE:/tmp/krb5cc_222
Default principal: zookeeper/[email protected]

Valid starting     Expires            Service principal
04/15/11 18:37:20  04/16/11 18:37:20  krbtgt/[email protected]
	renew until 04/15/11 18:37:20

Test Kerberos Authentication with KDC : Zookeeper Client

[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ kinit -k -t conf/myzkclient.keytab  myzkclient
[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ klist
Ticket cache: FILE:/tmp/krb5cc_222
Default principal: [email protected]

Valid starting     Expires            Service principal
04/15/11 18:37:20  04/16/11 18:37:31  krbtgt/[email protected]
	renew until 04/15/11 18:37:31

Configure Zookeeper Server

[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ cat conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/home/ec2-user/zk-data
clientPort=2181
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
jaasLoginRenew=3600000

The jaasLoginRenew line in the above will cause the Zookeeper to renew its Kerberos ticket once an hour (3,600,000 milliseconds = 3600 seconds = 60 minutes).

[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ cat conf/java.env
export JVMFLAGS="-Djava.security.auth.login.config=/home/ec2-user/zookeeper/conf/jaas.conf"

Note that, as we’ve configured it here, a single JAAS configuration file, conf/jaas.conf is used for SASL authentication by both the Client and Server. Below we use export JVMFLAGS to specify the location of this JAAS configuration file. The two configuration sections “Client” and “Server” need not be in the same file, however: if you want to separate them, simply use different values for the -Djava.security.auth.login.config in separate SERVER_JVMFLAGS and CLIENT_JVMFLAGS lines in java.env.

[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ cat conf/jaas.conf
Server {
       com.sun.security.auth.module.Krb5LoginModule required
       useKeyTab=true
       keyTab="/home/ec2-user/zookeeper/conf/zookeeper.keytab"
       storeKey=true
       useTicketCache=false
       principal="zookeeper/localhost";
};
Client {
       com.sun.security.auth.module.Krb5LoginModule required
       useKeyTab=true
       keyTab="/home/ec2-user/zookeeper/conf/myzkclient.keytab"
       principal="myzkclient"
       useTicketCache=false
       debug=true;
};

_Note that I am using keytabs for clients in the above. You may also use: useTicketCache=true and useKeyTab=false. If so, you must do kinit with the appropriate arguments to initialize your ticket cache. You must also run kinit -R from your command line after this. This seems to be necessary due to a bug in the relevant Java libraries when parsing ticket cache. See this thread on the Kerberos development mailing list for more information.

Start Zookeeper Server

Test execution command-line with print-cmd (note the presence of -Djava.security.auth.login.config= : in your environment, make sure that its value points to the correct JAAS configuration file):

[[ec2-user@domU-12-31-39-14-39-61 zookeeper]$ bin/zkServer.sh print-cmd
JMX enabled by default
Using config: bin/../conf/zoo.cfg
java -Dzookeeper.log.dir="." -Dzookeeper.root.logger="INFO,CONSOLE" -cp "bin/../build/classes:bin/../build/lib/netty-
3.2.2.Final.jar:bin/../build/lib/log4j-1.2.15.jar:bin/../build/lib/jline-0.9.94.jar:bin/../zookeeper-*.jar:bin/../lib
/*.jar:bin/../src/java/lib/ivy-2.1.0.jar:bin/../conf:" -Djava.security.auth.login.config=/home/ec2-user/conf/jaas.con
f -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.Q
uorumPeerMain "bin/../conf/zoo.cfg"
[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ bin/zkServer.sh start-foreground

JMX enabled by default

Using config: bin/../conf/zoo.cfg

2011-01-27 20:25:54,413 [myid:] - INFO  [main:QuorumPeerConfig@94] - Reading configuration from: bin/../conf/zoo.cfg
2011-01-27 20:25:54,417 [myid:] - WARN  [main:QuorumPeerMain@106] - Either no config or no quorum defined in config, 
running  in standalone mode

At the bottom of your console you should see:

2011-02-09 21:19:17,260 [myid:] - INFO  [main:ServerCnxnFactory@204] - Using java.security.auth.login.config=/h
ome/ec2-user/zookeeper/conf/jaas.conf for doing server-side subject authentication.
2011-02-09 21:19:17,323 [myid:] - INFO  [main:ServerCnxnFactory@212] - Server successfully authenticated.
2011-02-09 21:19:17,347 [myid:] - INFO  [main:FileSnap@82] - Reading snapshot /home/ec2-user/zk-data/version-2/
snapshot.0
2011-02-09 21:19:17,351 [myid:] - INFO  [main:FileTxnSnapLog@208] - Snapshotting: 0

(in shell #2) Start Zookeeper Client

(Unlike the server’s command-line starter script bin/zkServer.sh, the client’s command-line starter script lacks a printcmd option).

[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ bin/zkCli.sh -server localhost:2181

You should see a line similar to the following in the client output:
2012-02-12 09:10:56,458 [myid:] - INFO  [main-SendThread(localhost.localdomain:2181):ZooKeeperSaslClient@98] - Found Login Context section 'Client': will use it to attempt to SASL-authenticate.

(in shell #1) Watch client activity in Zookeeper Server log:

2011-04-15 18:58:54,422 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@207] - Accepted socket connection from /127.0.0.1:36815
2011-04-15 18:58:54,450 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@801] - Client attempting to establish new session at /127.0.0.1:36815
2011-04-15 18:58:54,459 [myid:] - INFO  [SyncThread:0:FileTxnLog@197] - Creating new log file: log.1
2011-04-15 18:58:54,479 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@577] - Established session 0x12f5a7b20d80000 with negotiated timeout 30000 for client /127.0.0.1:36815
2011-04-15 18:58:54,628 [myid:] - INFO  [SyncThread:0:ServerCnxnFactory$SaslServerCallbackHandler@349] - Successfully authenticated client: [email protected];  [email protected].

Note the Successfully authenticated client line in the above log output.

(in shell #2) Create a node that only a certain SASL-authenticated client can read

[zk: (CONNECTED) 5] create /test content sasl:[email protected]:cdwra

(in shell #2) Test that ACLs were set correctly.

[zk: (CONNECTED) 6] getAcl /test
'sasl,'[email protected]
: cdrwa

Test authorization with authorized client.

Please see the following JUnit tests, which verify that SASL-authenticated clients are granted the correct access by the Zookeeper server to the nodes to which they’ve been authorized:

Test authorization failure with unauthorized client.

Please see the following JUnit tests, which verify that clients which are denied access to resources to which they have not been been authorized:

Add a custom footer
Add a custom sidebar
Clone this wiki locally