[PPML] BigDL K8S Key Management Service (intel-analytics#6962)

* [PPML] BigDL KMS * Update README.md * Update README.md * Update README.md * develop according to test * recover path of readme * frontend * add frontend scala * implement keywhiz cli APIs * Update App.scala * refine * Implement create user API and add uninstall script * integrate Dockerfiles of frontend and keywhiz * refine * modify build codes after test * modify codes after maven test * refine after k8s deploy test * Delete build.sh * fix and add kms client * fix * refine after testing restAPI * refine after post/get dataKey test * add bkeywhiz kms example * refine readme * fix Readme typo * refactor * remove remaining password * delete password in log string * fix long line * Update README.md * Update gbtClassifierTrainingExampleOnCriteoClickLogsDataset.scala * fix * Update start-https-frontend.sh
ForJadeForest · Jan 10, 2023 · 438976f · 438976f
1 parent e3c1c44
commit 438976f
Show file tree

Hide file tree

Showing 21 changed files with 953 additions and 16 deletions.
diff --git a/ppml/services/bigdl-kms/docker/Dockerfile b/ppml/services/bigdl-kms/docker/Dockerfile
@@ -0,0 +1,115 @@
+ARG BIGDL_VERSION=2.2.0-SNAPSHOT
+ARG SPARK_VERSION=3.1.3
+
+# java
+FROM ubuntu:20.04 as java
+ARG JDK_VERSION=8u192
+ARG JDK_URL
+
+RUN apt-get update --fix-missing && \
+    env DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt-get install -y tzdata apt-utils wget unzip && \
+    wget $JDK_URL && \
+    gunzip jdk-$JDK_VERSION-linux-x64.tar.gz && \
+    tar -xf jdk-$JDK_VERSION-linux-x64.tar -C /opt && \
+    rm jdk-$JDK_VERSION-linux-x64.tar && \
+    mv /opt/jdk* /opt/jdk$JDK_VERSION && \
+    ln -s /opt/jdk$JDK_VERSION /opt/jdk
+
+# bigdl
+FROM ubuntu:20.04 as bigdl
+ARG BIGDL_VERSION
+ARG SPARK_VERSION
+ENV SPARK_VERSION               ${SPARK_VERSION}
+ENV BIGDL_VERSION               ${BIGDL_VERSION}
+ENV BIGDL_HOME                  /bigdl-${BIGDL_VERSION}
+RUN apt-get update --fix-missing && \
+    apt-get install -y apt-utils curl wget unzip git
+RUN wget https://raw.githubusercontent.com/intel-analytics/analytics-zoo/bigdl-2.0/docker/hyperzoo/download-bigdl.sh && \
+    chmod a+x ./download-bigdl.sh
+RUN ./download-bigdl.sh && \
+    rm bigdl*.zip
+
+# bigdl keywhiz
+FROM maven:3.6-jdk-11
+ARG http_proxy
+ARG https_proxy
+
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends --no-upgrade \
+    gettext vim-common default-mysql-server && \
+    echo validate_password.policy=LOW >> /etc/mysql/conf.d/mysql.cnf && \
+    mkdir -p /usr/src/app
+WORKDIR /usr/src/app
+
+# caching trick to speed up build, this should allow non-dynamic dependencies to be cached
+RUN git clone https://github.com/analytics-zoo/keywhiz.git -b BigDL-Keywhiz && \
+    cd keywhiz && \
+    mkdir -p /usr/src/app/api \
+             /usr/src/app/cli \
+             /usr/src/app/client \
+             /usr/src/app/hkdf \
+             /usr/src/app/model \
+             /usr/src/app/server \
+             /usr/src/app/testing \
+             /usr/src/app/log
+RUN cd keywhiz &&\
+    cp *.xml /usr/src/app/ && \
+    cp api/pom.xml /usr/src/app/api/ && \
+    cp cli/pom.xml /usr/src/app/cli/ && \
+    cp client/pom.xml /usr/src/app/client/ && \
+    cp hkdf/pom.xml /usr/src/app/hkdf/ && \
+    cp model/pom.xml /usr/src/app/model/ && \
+    cp server/pom.xml /usr/src/app/server/ && \
+    cp testing/pom.xml /usr/src/app/testing/ && \
+    cp log/pom.xml /usr/src/app/log/
+RUN rm /usr/share/maven/conf/settings.xml && \
+    cp /usr/src/app/keywhiz/settings.xml /usr/share/maven/conf/settings.xml && \
+    mvn dependency:copy-dependencies --fail-never
+
+# copy source required for build and install
+RUN cd keywhiz && \
+    cp -r api/* /usr/src/app/api/ && \
+    cp -r cli/* /usr/src/app/cli/ && \
+    cp -r client/* /usr/src/app/client/ && \
+    cp -r hkdf/* /usr/src/app/hkdf/ && \
+    cp -r model/* /usr/src/app/model/ && \
+    cp -r server/* /usr/src/app/server/ && \
+    cp -r testing/* /usr/src/app/testing/ && \
+    cp -r log/* /usr/src/app/log/
+RUN service mysql start && mvn -DskipTests=true package
+
+# Drop privs inside container
+RUN useradd -ms /bin/false keywhiz && \
+    mkdir /data && \
+    chown keywhiz:keywhiz /data && \
+    mkdir /secrets && \
+    chown keywhiz:keywhiz /secrets && \
+    echo 'alias keywhiz.cli="/usr/src/app/cli/target/keywhiz-cli-*-SNAPSHOT-shaded.jar --devTrustStore --url https://keywhiz-service:4444"' >> ~/.bashrc && \
+    echo 'alias key.provider="java -jar server/target/keywhiz-server-*-SNAPSHOT-shaded.jar"' >> ~/.bashrc && \
+    echo salt > /usr/src/app/salt
+
+EXPOSE 4444
+
+VOLUME ["/data", "/secrets"]
+
+RUN cd keywhiz && \
+    cp docker/entry.sh /usr/src/app && \
+    cp docker/wizard.sh /usr/src/app && \
+    cp docker/keywhiz-config.tpl /usr/src/app && \
+    cp frontend-keywhiz-conf.yaml /usr/src/app && \
+    chmod a+x /usr/src/app/entry.sh
+ADD ./start-https-frontend.sh /usr/src/app/start-https-frontend.sh
+RUN chmod a+x /usr/src/app/start-https-frontend.sh && \
+    chmod a+x /usr/src/app/server/target/keywhiz-server-0.10.2-SNAPSHOT-shaded.jar
+
+ENV LC_ALL	C.UTF-8
+ENV LANG	C.UTF-8
+ARG BIGDL_VERSION
+ARG SPARK_VERSION
+ENV SPARK_VERSION               ${SPARK_VERSION}
+ENV BIGDL_VERSION               ${BIGDL_VERSION}
+ENV BIGDL_HOME                  /bigdl-${BIGDL_VERSION}
+COPY --from=java /opt/jdk  /opt/jdk8
+COPY --from=bigdl /bigdl-${BIGDL_VERSION} ${BIGDL_HOME}
+
+ENTRYPOINT ["/usr/src/app/entry.sh"]
diff --git a/ppml/services/bigdl-kms/docker/README.md b/ppml/services/bigdl-kms/docker/README.md
@@ -0,0 +1,12 @@
+## Pull/Build container image
+
+Download image as below:
+
+```bash
+docker pull intelanalytics/bkeywhiz:2.2.0-SNAPSHOT
+```
+
+Or you are allowed to build the image manually:
+```
+# Note: set the arguments inside the build script first
+bash build-docker-image.sh
diff --git a/ppml/services/bigdl-kms/docker/build-docker-image.sh b/ppml/services/bigdl-kms/docker/build-docker-image.sh
@@ -0,0 +1,12 @@
+export HTTP_PROXY_HOST=your_http_proxy_host
+export HTTP_PROXY_PORT=your_http_proxy_port
+export HTTPS_PROXY_HOST=your_https_proxy_host
+export HTTPS_PROXY_PORT=your_https_proxy_port
+export JDK_URL=http://your-http-url-to-download-jdk
+
+sudo docker build \
+    --build-arg http_proxy=http://$HTTP_PROXY_HOST:$HTTP_PROXY_PORT \
+    --build-arg https_proxy=http://$HTTPS_PROXY_HOST:$HTTPS_PROXY_PORT \
+    --build-arg JDK_VERSION=8u192 \
+    --build-arg JDK_URL=$JDK_URL \
+    -t intelanalytics/bigdl-kms:2.2.0-SNAPSHOT -f ./Dockerfile .
diff --git a/ppml/services/bigdl-kms/docker/start-https-frontend.sh b/ppml/services/bigdl-kms/docker/start-https-frontend.sh
@@ -0,0 +1,41 @@
+#!/bin/bash
+
+#
+# Copyright 2016 The BigDL Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+
+set -x
+
+echo "[INFO] Launching BigDL KMS HTTPS Frontend"
+
+keywhiz_port=$KEYWHIZ_PORT
+https_key_store_path=/usr/src/app/server/src/main/resources/dev_and_test_keystore.p12
+https_secure_password=$HTTPS_SECURE_PASSWORD # k8s secret
+
+java \
+    -Xms2g \
+    -Xmx10g \
+    -Dcom.intel.analytics.zoo.shaded.io.netty.tryReflectionSetAccessible=true \
+    -Dakka.http.host-connection-pool.max-connections=100 \
+    -Dakka.http.host-connection-pool.max-open-requests=128 \
+    -Dakka.actor.default-dispatcher.fork-join-executor.parallelism-min=100 \
+    -Dakka.actor.default-dispatcher.fork-join-executor.parallelism-max=120 \
+    -Dakka.actor.default-dispatcher.fork-join-executor.parallelism-factor=1 \
+    -cp "$BIGDL_HOME/jars/*" \
+    com.intel.analytics.bigdl.ppml.kms.frontend.BigDLKMSFrontend \
+    --keywhizHost "keywhiz-service" \
+    --httpsKeyStorePath "${https_key_store_path}" \
+    --httpsKeyStoreToken "${https_secure_password}" | tee ./bkeywhiz-https-frontend.log
diff --git a/ppml/services/bigdl-kms/kubernetes/README.md b/ppml/services/bigdl-kms/kubernetes/README.md
@@ -0,0 +1,90 @@
+# Deploy BigDL KMS (Key Management Service) on Kubernetes
+
+## BigDL KMS Architecture
+![BigDLKMS](https://user-images.githubusercontent.com/60865256/211248045-dea5dac3-3169-4e02-b472-9cff901f4de7.jpg)
+
+
+**bigdl-kms-frontend**: The frontend REST API provider towards KMS user.
+
+**keywhiz**: Secret engine serving as a backend managing keys as secret.
+
+**mysql**: Encrypted storage to save keys.
+
+## Prerequests
+
+- Make sure you have a workable **Kubernetes cluster/machine**
+- Make sure you have a reachable **NFS**
+- Prepare [bigdl-keywhiz image](https://github.com/intel-analytics/BigDL/tree/main/ppml/services/BKeywhiz/docker#pullbuild-container-image)
+
+## Start BigDL KMS on Kubernetes
+Modify parameters in script `install-bigdl-kms.sh`:
+
+```
+nfsServerIP: your_nfs_server_ip                   --->   <the_IP_address_of_your_NFS_server>
+nfsPath: a_nfs_shared_folder_path_on_the_server   --->   <an_existing_shared_folder_path_on_NFS_server>
+......
+kmsIP: your_kms_ip_to_use_as                      --->   <an_unused_ip_address_in_your_subnetwork_to_assign_to_kms>
+```
+
+Then, deploy bigdl-kms on kubernetes by one command:
+
+```bash
+bash install-bigdl-kms.sh
+```
+
+Check the service whether it has successfully been running (it may take seconds).
+```bash
+kubectl get all -n bigdl-kms
+
+# you will get similar to below
+NAME                                      READY   STATUS    RESTARTS   AGE
+pod/bigdl-kms-frontend-6d6b5f87b6-jjm76   1/1     Running   0          4m56s
+pod/keywhiz-0                             2/2     Running   0          4m56s
+
+NAME                                 TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)          AGE
+service/bigdl-kms-frontend-service   LoadBalancer   10.103.34.88   <kmsIP>         9876:31634/TCP   4m56s
+service/keywhiz-service              ClusterIP      None           <none>          4444/TCP         4m56s
+
+NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
+deployment.apps/bigdl-kms-frontend   1/1     1            1           4m56s
+
+NAME                                            DESIRED   CURRENT   READY   AGE
+replicaset.apps/bigdl-kms-frontend-6d6b5f87b6   1         1         1       4m56s
+
+NAME                       READY   AGE
+statefulset.apps/keywhiz   1/1     4m56s
+```
+
+## Validate Status of BigDL KMS
+
+You can communicate with BKeywhiz KMS using client [BigDLKeyManagementService](https://github.com/intel-analytics/BigDL/blob/main/scala/ppml/src/main/scala/com/intel/analytics/bigdl/ppml/kms/BigDLManagementService.scala), or simply verify through requesting REST API like below:
+
+```
+curl -k -v "https://<kmsIP>:9876/" # default port of bigdl-kms is 9876 and can be configured in bigdl-kms.yaml
+
+# you will get similar to below
+welcome to BigDL KMS Frontend
+
+create a user like: POST /user/{userName}?token=a_token_string_for_the_user
+get a primary key like: POST /primaryKey/{primaryKeyName}?user=your_username&&token=your_token
+get a data key like: POST /dataKey/{dataKeyName}?primaryKeyName=the_primary_key_name&&user=your_username&&token=your_token
+get the data key like: GET /dataKey/{dataKeyName}?primaryKeyName=the_primary_key_name&&user=your_username&&token=your_token
+
+curl -X POST -k -v "https://<kmsIP>:9876/user/<userName>?token=<userToken>"
+user [<userName>] is created successfully!
+
+curl -X POST -k -v "https://<kmsIP>:9876/primaryKey/<primaryKeyName>?user=<userName>&&token=<userToken>"
+primaryKey [<primaryKeyName>] is generated successfully!
+
+curl -X POST -k -v "https://<kmsIP>:9876/dataKey/<dataKeyName>?user=<userName>&&token=<userToken>&&primaryKeyName=<primaryKeyName>"
+dataKey [<dataKeyName>] is generated successfully!
+
+curl -X GET -k -v "https://<kmsIP>:9876/dataKey/<dataKeyName>?user=<userName>&&token=<userToken>&&primaryKeyName=<primaryKeyName>"
+XY********Yw==
+
+```
+
+## Test KMS with PPML end-to-end example
+
+[LocalCryptoExample](https://github.com/intel-analytics/BigDL/tree/main/scala/ppml/src/main/scala/com/intel/analytics/bigdl/ppml/examples#localcryptoexample-with-bigdl-kms)
+[PPMLContext]() (TODO)