From f75e983045f4b64958dc0e93e8b94a7cfd7663be Mon Sep 17 00:00:00 2001 From: Mika Naylor Date: Mon, 16 Aug 2021 17:28:15 +0200 Subject: [PATCH] [FLINK-23756] Update FrocksDB release document with more info Also make some slight improvements to the Maven upload script. --- FROCKSDB-RELEASE.md | 244 +++++++++++++++++++++++++++--------- java/publish-frocksdbjni.sh | 8 +- 2 files changed, 188 insertions(+), 64 deletions(-) diff --git a/FROCKSDB-RELEASE.md b/FROCKSDB-RELEASE.md index 9538817f7..3ec3c2724 100644 --- a/FROCKSDB-RELEASE.md +++ b/FROCKSDB-RELEASE.md @@ -1,94 +1,190 @@ +# FRocksDB Release Process + ## Summary -For FRocksDB-6.x, we need to release jar package which contains .so files for linux32 and linux64 (glibc and musl-libc), jnilib files for Mac OSX, and a .dll for Windows x64. -## Build in Windows +FrocksDB-6.x releases are a fat jar file that contain the following binaries: +* .so files for linux32 (glibc and musl-libc) +* .so files for linux64 (glibc and musl-libc) +* .so files for linux [aarch64](https://en.wikipedia.org/wiki/AArch64) (glibc and musl-libc) +* .so files for linux [ppc64le](https://en.wikipedia.org/wiki/Ppc64le) (glibc and musl-libc) +* .jnilib file for Mac OSX +* .dll for Windows x64 -Use Windows 64 bit machine (e.g. base AWS Windows instance: 4 cores, 16GB RAM, 40GB storage for build). +To build the binaries for a FrocksDB release, building on native architectures is advised. Building the binaries for ppc64le and aarch64 *can* be done using QEMU, but you may run into emulation bugs and the build times will be dramatically slower (up to x20). -Install: - * git - * java 8 - * maven - * Visual Studio Community 15 (2017) +We recommend building the binaries on environments with at least 4 cores, 16GB RAM and 40GB of storage. The following environments are recommended for use in the build process: +* Windows x64 +* Linux aarch64 +* Linux ppc64le +* Mac OSX -With [chocolatey](https://chocolatey.org/install): +## Build for Windows - choco install git.install jdk8 maven visualstudio2017community +For the Windows binary build, we recommend using a base [AWS Windows EC2 instance](https://aws.amazon.com/windows/products/ec2/) with 4 cores, 16GB RAM, 40GB storage for the build. -Optionally: +Firstly, install [chocolatey](https://chocolatey.org/install). Once installed, the following required components can be installed using Powershell: - choco install intellijidea-community vscode + choco install git.install jdk8 maven visualstudio2017community visualstudio2017-workload-nativedesktop -Open developer command prompt for vs 2017 and run commands: +Open the "Developer Command Prompt for VS 2017" and run the following commands: git clone git@github.com:ververica/frocksdb.git cd frocksdb git checkout FRocksDB-6.20.3 # release branch java\crossbuild\build-win.bat -The result native library is `build\java\Release\rocksdbjni-shared.dll` and you can also find it under project foler with name `librocksdbjni-win64.dll` +The resulting native binary will be built and available at `build\java\Release\rocksdbjni-shared.dll`. You can also find it under project folder with name `librocksdbjni-win64.dll`. The result windows jar is `build\java\rocksdbjni_classes.jar`. There is also a how-to in CMakeLists.txt. +**Once finished, extract the `librocksdbjni-win64.dll` from the build environment. You will need this .dll in the final crossbuild.** + +## Build for aarch64 + +For the Linux aarch64 binary build, we recommend using a base [AWS Ubuntu Server 20.04 LTS EC2](https://aws.amazon.com/windows/products/ec2/) with a 4 core Arm processor, 16GB RAM, 40GB storage for the build. You can also attempt to build with QEMU on a non-aarch64 processor, but you may run into emulation bugs and very long build times. + +### Building in aarch64 environment + +First, install the required packages such as Java 8 and make: + + sudo apt-get update + sudo apt-get install build-essential openjdk-8-jdk + +then, install and setup [Docker](https://docs.docker.com/engine/install/ubuntu/): + + sudo apt-get install apt-transport-https ca-certificates curl gnupg lsb-release + + curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg + echo "deb [arch=arm64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null + + sudo apt-get update + sudo apt-get install docker-ce docker-ce-cli containerd.io + + sudo groupadd docker + sudo usermod -aG docker $USER + newgrp docker + +Then, clone the FrocksDB repo: + + git clone https://github.com/ververica/frocksdb.git + cd frocksdb + git checkout FRocksDB-6.20.3 # release branch + + +First, build the glibc binary: + + make jclean clean rocksdbjavastaticdockerarm64v8 + +**Once finished, extract the `java/target/librocksdbjni-linux-aarch64.so` from the build environment. You will need this .so in the final crossbuild.** + +Next, build the musl-libc binary: + + make jclean clean rocksdbjavastaticdockerarm64v8musl + +**Once finished, extract the `java/target/librocksdbjni-linux-aarch64-musl.so` from the build environment. You will need this .so in the final crossbuild.** + +### Building via QEMU + +You can use QEMU on, for example, an `x86_64` system to build the aarch64 binaries. To set this up on an Ubuntu envirnment: + + sudo apt-get install qemu binfmt-support qemu-user-static + docker run --rm --privileged multiarch/qemu-user-static --reset -p yes + +To verify that you can now run aarch64 docker images: + + docker run --rm -t arm64v8/ubuntu uname -m + > aarch64 + +You can now attempt to build the aarch64 binaries as in the previous section. + ## Build in PPC64LE -We can build binaries on ppc64le platform in two ways: build on a powerPC machine directly or build within a docker machine via QEMU. -I strongly suggest to build on a powerPC machine. -### Build on a powerPC machine -Run commands separately to get binaries. +For the ppc64le binaries, we recommend building on a PowerPC machine if possible, as it can be tricky to spin up a ppc64le cloud environment. However, if a PowerPC machine is not available, [Travis-CI](https://www.travis-ci.com/) offers ppc64le build environments that work perfectly for building these binaries. If neither a machine or Travis are an option, you can use QEMU but the build may take a very long time and be prone to emulation errors. + +### Building in ppc64le environment + +As with the aarch64 environment, the ppc64le environment will require Java 8, Docker and build-essentials installed. Once installed, you can build the 2 binaries: make jclean clean rocksdbjavastaticdockerppc64le + +**Once finished, extract the `java/target/librocksdbjni-linux-ppc64le.so` from the build environment. You will need this .so in the final crossbuild.** + make jclean clean rocksdbjavastaticdockerppc64lemusl -### Build within a docker machine via QEMU -**warning** It would be extremely slow to build within a docker machine via QEMU, from my experiense, it might need at least 8 hours to build FRocksDB once. +**Once finished, extract the `java/target/librocksdbjni-linux-ppc64le-musl.so` from the build environment. You will need this .so in the final crossbuild.** -Use Ubuntu 16.04 (e.g. AWS instance 4 cores, 16GB RAM, 40GB storage for build). -Install git if not installed. If docker is installed, it might need to be removed. +### Building via Travis -Setup ppc64le docker machine ([source](https://developer.ibm.com/linuxonpower/2017/06/08/build-test-ppc64le-docker-images-intel/)): +Travis-CI supports ppc64le build environments, and this can be a convienient way of building in the absence of a PowerPC machine. Assuming that you have an S3 bucket called **my-frocksdb-release-artifacts**, the following Travis configuration will build the release artifacts and push them to the S3 bucket: - wget http://ftp.unicamp.br/pub/ppc64el/boot2docker/install.sh && chmod +x ./install.sh && ./install.sh -s - docker-machine create -d qemu \ - --qemu-boot2docker-url=/home/ubuntu/.docker/machine/boot2docker.iso \ - --qemu-memory 8192 \ - --qemu-cpu-count 4 \ - --qemu-cache-mode none \ - --qemu-arch ppc64le \ - vm-ppc64le +``` +dist: xenial +language: cpp +os: + - linux +arch: + - ppc64le + +services: + - docker +addons: + artifacts: + paths: + - $TRAVIS_BUILD_DIR/java/target/librocksdbjni-linux-ppc64le-musl.so + - $TRAVIS_BUILD_DIR/java/target/librocksdbjni-linux-ppc64le.so + +env: + global: + - ARTIFACTS_BUCKET=my-rocksdb-release-artifacts + jobs: + - CMD=rocksdbjavastaticdockerppc64le + - CMD=rocksdbjavastaticdockerppc64lemusl + +install: + - sudo apt-get install -y openjdk-8-jdk || exit $? + - export PATH=/usr/lib/jvm/java-8-openjdk-$(dpkg --print-architecture)/bin:$PATH + - export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-$(dpkg --print-architecture) + - echo "JAVA_HOME=${JAVA_HOME}" + - which java && java -version + - which javac && javac -version + +script: + - make jclean clean $CMD +``` -Regenerate certs as suggested if it did not work at once. +**Make sure to set the `ARTIFACTS_KEY` and `ARTIFACTS_SECRET` environment variables in the Travis Job with valid AWS credentials to access the S3 bucket you defined.** -Prepare docker machine to run rocksdbjni docker image for ppc64le build: +**Once finished, the`librocksdbjni-linux-ppce64le.so` and `librocksdbjni-linux-ppce64le-musl.so` binaries will be in the S3 bucket. You will need these .so binaries in the final crossbuild.** - eval $(docker-machine env vm-ppc64le) - git clone git@github.com:ververica/frocksdb.git - cd frocksdb - git checkout FRocksDB-6.20.3 # release branch - docker-machine ssh vm-ppc64le mkdir -p `pwd` - docker-machine scp -r . vm-ppc64le:`pwd` -Build frocksdb: +### Building via QEMU - make jclean clean rocksdbjavastaticdockerppc64le - docker-machine scp vm-ppc64le:`pwd`/java/target/librocksdbjni-linux-ppc64le.so java/target/. - make jclean clean rocksdbjavastaticdockerppc64lemusl - docker-machine scp vm-ppc64le:`pwd`/java/target/librocksdbjni-linux-ppc64le.so java/target/. +You can use QEMU on, for example, an `x86_64` system to build the ppc64le binaries. To set this up on an Ubuntu envirnment: + + sudo apt-get install qemu binfmt-support qemu-user-static + docker run --rm --privileged multiarch/qemu-user-static --reset -p yes -The result native libraries are `java/target/librocksdbjni-linux-ppc64le.so` and `java/target/librocksdbjni-linux-ppc64le-musl.so`. +To verify that you can now run ppc64le docker images: -## Build on ARM machines -Run commands separately to get binaries. + docker run --rm -t ppc64le/ubuntu uname -m + > ppc64le - make jclaen clean rocksdbjavastaticdockerarm64v8 - make jclaen clean rocksdbjavastaticdockerarm64v8musl +You can now attempt to build the ppc64le binaries as in the previous section. ## Final crossbuild in Mac OSX -Read how to Build cross jar for Mac OSX and linux as described in java/RELEASE.md but do not run it yet. +Documentation for the final crossbuild for Mac OSX and Linux is described in [java/RELEASE.md](java/RELEASE.md) as has information on dependencies that should be installed. As above, this tends to be Java 8, build-essentials and Docker. -Run commands: +Before you run this step, you should have 5 binaries from the previous build steps: + + 1. `librocksdbjni-win64.dll` from the Windows build step. + 2. `librocksdbjni-linux-aarch64.so` from the aarch64 build step. + 3. `librocksdbjni-linux-aarch64-musl.so` from the aarch64 build step. + 3. `librocksdbjni-linux-ppc64le.so` from the ppc64le build step. + 4. `librocksdbjni-linux-ppc64le-musl.so` from the ppc64le build step. + +To start the crossbuild within a Mac OSX environment: make jclean clean mkdir -p java/target @@ -97,19 +193,48 @@ Run commands: cp /librocksdbjni-linux-ppc64le-musl.so java/target/librocksdbjni-linux-ppc64le-musl.so cp /librocksdbjni-linux-aarch64.so java/target/librocksdbjni-linux-aarch64.so cp /librocksdbjni-linux-aarch64-musl.so java/target/librocksdbjni-linux-aarch64-musl.so - FROCKSDB_VERSION=1.0 PORTABLE=1 ROCKSDB_DISABLE_JEMALLOC=true make clean frocksdbjavastaticreleasedocker + FROCKSDB_VERSION=1.0 PORTABLE=1 ROCKSDB_DISABLE_JEMALLOC=true DEBUG_LEVEL=0 make frocksdbjavastaticreleasedocker + +*Note, we disable jemalloc on mac due to https://github.com/facebook/rocksdb/issues/5787*. + +Once finished, there should be a directory at `java/target/frocksdb-release` with the FRocksDB jar, javadoc jar, sources jar and pom in it. You can inspect the jar file and ensure that contains the binaries, history file, etc: + +``` +$ jar tf frocksdbjni-6.20.3-ververica-1.0.jar +META-INF/ +META-INF/MANIFEST.MF +HISTORY-JAVA.md +HISTORY.md +librocksdbjni-linux-aarch64-musl.so +librocksdbjni-linux-aarch64.so +librocksdbjni-linux-ppc64le-musl.so +librocksdbjni-linux-ppc64le.so +librocksdbjni-linux32-musl.so +librocksdbjni-linux32.so +librocksdbjni-linux64-musl.so +librocksdbjni-linux64.so +librocksdbjni-osx.jnilib +librocksdbjni-win64.dl +... +``` + +*Note that it contains linux32/64.so binaries as well as librocksdbjni-osx.jnilib*. + +## Push to Maven Central -* Note, we disable jemalloc on mac due to https://github.com/facebook/rocksdb/issues/5787 +For this step, you will need the following: -## Push to maven central +- The OSX Crossbuild artifacts built in `java/target/frocksdb-release` as above. +- A Sonatype account with access to the staging repository. If you do not have permission, open a ticket with Sonatype, [such as this one](https://issues.sonatype.org/browse/OSSRH-72185). +- A GPG key to sign the release, with your public key available for verification (for example, by uploading it to https://keys.openpgp.org/) -Run: +To upload the release to the Sonatype staging repository: ```bash VERSION= \ USER= \ PASSWORD= \ -KEYNAME= \ -PASSPHRASE= \ +KEYNAME= \ +PASSPHRASE= \ java/publish-frocksdbjni.sh ``` @@ -119,7 +244,6 @@ https://oss.sonatype.org/#stagingRepositories Select the open staging repository and click on "Close". -Test the files in the staging repository -which will look something like this `https://oss.sonatype.org/content/repositories/xxxx-1020`. +The staging repository will look something like `https://oss.sonatype.org/content/repositories/xxxx-1020`. You can use this staged release to test the artifacts and ensure they are correct. -Press the "Release" button (WARNING: this can not be undone). \ No newline at end of file +Once you have verified the artifacts are correct, press the "Release" button. **WARNING: this can not be undone**. Within 24-48 hours, the artifact will be available on Maven Central for use. diff --git a/java/publish-frocksdbjni.sh b/java/publish-frocksdbjni.sh index 474c62e50..2a6bd2865 100644 --- a/java/publish-frocksdbjni.sh +++ b/java/publish-frocksdbjni.sh @@ -26,19 +26,19 @@ function deploy() { FILE=$1 CLASSIFIER=$2 echo "Deploying file=${FILE} with classifier=${CLASSIFIER} to sonatype with prefix=${PREFIX}" - sonatype_user=${USER} sonatype_pw=${PASSWORD} mvn gpg:sign-and-deploy-file \ + sonatype_user="${USER}" sonatype_pw="${PASSWORD}" mvn gpg:sign-and-deploy-file \ --settings java/deploysettings.xml \ -Durl=https://oss.sonatype.org/service/local/staging/deploy/maven2/ \ -DrepositoryId=sonatype-nexus-staging \ -DpomFile=${PREFIX}.pom \ -Dfile=$FILE \ -Dclassifier=$CLASSIFIER \ - -Dgpg.keyname=${KEYNAME} \ - -Dgpg.passphrase=${PASSPHRASE} + -Dgpg.keyname="${KEYNAME}" \ + -Dgpg.passphrase="${PASSPHRASE}" } PREFIX=java/target/frocksdb-release/frocksdbjni-${VERSION} deploy ${PREFIX}-sources.jar sources deploy ${PREFIX}-javadoc.jar javadoc -deploy ${PREFIX}.jar \ No newline at end of file +deploy ${PREFIX}.jar