Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

move validate release to hugegraph-doc #2109

Merged
merged 9 commits into from
Mar 27, 2023
Merged

move validate release to hugegraph-doc #2109

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
e7c15e6
validate release support mulit os
z7658329 Feb 11, 2023
e82fc53
validate release support mulit os
z7658329 Feb 12, 2023
785dcdb
update validate-release.sh
z7658329 Feb 13, 2023
40a9a25
latest action refer doc validate-release.yml
z7658329 Feb 13, 2023
debcbdc
check wget command exists
z7658329 Feb 13, 2023
9c3e6ff
delete validate-release.yml
z7658329 Feb 17, 2023
22176d6
update validate-release.yml
z7658329 Feb 17, 2023
017bb90
remove validate-release.sh
z7658329 Feb 22, 2023
7666f8f
Merge branch 'master' into feat/validate_support_mulit_os
javeme Feb 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
240 changes: 228 additions & 12 deletions .github/workflows/validate-release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,21 +1,33 @@
name: Validate Apache Release

name: "Validate Apache Release"
#latest please refer: https://github.com/apache/incubator-hugegraph-doc/blob/master/.github/workflows/validate-release.yml
z7658329 marked this conversation as resolved.
Show resolved Hide resolved
z7658329 marked this conversation as resolved.
Show resolved Hide resolved
on:
workflow_dispatch:
inputs:
release_version:
required: true
default: '1.0.0'
deploy:
gpg_user:
required: true
default: false
type: boolean
default: 'imbajin'


push:
branches:
- 'release-*'
pull_request:
branches:
- 'release-*'

jobs:
build:
runs-on: ubuntu-latest
name: "Build On ${{ matrix.os }}(jdk-${{ matrix.java_version }})"
runs-on: ${{ matrix.os }}
env:
SCRIPT_PATH: hugegraph-dist/scripts/
URL_PREFIX: https://dist.apache.org/repos/dist/dev/incubator/hugegraph/
USER: ${{ inputs.gpg_user }}
# TODO: parse version from the running branch name & also adapt the input version
RELEASE_VERSION: ''
steps:
- name: Checkout source
uses: actions/checkout@v3
Expand Down Expand Up @@ -43,16 +55,220 @@ jobs:
restore-keys: |
${{ runner.os }}-yarn-

- name: 1. Download SVN Sources
run: |
rm -rf dist/${{ inputs.release_version }}
svn co ${URL_PREFIX}/${{ inputs.release_version }} dist/${{ inputs.release_version }}
cd dist/${{ inputs.release_version }} || exit

- name: 2. Check Environment & Import Public Keys
run: |
cd dist/${{ inputs.release_version }}
shasum --version 1>/dev/null || exit
gpg --version 1>/dev/null || exit

wget https://downloads.apache.org/incubator/hugegraph/KEYS || exit
echo "Import KEYS:" && gpg --import KEYS

echo -e "5\ny\n" | gpg --batch --command-fd 0 --edit-key $USER trust

echo "trust all pk"
for key in $(gpg --no-tty --list-keys --with-colons | awk -F: '/^pub/ {print $5}');
do
echo -e "5\ny\n" | gpg --batch --command-fd 0 --edit-key "$key" trust
done

# TODO: do we need svn & gpg environment?
- name: Test Building Source & Running
- name: 3. Check SHA512 & GPG Signature
run: |
bash $SCRIPT_PATH/validate-release.sh ${{ inputs.release_version }} ${{ matrix.java_version }}
- name: Test Running Binary
cd dist/${{ inputs.release_version }}
for i in *.tar.gz; do
echo "$i"
shasum -a 512 --check "$i".sha512 || exit
eval gpg "${GPG_OPT}" --verify "$i".asc "$i" || exit
done

- name: 4. Validate Source Packages
run: |
cd dist/${{ inputs.release_version }} && ls -lh ./*.tar.gz
for i in *src.tar.gz; do
echo "$i"
# 4.0 check the directory name include "incubating"
if [[ ! "$i" =~ "incubating" ]]; then
echo "The package name should include incubating" && exit 1
fi
tar xzvf "$i" || exit
cd "$(basename "$i" .tar.gz)" || exit

# 4.1 check the directory include "NOTICE" and "LICENSE" and "DISCLAIMER" file
if [[ ! -f "LICENSE" ]]; then
echo "The package should include LICENSE file" && exit 1
fi
if [[ ! -f "NOTICE" ]]; then
echo "The package should include NOTICE file" && exit 1
fi
if [[ ! -f "DISCLAIMER" ]]; then
echo "The package should include DISCLAIMER file" && exit 1
fi
# 4.2 ensure doesn't contains *GPL/BCL/JSR-275/RSAL/QPL/SSPL/CPOL/NPL1.*/CC-BY
# dependency in LICENSE and NOTICE file
COUNT=$(grep -E "GPL|BCL|JSR-275|RSAL|QPL|SSPL|CPOL|NPL1|CC-BY" LICENSE NOTICE | wc -l)
if [[ $COUNT -ne 0 ]]; then
grep -E "GPL|BCL|JSR-275|RSAL|QPL|SSPL|CPOL|NPL1.0|CC-BY" LICENSE NOTICE
echo "The package shouldn't include GPL* invalid dependency, but get $COUNT" && exit 1
fi

# 4.3 ensure doesn't contains empty directory or file
COUNT=$(find . -type d -empty | wc -l)
if [[ $COUNT -ne 0 ]]; then
find . -type d -empty
echo "The package shouldn't include empty directory, but get $COUNT" && exit 1
fi

# 4.4 ensure any file should less than 900kb & not include binary file
COUNT=$(find . -type f -size +900k | wc -l)
if [[ $COUNT -ne 0 ]]; then
find . -type f -size +900k
echo "The package shouldn't include file larger than 900kb, but get $COUNT" && exit 1
fi
COUNT=$(find . -type f | perl -lne 'print if -B' | grep -v *.txt | wc -l)
if [[ $COUNT -ne 0 ]]; then
find . -type f | perl -lne 'print if -B'
echo "The package shouldn't include binary file, but get $COUNT"
fi

# 4.5 test compile the packages
if [[ $JAVA_VERSION == 8 && "$i" =~ "computer" ]]; then
cd .. && echo "skip computer module in java8"
continue
fi
mvn package -DskipTests -ntp && ls -lh
cd .. || exit
done

- name: 5. Run Compiled Packages In Server
run: |
cd dist/${{ inputs.release_version }} && ls -lh
cd ./*hugegraph-incubating*src/*hugegraph*${{ inputs.release_version }} || exit
bin/init-store.sh && sleep 1
bin/start-hugegraph.sh && ls ../../
cd ../../ || exit

- name: 6. Run Compiled Packages In ToolChain (Loader & Tool & Hubble)
run: |
echo "TODO: separate script to test binary"
cd dist/${{ inputs.release_version }}
cd ./*toolchain*src || exit
ls -lh
cd ./*toolchain*${{ inputs.release_version }} || exit
ls -lh

# 6.1 load some data first
echo "test loader"
cd ./*loader*${{ inputs.release_version }} || exit
bin/hugegraph-loader.sh -f ./example/file/struct.json -s ./example/file/schema.groovy \
-g hugegraph || exit
cd .. || exit

# 6.2 try some gremlin query & api in tool
echo "test tool"
cd ./*tool*${{ inputs.release_version }} || exit
bin/hugegraph gremlin-execute --script 'g.V().count()' || exit
bin/hugegraph task-list || exit
bin/hugegraph backup -t all --directory ./backup-test || exit
cd .. || exit

# 6.3 start hubble and connect to server
echo "test hubble"
cd ./*hubble*${{ inputs.release_version }} || exit
cat conf/hugegraph-hubble.properties && bin/start-hubble.sh
cd ../../../ || exit
rm -rf ./*src* && ls -lh

- name: 7. Validate Binary Packages
run: |
cd dist/${{ inputs.release_version }}
for i in *.tar.gz; do
echo "$i"
# 7.0 check the directory name include "incubating"
if [[ ! "$i" =~ "incubating" ]]; then
echo "The package name should include incubating" && exit 1
fi
tar xzvf "$i" || exit

# 7.1 check root dir include "NOTICE"/"LICENSE"/"DISCLAIMER" files & "licenses" dir
cd "$(basename "$i" .tar.gz)" && ls -lh || exit
if [[ ! -f "LICENSE" ]]; then
echo "The package should include LICENSE file" && exit 1
fi
if [[ ! -f "NOTICE" ]]; then
echo "The package should include NOTICE file" && exit 1
fi
if [[ ! -f "DISCLAIMER" ]]; then
echo "The package should include DISCLAIMER file" && exit 1
fi
if [[ ! -d "licenses" ]]; then
echo "The package should include licenses dir" && exit 1
fi

# 7.2 ensure doesn't contains *GPL/BCL/JSR-275/RSAL/QPL/SSPL/CPOL/NPL1.*/CC-BY
# dependency in LICENSE/NOTICE and licenses/* files
COUNT=$(grep -r -E "GPL|BCL|JSR-275|RSAL|QPL|SSPL|CPOL|NPL1|CC-BY" LICENSE NOTICE licenses | wc -l)
if [[ $COUNT -ne 0 ]]; then
grep -r -E "GPL|BCL|JSR-275|RSAL|QPL|SSPL|CPQL|NPL1|CC-BY" LICENSE NOTICE licenses
echo "The package shouldn't include GPL* invalid dependency, but get $COUNT" && exit 1
fi

# 7.3 ensure doesn't contains empty directory or file
COUNT=$(find . -type d -empty | wc -l)
if [[ $COUNT -ne 0 ]]; then
find . -type d -empty
echo "The package shouldn't include empty directory, but get $COUNT" && exit 1
fi

cd - || exit
done

- name: 8. Validate Binary Packages(Start Server)
run: |
cd dist/${{ inputs.release_version }}
cd ./*hugegraph-incubating*${{ inputs.release_version }} || exit
bin/init-store.sh && sleep 1
# kill the HugeGraphServer process by jps
jps | grep HugeGraphServer | awk '{print $1}' | xargs kill -9
bin/start-hugegraph.sh && ls ../
cd - || exit

- name: 9. Validate Binary Packages(Start ToolChain(Loader/Tool/Hubble))
run: |
cd dist/${{ inputs.release_version }}
cd ./*toolchain*${{ inputs.release_version }} || exit
ls -lh

# 9.1 loader some data first
echo "test loader"
cd ./*loader*${{ inputs.release_version }} || exit
bin/hugegraph-loader.sh -f ./example/file/struct.json -s ./example/file/schema.groovy \
-g hugegraph || exit
cd - || exit

# 9.2 try some gremlin query & api in tool
echo "test tool"
cd ./*tool*${{ inputs.release_version }} || exit
bin/hugegraph gremlin-execute --script 'g.V().count()' || exit
bin/hugegraph task-list || exit
bin/hugegraph backup -t all --directory ./backup-test || exit
cd - || exit

# 9.3 start hubble and connect to server
echo "test hubble"
cd ./*hubble*${{ inputs.release_version }} || exit
# TODO: add hubble doc & test it
cat conf/hugegraph-hubble.properties
bin/stop-hubble.sh && bin/start-hubble.sh
cd - || exit

strategy:
fail-fast: false
matrix:
java_version: [ '8', '11' ]
java_version: [ '8','11' ]
# TODO: support windows-latest or other os in future
os: [ubuntu-latest, macos-latest]
1 change: 1 addition & 0 deletions hugegraph-dist/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,7 @@
<configuration>
<tasks>
<echo file="${project.basedir}/dist.sh">
wget --version 1>/dev/null || exit
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: use curl to replace it (and refer a better way to handle it -- like maven property)

wget https://github.com/swagger-api/swagger-ui/archive/refs/tags/v4.15.5.tar.gz
tar zxvf v4.15.5.tar.gz
echo "window.onload = function() { window.ui = SwaggerUIBundle({
Expand Down
Loading