-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add orca horovod tf2 test to github actions #5601
Merged
Merged
Changes from all commits
Commits
Show all changes
34 commits
Select commit
Hold shift + click to select a range
6e2f762
add orca horovod tf2 test to github actions
pinggao18 81d524a
update pull_request
pinggao18 79c352d
update
pinggao18 70cd169
add keras
pinggao18 398c820
update python version
pinggao18 2ce03f1
update horovod
pinggao18 aa58b5f
update cmake
pinggao18 8fd3fd4
install bigdl-orca-spark3
pinggao18 1c2dab7
update
pinggao18 0c9ddfd
update mxnet
pinggao18 7036e26
update
pinggao18 dc17aaf
update numpy
pinggao18 4458bba
update
pinggao18 0aa21a2
update ray version
pinggao18 3ebd4d6
unintsall intel-tensorflow
pinggao18 21d09b3
add conda env
pinggao18 fa0245c
install horovod
pinggao18 11b5614
remove conda env
pinggao18 eaeac8e
install bigdl-orca-spark3
pinggao18 ff69478
split conda env
pinggao18 d46f182
source conda
pinggao18 4176900
split conda env
pinggao18 e26de8f
update
pinggao18 27b0ec5
update env
pinggao18 30cb3b4
remove env
pinggao18 09a66e4
update
pinggao18 63aee4d
update
pinggao18 73d8241
uninstall requirements
pinggao18 8ec9a2e
update
pinggao18 42a3560
update install whl
pinggao18 6d00823
update
pinggao18 bdc0b5b
add remove env
pinggao18 1d92e93
install bigdl-orca-spark3
pinggao18 9ab6fa4
update
pinggao18 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
17 changes: 17 additions & 0 deletions
17
.github/actions/orca/orca-python-horovod-tf2-py37-spark3-action/nightly-test/action.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
name: 'Run Orca Python Horovod Tf2 Py37 Spark3' | ||
description: 'Run Orca Python Horovod Tf2 Py37 Spark3' | ||
runs: | ||
using: "composite" | ||
steps: | ||
- name: Run Test | ||
shell: bash | ||
run: | | ||
source activate py37 | ||
export SPARK_LOCAL_HOSTNAME=localhost | ||
chmod a+x python/orca/dev/test/run-pytests-horovod-tf.sh | ||
python/orca/dev/test/run-pytests-horovod-tf.sh | ||
ray stop | ||
source deactivate | ||
env: | ||
BIGDL_ROOT: ${{ github.workspace }} | ||
ANALYTICS_ZOO_ROOT: ${{ github.workspace }} |
50 changes: 50 additions & 0 deletions
50
.github/actions/orca/setup-env/setup-horovod-tf2-py37/action.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,50 @@ | ||
name: 'Run Orca Python Horovod Tf2 Py37 Spark3' | ||
description: 'Run Orca Python Horovod Tf2 Py37 Spark3' | ||
runs: | ||
using: "composite" | ||
steps: | ||
- name: Set up Python | ||
uses: actions/setup-python@v2 | ||
with: | ||
python-version: '3.7.10' | ||
|
||
- name: Install dependencies | ||
shell: bash | ||
run: | | ||
apt-get update | ||
apt-get install wget | ||
python -m pip install --upgrade pip | ||
python -m pip install --upgrade setuptools==58.0.4 | ||
|
||
- name: Setup env | ||
shell: bash | ||
run: | | ||
if conda info --env | grep "py37"; then | ||
source activate py37 | ||
else | ||
conda create -n py37 -y python==3.7.10 setuptools==58.0.4 | ||
conda info --env | ||
source activate py37 | ||
fi | ||
pip uninstall -y bigdl-friesian bigdl-friesian-spark3 bigdl-dllib bigdl-dllib-spark3 bigdl-orca pyspark bigdl-orca-spark3 bigdl-chronos bigdl-chronos-spark3 bigdl-friesian bigdl-friesian-spark3 | ||
pip uninstall -y intel-tensorflow | ||
pip install cmake mxnet==1.6.0 | ||
pip uninstall -r .github/actions/python-requirements/requirements-horovod-tf2.txt -y | ||
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple -r .github/actions/python-requirements/requirements-horovod-tf2.txt | ||
pip install pytest | ||
HOROVOD_WITH_PYTORCH=1; HOROVOD_WITH_GLOO=1; pip install --no-cache-dir horovod==0.19.2 | ||
pip uninstall -y opencv-python-headless | ||
pip install opencv-python-headless | ||
pip install --pre --upgrade bigdl-orca-spark3 | ||
#sed -i "s/'bigdl-core=='+VERSION/'bigdl-core==2.1.0b20220811'/g" python/dllib/src/setup.py | ||
#bash python/dev/release_default_linux_spark312.sh default false false | ||
|
||
# install dllib | ||
#pip install python/dllib/src/dist/bigdl_dllib*-py3-none-manylinux1_x86_64.whl | ||
|
||
# install orca | ||
#pip install python/orca/src/dist/bigdl_orca*-py3-none-manylinux1_x86_64.whl | ||
|
||
env: | ||
BIGDL_ROOT: ${{ github.workspace }} | ||
ANALYTICS_ZOO_ROOT: ${{ github.workspace }} |
33 changes: 33 additions & 0 deletions
33
.github/actions/python-requirements/requirements-horovod-tf2.txt
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,33 @@ | ||
torch==1.7.1 | ||
torchvision | ||
torchmetrics | ||
albumentations | ||
scikit-learn==0.22.2.post1 | ||
opencv-python | ||
matplotlib | ||
tqdm | ||
protobuf==3.15.3 | ||
pandas | ||
ray[default]==1.9.2 | ||
aiohttp==3.8.1 | ||
async-timeout==4.0.1 | ||
dm_tree | ||
gym[atari]==0.17.1 | ||
atari-py==0.2.5 | ||
tabulate==0.8.7 | ||
tensorboard==2.3.0 | ||
tensorboard-data-server==0.6.1 | ||
tensorboard-plugin-wit==1.7.0 | ||
tensorboardX==2.1 | ||
tensorflow==2.3.0 | ||
tensorflow-estimator==2.3.0 | ||
xgboost_ray | ||
argparse | ||
Pillow | ||
pyarrow==4.0.1 | ||
h5py==2.10.0 | ||
fsspec==2021.9.0 | ||
scikit-optimize==0.8.1 | ||
pandas==1.1.5 | ||
Keras-Applications==1.0.8 | ||
Keras-Preprocessing==1.1.2 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
name: 'Remove Env' | ||
description: 'Remove Env' | ||
runs: | ||
using: "composite" | ||
steps: | ||
- name: Remove Env | ||
shell: bash | ||
run: | | ||
source activate py37 | ||
pip uninstall -y pyspark bigdl-friesian bigdl bigdl-dllib-spark3 bigdl-orca-spark3 bigdl-tf bigdl-math bigdl-orca bigdl-dllib bigdl-tf bigdl-chronos bigdl-nano | ||
source deactivate | ||
conda remove -n py37 -y --all | ||
env: | ||
BIGDL_ROOT: ${{ github.workspace }} | ||
ANALYTICS_ZOO_ROOT: ${{ github.workspace }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will a remove-env action be added?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
removing the env will cause each test to take too long, will no longer add remove-env.