Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hardcoding pip version for workflows #887

Merged
merged 2 commits into from
Jun 27, 2024
Merged

Hardcoding pip version for workflows #887

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
14f8446
change the pip_version to 24.0 because of an error
benmalef Jun 27, 2024
b87c431
change the pip_version in python_test
benmalef Jun 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
110 changes: 54 additions & 56 deletions .github/workflows/mlcube-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,35 +4,33 @@ name: MLCube-Test

on:
push:
branches: [ master ]
branches: [master]
pull_request: {}


jobs:
test-deploy:

runs-on: ubuntu-latest

steps:
- name: Free space
run: |
df -h
sudo rm -rf /opt/ghc
sudo rm -rf "/usr/local/share/boost"
sudo rm -rf "$AGENT_TOOLSDIRECTORY"
sudo rm -rf "$ANDROID_SDK_ROOT"
df -h
- name: Checkout
uses: actions/checkout@v3
# Use changed-files-specific action to collect file changes.
# The following commented condition applied to a step will run that step only if non-docs files have changed.
# It should be applied to all functionality-related steps.
# if: steps.changed-files-specific.outputs.only_modified == 'false'
- name: Detect and screen file changes
id: changed-files-specific
uses: tj-actions/changed-files@v41
with:
- name: Free space
run: |
df -h
sudo rm -rf /opt/ghc
sudo rm -rf "/usr/local/share/boost"
sudo rm -rf "$AGENT_TOOLSDIRECTORY"
sudo rm -rf "$ANDROID_SDK_ROOT"
df -h
- name: Checkout
uses: actions/checkout@v3

# Use changed-files-specific action to collect file changes.
# The following commented condition applied to a step will run that step only if non-docs files have changed.
# It should be applied to all functionality-related steps.
# if: steps.changed-files-specific.outputs.only_modified == 'false'
- name: Detect and screen file changes
id: changed-files-specific
uses: tj-actions/changed-files@v41
with:
files: |
.github/*.md
.github/ISSUE_TEMPLATE/*.md
Expand All @@ -45,37 +43,37 @@ jobs:
LICENSE
Dockerfile-*

- name: Summarize docs and non-docs modifications
run: |
echo "List of docs files that have changed: ${{ steps.changed-files-specific.outputs.all_modified_files }}"
echo "Changed non-docs files: ${{ steps.changed-files-specific.outputs.other_modified_files }}"
# This second step is unnecessary but highly recommended because
# It will cache database and saves time re-downloading it if database isn't stale.
- name: Cache pip
uses: actions/cache@v3
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/setup.py') }}
restore-keys: |
${{ runner.os }}-pip-
- name: Set up Python 3.9
if: steps.changed-files-specific.outputs.only_modified == 'false' # Run on any non-docs change
uses: actions/setup-python@v4
with:
python-version: 3.9
- name: Install dependencies and package
if: steps.changed-files-specific.outputs.only_modified == 'false' # Run on any non-docs change
run: |
sudo apt-get update
sudo apt-get install libvips libvips-tools -y
python -m pip install --upgrade pip
python -m pip install wheel
python -m pip install openvino-dev==2023.0.1 mlcube_docker
pip install torch==2.2.1 torchvision==0.17.1 torchaudio==2.2.1 --index-url https://download.pytorch.org/whl/cpu
pip install -e .
- name: Run mlcube deploy tests
working-directory: ./testing
if: steps.changed-files-specific.outputs.only_modified == 'false' # Run on any non-docs change
run: |
sh test_deploy.sh
- name: Summarize docs and non-docs modifications
run: |
echo "List of docs files that have changed: ${{ steps.changed-files-specific.outputs.all_modified_files }}"
echo "Changed non-docs files: ${{ steps.changed-files-specific.outputs.other_modified_files }}"

# This second step is unnecessary but highly recommended because
# It will cache database and saves time re-downloading it if database isn't stale.
- name: Cache pip
uses: actions/cache@v3
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/setup.py') }}
restore-keys: |
${{ runner.os }}-pip-
- name: Set up Python 3.9
if: steps.changed-files-specific.outputs.only_modified == 'false' # Run on any non-docs change
uses: actions/setup-python@v4
with:
python-version: 3.9
- name: Install dependencies and package
if: steps.changed-files-specific.outputs.only_modified == 'false' # Run on any non-docs change
run: |
sudo apt-get update
sudo apt-get install libvips libvips-tools -y
python -m pip install --upgrade pip==24.0
python -m pip install wheel
python -m pip install openvino-dev==2023.0.1 mlcube_docker
pip install torch==2.2.1 torchvision==0.17.1 torchaudio==2.2.1 --index-url https://download.pytorch.org/whl/cpu
pip install -e .
- name: Run mlcube deploy tests
working-directory: ./testing
if: steps.changed-files-specific.outputs.only_modified == 'false' # Run on any non-docs change
run: |
sh test_deploy.sh
154 changes: 76 additions & 78 deletions .github/workflows/openfl-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,35 +5,33 @@ name: OpenFL-Test

on:
push:
branches: [ master ]
branches: [master]
pull_request: {}


jobs:
openfl-test:

runs-on: ubuntu-latest

steps:
- name: Free space
run: |
df -h
sudo rm -rf /opt/ghc
sudo rm -rf "/usr/local/share/boost"
sudo rm -rf "$AGENT_TOOLSDIRECTORY"
sudo rm -rf "$ANDROID_SDK_ROOT"
df -h
- name: Checkout
uses: actions/checkout@v3
# Use changed-files-specific action to collect file changes.
# The following commented condition applied to a step will run that step only if non-docs files have changed.
# It should be applied to all functionality-related steps.
# if: steps.changed-files-specific.outputs.only_modified == 'false'
- name: Detect and screen file changes
id: changed-files-specific
uses: tj-actions/changed-files@v41
with:
- name: Free space
run: |
df -h
sudo rm -rf /opt/ghc
sudo rm -rf "/usr/local/share/boost"
sudo rm -rf "$AGENT_TOOLSDIRECTORY"
sudo rm -rf "$ANDROID_SDK_ROOT"
df -h
- name: Checkout
uses: actions/checkout@v3

# Use changed-files-specific action to collect file changes.
# The following commented condition applied to a step will run that step only if non-docs files have changed.
# It should be applied to all functionality-related steps.
# if: steps.changed-files-specific.outputs.only_modified == 'false'
- name: Detect and screen file changes
id: changed-files-specific
uses: tj-actions/changed-files@v41
with:
files: |
.github/*.md
.github/ISSUE_TEMPLATE/*.md
Expand All @@ -46,59 +44,59 @@ jobs:
LICENSE
Dockerfile-*

- name: Summarize docs and non-docs modifications
run: |
echo "List of docs files that have changed: ${{ steps.changed-files-specific.outputs.all_modified_files }}"
echo "Changed non-docs files: ${{ steps.changed-files-specific.outputs.other_modified_files }}"
# This second step is unnecessary but highly recommended because
# It will cache database and saves time re-downloading it if database isn't stale.
- name: Cache pip
uses: actions/cache@v3
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/setup.py') }}
restore-keys: |
${{ runner.os }}-pip-
- name: Set up Python 3.9
if: steps.changed-files-specific.outputs.only_modified == 'false' # Run on any non-docs change
uses: actions/setup-python@v4
with:
python-version: 3.9
- name: Install dependencies and package
if: steps.changed-files-specific.outputs.only_modified == 'false' # Run on any non-docs change
run: |
sudo apt-get update
sudo apt-get install libvips libvips-tools -y
python -m pip install --upgrade pip
python -m pip install wheel
pip install torch==2.2.1 torchvision==0.17.1 torchaudio==2.2.1 --index-url https://download.pytorch.org/whl/cpu
pip install -e .
- name: Run generic unit tests to download data and construct CSVs
if: steps.changed-files-specific.outputs.only_modified == 'false' # Run on any non-docs change
run: |
pytest --cov=. --cov-report=xml -k "prepare_data_for_ci"
# openfl tests start here
- name: Run OpenFL tests
if: steps.changed-files-specific.outputs.only_modified == 'false' # Run on any non-docs change
run: |
echo "Removing onnx because of protobuf version conflict"
pip uninstall onnx -y
echo "Installing OpenFL"
git clone --depth=1 https://github.com/securefederatedai/openfl.git
cd openfl
git fetch --tags
# echo "Checkout the latest OpenFL tag"
# latestTag=$(git describe --tags "$(git rev-list --tags --max-count=1)")
# git checkout $latestTag
# sed -i -e 's/protobuf==3.19.6/protobuf/g' setup.py ## this should NOT be there
pip install -e .
cd ..
echo "Copying files to appropriate directories and updated headers"
head -n 1 testing/data/train_2d_rad_segmentation.csv > /home/runner/work/GaNDLF/GaNDLF/openfl/valid.csv
tail -n +9 testing/data/train_2d_rad_segmentation.csv >> /home/runner/work/GaNDLF/GaNDLF/openfl/valid.csv
head -n 8 testing/data/train_2d_rad_segmentation.csv > /home/runner/work/GaNDLF/GaNDLF/openfl/train.csv
sed -i 's/# n_channels: 3/num_channels: 3/g' testing/config_segmentation.yaml
config_to_use=$(pwd)/testing/config_segmentation.yaml
cd openfl
python -m tests.github.test_gandlf --template gandlf_seg_test --fed_workspace aggregator --col1 one --col2 two --rounds-to-train 1 --gandlf_config $config_to_use
- name: Summarize docs and non-docs modifications
run: |
echo "List of docs files that have changed: ${{ steps.changed-files-specific.outputs.all_modified_files }}"
echo "Changed non-docs files: ${{ steps.changed-files-specific.outputs.other_modified_files }}"

# This second step is unnecessary but highly recommended because
# It will cache database and saves time re-downloading it if database isn't stale.
- name: Cache pip
uses: actions/cache@v3
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/setup.py') }}
restore-keys: |
${{ runner.os }}-pip-
- name: Set up Python 3.9
if: steps.changed-files-specific.outputs.only_modified == 'false' # Run on any non-docs change
uses: actions/setup-python@v4
with:
python-version: 3.9
- name: Install dependencies and package
if: steps.changed-files-specific.outputs.only_modified == 'false' # Run on any non-docs change
run: |
sudo apt-get update
sudo apt-get install libvips libvips-tools -y
python -m pip install --upgrade pip==24.0
python -m pip install wheel
pip install torch==2.2.1 torchvision==0.17.1 torchaudio==2.2.1 --index-url https://download.pytorch.org/whl/cpu
pip install -e .
- name: Run generic unit tests to download data and construct CSVs
if: steps.changed-files-specific.outputs.only_modified == 'false' # Run on any non-docs change
run: |
pytest --cov=. --cov-report=xml -k "prepare_data_for_ci"
# openfl tests start here
- name: Run OpenFL tests
if: steps.changed-files-specific.outputs.only_modified == 'false' # Run on any non-docs change
run: |
echo "Removing onnx because of protobuf version conflict"
pip uninstall onnx -y
echo "Installing OpenFL"
git clone --depth=1 https://github.com/securefederatedai/openfl.git
cd openfl
git fetch --tags
# echo "Checkout the latest OpenFL tag"
# latestTag=$(git describe --tags "$(git rev-list --tags --max-count=1)")
# git checkout $latestTag
# sed -i -e 's/protobuf==3.19.6/protobuf/g' setup.py ## this should NOT be there
pip install -e .
cd ..
echo "Copying files to appropriate directories and updated headers"
head -n 1 testing/data/train_2d_rad_segmentation.csv > /home/runner/work/GaNDLF/GaNDLF/openfl/valid.csv
tail -n +9 testing/data/train_2d_rad_segmentation.csv >> /home/runner/work/GaNDLF/GaNDLF/openfl/valid.csv
head -n 8 testing/data/train_2d_rad_segmentation.csv > /home/runner/work/GaNDLF/GaNDLF/openfl/train.csv
sed -i 's/# n_channels: 3/num_channels: 3/g' testing/config_segmentation.yaml
config_to_use=$(pwd)/testing/config_segmentation.yaml
cd openfl
python -m tests.github.test_gandlf --template gandlf_seg_test --fed_workspace aggregator --col1 one --col2 two --rounds-to-train 1 --gandlf_config $config_to_use
Loading
Loading