Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix CI workflow runs using manually created virtual environments #10601

Merged
merged 3 commits into from
Dec 29, 2021
Merged

Fix CI workflow runs using manually created virtual environments #10601

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2a8895c
update workflows to manually create a venv
m-vdb Dec 29, 2021
7717373
listen to changes on workflow files in change_filters.yml
m-vdb Dec 29, 2021
eba6546
add changelog fragment
m-vdb Dec 29, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/change_filters.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ backend:
- 'data/**/*'
- 'examples/**/*'
- 'Makefile'
- '.github/workflows/continous-integration.yml'
- '.github/workflows/security-scans.yml'

docker:
- 'pyproject.toml'
Expand All @@ -22,3 +24,5 @@ docs:
- 'data/**/*'
- 'examples/**/*'
- 'Makefile'
- '.github/workflows/documentation.yml'
- '.github/workflows/ci-docs-tests.yml'
15 changes: 14 additions & 1 deletion .github/workflows/ci-docs-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,12 +60,25 @@ jobs:
poetry-version: ${{ env.POETRY_VERSION }}

- name: Load Poetry Cached Libraries ⬇
id: cache-poetry
uses: actions/cache@v2
with:
path: ~/.cache/pypoetry/virtualenvs
path: .venv
key: ${{ runner.os }}-poetry-${{ env.POETRY_VERSION }}-${{ env.DEFAULT_PYTHON_VERSION }}-${{ hashFiles('**/poetry.lock') }}-${{ secrets.POETRY_CACHE_VERSION }}
restore-keys: ${{ runner.os }}-poetry-${{ env.DEFAULT_PYTHON_VERSION }}

- name: Clear Poetry cache
if: steps.cache-poetry.outputs.cache-hit == 'true' && contains(github.event.pull_request.labels.*.name, 'tools:clear-poetry-cache-docs-tests')
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we document PR labels like tools:clear-poetry-cache-docs-tests somewhere? I haven't created the label yet on github (same for other new labels introduced in this PR)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This notion page is the closest I can find

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 I guess I'll just create the labels in GitHub for now, and describe them here

run: rm -r .venv

- name: Create virtual environment
if: (steps.cache-poetry.outputs.cache-hit != 'true' || contains(github.event.pull_request.labels.*.name, 'tools:clear-poetry-cache-docs-tests'))
run: python -m venv create .venv

- name: Set up virtual environment
if: needs.changes.outputs.docs == 'true'
run: poetry config virtualenvs.in-project true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it correct to set this in-project option to True? As far as I can see, poetry will create a new virtualenv itself with this option and it will be overwritten by python -v venv create .venv later on I'm guessing? If not set -which is the default- it will use existing venv if available (which will exist after manual creation and if it had not existed I think poetry would be creating a new venv by default and this is handled by virtualenvs.create option).

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is bypassing the creation of the venv in poetry somehow, because this is what failed before

If set to true, the virtualenv wil be created and expected in a folder named .venv within the root directory of the project.

In the case the .venv already exists (which we now enforce), I'm guessing poetry doesn't do anything (expected in a folder named .venv)


- name: Load Yarn Cached Packages ⬇
uses: actions/cache@v2
with:
Expand Down
15 changes: 14 additions & 1 deletion .github/workflows/continous-integration.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,13 +125,26 @@ jobs:
poetry-version: ${{ env.POETRY_VERSION }}

- name: Load Poetry Cached Libraries ⬇
id: cache-poetry
if: needs.changes.outputs.backend == 'true'
uses: actions/cache@v2
with:
path: ~/.cache/pypoetry/virtualenvs
path: .venv
key: ${{ runner.os }}-poetry-${{ env.POETRY_VERSION }}-${{ env.DEFAULT_PYTHON_VERSION }}-${{ hashFiles('**/poetry.lock') }}-${{ secrets.POETRY_CACHE_VERSION }}
restore-keys: ${{ runner.os }}-poetry-${{ env.DEFAULT_PYTHON_VERSION }}

- name: Clear Poetry cache
if: steps.cache-poetry.outputs.cache-hit == 'true' && needs.changes.outputs.backend == 'true' && contains(github.event.pull_request.labels.*.name, 'tools:clear-poetry-cache-unit-tests')
run: rm -r .venv

- name: Create virtual environment
if: (steps.cache-poetry.outputs.cache-hit != 'true' || contains(github.event.pull_request.labels.*.name, 'tools:clear-poetry-cache-unit-tests')) && needs.changes.outputs.backend == 'true'
run: python -m venv create .venv

- name: Set up virtual environment
if: needs.changes.outputs.backend == 'true'
run: poetry config virtualenvs.in-project true

- name: Install Dependencies 📦
if: needs.changes.outputs.backend == 'true'
run: |
Expand Down
29 changes: 27 additions & 2 deletions .github/workflows/documentation.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,12 +160,24 @@ jobs:
poetry-version: ${{ env.POETRY_VERSION }}

- name: Load Poetry Cached Libraries ⬇
id: cache-poetry
uses: actions/cache@v1
with:
path: ~/.cache/pypoetry/virtualenvs
path: .venv
key: ${{ runner.os }}-poetry-${{ env.POETRY_VERSION }}-3.7-non-full-${{ hashFiles('**/poetry.lock') }}-${{ secrets.POETRY_CACHE_VERSION }}
restore-keys: ${{ runner.os }}-poetry-3.7-non-full

- name: Clear Poetry cache
if: steps.cache-poetry.outputs.cache-hit == 'true'
run: rm -r .venv

- name: Create virtual environment
if: steps.cache-poetry.outputs.cache-hit != 'true'
run: python -m venv create .venv

- name: Set up virtual environment
run: poetry config virtualenvs.in-project true

- name: Load Yarn Cached Packages ⬇
uses: actions/cache@v1
with:
Expand Down Expand Up @@ -243,13 +255,26 @@ jobs:
poetry-version: ${{ env.POETRY_VERSION }}

- name: Load Poetry Cached Libraries ⬇
id: cache-poetry
if: needs.changes.outputs.docs == 'true'
uses: actions/cache@v1
with:
path: ~/.cache/pypoetry/virtualenvs
path: .venv
key: ${{ runner.os }}-poetry-${{ env.POETRY_VERSION }}-3.7-non-full-${{ hashFiles('**/poetry.lock') }}-${{ secrets.POETRY_CACHE_VERSION }}
restore-keys: ${{ runner.os }}-poetry-3.7-non-full

- name: Clear Poetry cache
if: steps.cache-poetry.outputs.cache-hit == 'true' && needs.changes.outputs.docs == 'true' && contains(github.event.pull_request.labels.*.name, 'tools:clear-poetry-cache-preview-docs')
run: rm -r .venv

- name: Create virtual environment
if: (steps.cache-poetry.outputs.cache-hit != 'true' || contains(github.event.pull_request.labels.*.name, 'tools:clear-poetry-cache-preview-docs')) && needs.changes.outputs.docs == 'true'
run: python -m venv create .venv

- name: Set up virtual environment
if: needs.changes.outputs.docs == 'true'
run: poetry config virtualenvs.in-project true

- name: Load Yarn Cached Packages ⬇
if: needs.changes.outputs.docs == 'true'
uses: actions/cache@v1
Expand Down
17 changes: 13 additions & 4 deletions .github/workflows/security-scans.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,18 +66,27 @@ jobs:
with:
poetry-version: ${{ env.POETRY_VERSION }}

- name: Set up virtual environment
if: needs.changes.outputs.backend == 'true'
run: poetry config virtualenvs.in-project true

- name: Load Poetry Cached Libraries ⬇
id: cache-poetry
if: needs.changes.outputs.backend == 'true'
uses: actions/cache@v1
with:
path: .venv
key: ${{ runner.os }}-poetry-${{ env.POETRY_VERSION }}-3.7-${{ hashFiles('**/poetry.lock') }}-${{ secrets.POETRY_CACHE_VERSION }}
restore-keys: ${{ runner.os }}-poetry-3.7

- name: Clear Poetry cache
if: steps.cache-poetry.outputs.cache-hit == 'true' && needs.changes.outputs.backend == 'true' && contains(github.event.pull_request.labels.*.name, 'tools:clear-poetry-cache-security-scans')
run: rm -r .venv

- name: Create virtual environment
if: (steps.cache-poetry.outputs.cache-hit != 'true' || contains(github.event.pull_request.labels.*.name, 'tools:clear-poetry-cache-security-scans')) && needs.changes.outputs.backend == 'true'
run: python -m venv create .venv

- name: Set up virtual environment
if: needs.changes.outputs.backend == 'true'
run: poetry config virtualenvs.in-project true

- name: Install Dependencies (Linux) 📦
if: needs.changes.outputs.backend == 'true'
run: make install
Expand Down
1 change: 1 addition & 0 deletions changelog/10601.misc.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Manually create Python virtual environments in CI workflows.