Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI Linux: Fix "optional", "experimental" jobs #36668

Merged
merged 1 commit into from
Nov 10, 2023
Merged

CI Linux: Fix "optional", "experimental" jobs #36668

merged 1 commit into from
Nov 10, 2023

Conversation

mkoeppe
Copy link
Contributor

@mkoeppe mkoeppe commented Nov 6, 2023 •
edited
Loading

Fixes #36667

📝 Checklist

  • The title is concise, informative, and self-explanatory.
  • The description explains in detail what this PR is about.
  • I have linked a relevant issue or discussion.
  • I have created tests covering the changes.
  • I have updated the documentation accordingly.

⌛ Dependencies

@mkoeppe
.github/workflows/docker.yml: Set DOCKER_TAG from pre-merge commit sh…
9c8f3b6 
…a even when docker_push_repository is not set
@mkoeppe mkoeppe added this to the sage-10.2 milestone Nov 6, 2023
@mkoeppe mkoeppe self-assigned this Nov 6, 2023
@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 6, 2023

Documentation preview for this PR (built with commit 9c8f3b6; changes) is ready! 🎉

kwankyu
kwankyu reviewed Nov 7, 2023
View reviewed changes
.github/workflows/docker.yml
@@ -191,6 +191,9 @@ jobs:
echo "DOCKER_PUSH_REPOSITORY=$(echo ${{ inputs.docker_push_repository }} | tr "[:upper:]" "[:lower:]")" >> $GITHUB_ENV
echo "DOCKER_CONFIG_FILE=$HOME/.docker/config.json" >> $GITHUB_ENV
fi

- name: Determine Docker tags to use
run: |
# This line needs to be run before the step "Merge CI fixes from sagemath/sage".
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fix seems to indicate that the previous step (logging to ghcr.io) is not really necessary (used) for subsequent steps. So perhaps the step is here to provide the environment variables to the calling workflow.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The environment variables are set for the following steps in this job, specifically for invoking tox.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant

(A): logging into docker and setting the environment variables DOCKER_PUSH_REPOSITORY and DOCKER_CONFIG_FILE

in the previous step to the "Determine Docker tags to use" step.

If the idea is to run "Determine Docker tags to use" step even when inputs.docker_push_repository is empty, then it means that (A) is not necessary for the steps following and including the "Determine Docker tags to use" step.

So I thought (A) could be moved out to the calling workflow and don't have to be in the docker workflow. I am not suggesting we should do it. I just want to know if I understand the situation correctly. Am I right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the idea is to run "Determine Docker tags to use" step even when inputs.docker_push_repository is empty, then it means that (A) is not necessary for the steps following and including the "Determine Docker tags to use" step.

That's right, the two parts now separated do not interact with each other. Each part sets some environment variables for the following steps.

So I thought (A) could be moved out to the calling workflow and don't have to be in the docker workflow. I am not suggesting we should do it. I just want to know if I understand the situation correctly. Am I right?

No; environment variables set by appending to the file $GITHUB_ENV only take effect for the next steps of a job. But a reusable workflow is run as a job, not a step of a job. See https://docs.github.com/en/actions/using-workflows/reusing-workflows#limitations

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. Thanks.

kwankyu
kwankyu approved these changes Nov 7, 2023
View reviewed changes
Copy link
Collaborator

@kwankyu kwankyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Nov 7, 2023

Thanks!

@vbraun vbraun merged commit 81bf4e5 into sagemath:develop Nov 10, 2023
18 of 26 checks passed
@soehms
Copy link
Member

soehms commented Nov 10, 2023

Thanks for fixing this! In principle it seems to work now (for example on Debian Buster). But a lot of jobs have been cancelled due to timeout, especially the ones running on Ubuntu (example Ubuntu Xenial)

In the log-files of successful jobs I noticed errors on testing the database_knotinfo installation (and other test suites of optional packages). For example in Ubuntu boinic:

  Running the test suite for database_knotinfo-2023.6.1...
  Testing: databases/ features/ knots/
  /sage/local/var/lib/sage/venv-python3.11.1/bin/sage-venv-config:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    __import__('pkg_resources').require('sagemath-standard==10.2rc0')
  Traceback (most recent call last):
  ...
  pkg_resources.DistributionNotFound: The 'fpylll>=0.5.9' distribution was not found and is required by sagemath-standard

It looks like missing dependencies, no?

@mkoeppe mkoeppe deleted the ci_linux_optional branch November 10, 2023 19:00
@mkoeppe
Copy link
Contributor Author

mkoeppe commented Nov 10, 2023

a lot of jobs have been cancelled due to timeout, especially the ones running on Ubuntu (example Ubuntu Xenial)

There I see (after setting "Show timestamps"):

Wed, 08 Nov 2023 04:59:22 GMT
make[1]: Entering directory '/sage/build/make'
Wed, 08 Nov 2023 04:59:22 GMT
make --no-print-directory libnauty-SAGE_LOCAL-no-deps
Wed, 08 Nov 2023 04:59:22 GMT
make --no-print-directory lrslib-SAGE_LOCAL-no-deps
Wed, 08 Nov 2023 07:53:55 GMT
Error: The operation was canceled.

This is a bug -- these two should not take 3 hours to complete.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Nov 12, 2023

Ubuntu Xenial

In the log-files of successful jobs I noticed errors on testing the database_knotinfo installation (and other test suites of optional packages). For example in Ubuntu boinic:

  Running the test suite for database_knotinfo-2023.6.1...
  Testing: databases/ features/ knots/
  /sage/local/var/lib/sage/venv-python3.11.1/bin/sage-venv-config:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    __import__('pkg_resources').require('sagemath-standard==10.2rc0')
  Traceback (most recent call last):
  ...
  pkg_resources.DistributionNotFound: The 'fpylll>=0.5.9' distribution was not found and is required by sagemath-standard

It looks like missing dependencies, no?

Thanks for catching this! Fixed in #36708

vbraun pushed a commit to vbraun/sage that referenced this pull request Nov 14, 2023
sagemathgh-36708: CI Linux: Fixes for tests of optional packages
7f6a2a7 
    
<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes sagemath#1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->

Fixes test failure of `database_knotinfo` seen in GH Actions, as
reported by @soehms in
sagemath#36668 (comment)

Fixes same also for `database_cubic_hecke` and `mathics`.

To test: `FROM_DOCKER_TARGET=with-targets-pre TARGETS_PRE=config.status
tox -e docker-ubuntu-bionic-gcc_8-maximal-incremental --
database_knotinfo database_cubic_hecke mathics`

We also remove packages starting with `_` from the list of tested
packages. The dependencies of `_develop` pulled in the dummy package
`github_cli`.

We also use output groups to fold away some repetitive output.

<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
<!-- If your change requires a documentation PR, please link it
appropriately. -->

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->

- [x] The title is concise, informative, and self-explanatory.
- [x] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->

<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
    
URL: sagemath#36708
Reported by: Matthias Köppe
Reviewer(s): Dima Pasechnik, Sebastian Oehms
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kwankyu kwankyu kwankyu approved these changes

Assignees

@mkoeppe mkoeppe

Labels
c: scripts p: blocker / 1
Projects
None yet
Milestone
sage-10.2
Development

Successfully merging this pull request may close these issues.

CI Linux: "optional", "experimental" jobs are broken
4 participants
@mkoeppe @soehms @kwankyu @vbraun