Retrieve upstream tarballs from GitHub release assets #36435
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 5 commits
October 9, 2023 12:32
…irror to mirror_list
…ces in SAGE_ROOT/.upstream.d
jhpalmieri
reviewed
Oct 9, 2023
| continue | ||
| line = line.replace('${SAGE_SERVER}',) | ||
| line = line.strip() | ||
| if line.startswith('#'): |
There was a problem hiding this comment.
Speed probably doesn't matter too much, but wouldn't it be a little faster to move if not line and if line.startswith('#') earlier in this block?
There was a problem hiding this comment.
Thanks, done in 7e65260
jhpalmieri
reviewed
Oct 9, 2023
src/bin/sage-update-version
Outdated
| ) > "$SAGE_ROOT/.upstream.d/20-github.com-sagemath-sage-releases" | ||
| rm -f "$SAGE_ROOT/.upstream.d/20-github.com-sagemath-sage-releases.tmp" | ||
|
|
||
| exit |
There was a problem hiding this comment.
Why exit? Doesn't that end the script before executing the rest of it? Also, do we need to git commit the new file?
There was a problem hiding this comment.
Whoops, that was debugging code. Fixed in 371be50
|
In principle it looks okay to me. I'm guessing that github will be a reliable place to store tarballs. Are there any downsides? |
|
I'm not aware of any downsides of this |
mkoeppe
force-pushed
the
gh_actions_release
branch
from
57bd6f3
to
423e48a
Compare
|
Documentation preview for this PR (built with commit 423e48a; changes) is ready! 🎉 |
|
Looks good, let's merge it. |
|
Thank you! |
|
I'm marking it as a blocker so that it takes effect in current CI runs, which currently can get stuck because of the missing tarballs (see https://github.com/sagemath/sage/actions/runs/6454925852/job/17521487027#step:8:14136) |
|
why is 10.1 hardcoded in .upstream.d/20-github.com-sagemath-sage-releases ? |
|
As the first two lines of the file say: |
vbraun
pushed a commit
to vbraun/sage
that referenced
this pull request
Oct 15, 2023
<!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> We propose to add GitHub release assets as a source from which upstream tarballs can be retrieved. This would take priority over trying the Sage mirrors, thus reducing the reliance of the project on self-maintained file servers. https://docs.github.com/en/repositories/releasing- projects-on-github/about-releases#storage-and-bandwidth-quotas For the purpose of testing this PR, one tarball missing on the mirrors (sagemath#36381 (comment)) has been uploaded to https://github.com/sagemath/sage/releases/tag/10.1 Creating GitHub releases and depositing the upstream tarballs can later be automated in a GH Actions workflow, e.g., using https://github.com/softprops/action-gh-release. As we don't make GitHub releases for beta versions, the previous stable release can be used. <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#36435 Reported by: Matthias Köppe Reviewer(s): Dima Pasechnik, John H. Palmieri, Matthias Köppe
vbraun
pushed a commit
to vbraun/sage
that referenced
this pull request
Oct 15, 2023
…se tag
<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes sagemath#1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->
We extend our workflow dist.yml, which gets run on every release tag, to
do the following:
- For stable releases (no beta, no rc), when run on sagemath/sage, it
automatically creates a Release. (This saves a few clicks by maintainers
on each release.)
- It uploads the sdist (the 1.2GB file which contains upstream tarballs
for all standard packages) as a Release Asset.
- It uploads the upstream tarballs as Release Assets. They will be used
by the Sage distribution after sagemath#36435
Example of the run:
https://github.com/mkoeppe/sage/actions/runs/6464035205/job/17549159925
(where I changed the condition from sagemath/sage to mkoeppe/sage),
leading to the example release
https://github.com/mkoeppe/sage/releases/tag/10.2.beta007
<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
<!-- If your change requires a documentation PR, please link it
appropriately. -->
### 📝 Checklist
<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->
- [x] The title is concise, informative, and self-explanatory.
- [x] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.
### ⌛ Dependencies
<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
URL: sagemath#36438
Reported by: Matthias Köppe
Reviewer(s): Dima Pasechnik
vbraun
pushed a commit
to vbraun/sage
that referenced
this pull request
Oct 17, 2023
<!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> We propose to add GitHub release assets as a source from which upstream tarballs can be retrieved. This would take priority over trying the Sage mirrors, thus reducing the reliance of the project on self-maintained file servers. https://docs.github.com/en/repositories/releasing- projects-on-github/about-releases#storage-and-bandwidth-quotas For the purpose of testing this PR, one tarball missing on the mirrors (sagemath#36381 (comment)) has been uploaded to https://github.com/sagemath/sage/releases/tag/10.1 Creating GitHub releases and depositing the upstream tarballs can later be automated in a GH Actions workflow, e.g., using https://github.com/softprops/action-gh-release. As we don't make GitHub releases for beta versions, the previous stable release can be used. <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#36435 Reported by: Matthias Köppe Reviewer(s): Dima Pasechnik, John H. Palmieri, Matthias Köppe
vbraun
pushed a commit
to vbraun/sage
that referenced
this pull request
Oct 17, 2023
…se tag
<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes sagemath#1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->
We extend our workflow dist.yml, which gets run on every release tag, to
do the following:
- For stable releases (no beta, no rc), when run on sagemath/sage, it
automatically creates a Release. (This saves a few clicks by maintainers
on each release.)
- It uploads the sdist (the 1.2GB file which contains upstream tarballs
for all standard packages) as a Release Asset.
- It uploads the upstream tarballs as Release Assets. They will be used
by the Sage distribution after sagemath#36435
Example of the run:
https://github.com/mkoeppe/sage/actions/runs/6464035205/job/17549159925
(where I changed the condition from sagemath/sage to mkoeppe/sage),
leading to the example release
https://github.com/mkoeppe/sage/releases/tag/10.2.beta007
<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
<!-- If your change requires a documentation PR, please link it
appropriately. -->
### 📝 Checklist
<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->
- [x] The title is concise, informative, and self-explanatory.
- [x] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.
### ⌛ Dependencies
<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
URL: sagemath#36438
Reported by: Matthias Köppe
Reviewer(s): Dima Pasechnik
vbraun
pushed a commit
to vbraun/sage
that referenced
this pull request
Oct 19, 2023
…se tag
<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes sagemath#1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->
We extend our workflow dist.yml, which gets run on every release tag, to
do the following:
- For stable releases (no beta, no rc), when run on sagemath/sage, it
automatically creates a Release. (This saves a few clicks by maintainers
on each release.)
- It uploads the sdist (the 1.2GB file which contains upstream tarballs
for all standard packages) as a Release Asset.
- It uploads the upstream tarballs as Release Assets. They will be used
by the Sage distribution after sagemath#36435
Example of the run:
https://github.com/mkoeppe/sage/actions/runs/6464035205/job/17549159925
(where I changed the condition from sagemath/sage to mkoeppe/sage),
leading to the example release
https://github.com/mkoeppe/sage/releases/tag/10.2.beta007
<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
<!-- If your change requires a documentation PR, please link it
appropriately. -->
### 📝 Checklist
<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->
- [x] The title is concise, informative, and self-explanatory.
- [x] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.
### ⌛ Dependencies
<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
URL: sagemath#36438
Reported by: Matthias Köppe
Reviewer(s): Dima Pasechnik
vbraun
pushed a commit
to vbraun/sage
that referenced
this pull request
Nov 6, 2023
…ve `gentoo-python3.12` for now, use `conda-forge` with `-python3.11` <!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> As part of the changes in sagemath#36435, we switched from http to https for downloading mirror_list. This broke centos-7; here we fix it. gentoo-python3.12 was activated a little bit too early. We remove it to clean up the portability CI workflow. conda-forge defaults to 3.12 already, and we're not ready for it. <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [ ] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#36534 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee
vbraun
pushed a commit
to vbraun/sage
that referenced
this pull request
Nov 7, 2023
…ve `gentoo-python3.12` for now, use `conda-forge` with `-python3.11`
<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes sagemath#1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->
As part of the changes in sagemath#36435, we switched from http to https for
downloading mirror_list.
This broke centos-7; here we fix it.
gentoo-python3.12 was activated a little bit too early. We remove it to
clean up the portability CI workflow.
conda-forge defaults to 3.12 already, and we're not ready for it.
<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
<!-- If your change requires a documentation PR, please link it
appropriately. -->
### 📝 Checklist
<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->
- [x] The title is concise, informative, and self-explanatory.
- [ ] The description explains in detail what this PR is about.
- [ ] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.
### ⌛ Dependencies
<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
URL: sagemath#36534
Reported by: Matthias Köppe
Reviewer(s): Kwankyu Lee
vbraun
pushed a commit
to vbraun/sage
that referenced
this pull request
Nov 9, 2023
…agemath#36435 <!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> This repairs the installation method described in https://github.com/sagemath/sage#alternative-installation-using-pypi broken by changes in sagemath#36400, sagemath#36435. We also expand the tests for this distribution in its tox.ini. Run `(cd pkgs/sage-conf_pypi && tox -v -v -e py311)` <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [ ] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#36533 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee
vbraun
pushed a commit
to vbraun/sage
that referenced
this pull request
Nov 9, 2023
…ve `gentoo-python3.12` for now, use `conda-forge` with `-python3.11`
<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes sagemath#1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->
As part of the changes in sagemath#36435, we switched from http to https for
downloading mirror_list.
This broke centos-7; here we fix it.
gentoo-python3.12 was activated a little bit too early. We remove it to
clean up the portability CI workflow.
conda-forge defaults to 3.12 already, and we're not ready for it.
<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
<!-- If your change requires a documentation PR, please link it
appropriately. -->
### 📝 Checklist
<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->
- [x] The title is concise, informative, and self-explanatory.
- [ ] The description explains in detail what this PR is about.
- [ ] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.
### ⌛ Dependencies
<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
URL: sagemath#36534
Reported by: Matthias Köppe
Reviewer(s): Kwankyu Lee
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We propose to add GitHub release assets as a source from which upstream tarballs can be retrieved. This would take priority over trying the Sage mirrors, thus reducing the reliance of the project on self-maintained file servers. https://docs.github.com/en/repositories/releasing-projects-on-github/about-releases#storage-and-bandwidth-quotas
For the purpose of testing this PR, one tarball missing on the mirrors (#36381 (comment)) has been uploaded to https://github.com/sagemath/sage/releases/tag/10.1
Creating GitHub releases and depositing the upstream tarballs can later be automated in a GH Actions workflow, e.g., using https://github.com/softprops/action-gh-release. As we don't make GitHub releases for beta versions, the previous stable release can be used.
📝 Checklist
⌛ Dependencies