Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DOC: SciPy 1.15.0 relnotes #21982

Merged
merged 11 commits into from
Dec 8, 2024
Merged

Conversation

tylerjereddy
Copy link
Contributor

@tylerjereddy tylerjereddy commented Dec 1, 2024

Early draft of SciPy 1.15.0 release notes. Skipping CI completely for now, though local render of the draft notes looks nice so far. Folks may continue to update the wiki or make suggestions here--either is fine--I simply start this because the initial transcription takes longer than you might think so getting it rolling helps avoid last minute pain.

TODO:

  • transcribe wiki version to RST, wrap at 80 char line lengths, fix links/errors, check render (last done: Nov 30th at 8:09 pm US mountain time)
  • check the wiki for updates again before branching and update this PR accordingly
  • manually scan through ENH PRs and add entries not already present in current notes (last done on Dec. 3/2024)
  • author list update (repeat just before branching)
  • issue/PR list update (repeat just before branching)
  • .mailmap update from DOC, REL: 1.15.0 author mappings #21940
  • gradually reactivate the CI here, when ready
  • finalize release notes highlights
  • Remove unused/empty (sub)sections in the release notes

@tylerjereddy tylerjereddy added the Documentation Issues related to the SciPy documentation. Also check https://github.com/scipy/scipy.org label Dec 1, 2024
@tylerjereddy tylerjereddy added this to the 1.15.0 milestone Dec 1, 2024
implementation of univariate, continuous distributions:

- Speed, accuracy, and memory advantages of the new infastructure are
detailed in
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mostly note to self--this is the only "incomplete" sentence I noticed last night, probably because I transcribed the wiki release notes a bit early

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I left a blank to add a link to a tutorial when it merges.

Copy link
Member

@rgommers rgommers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Tyler! Looks good overall.

For the first paragraph, I suggest changing Python 3.10+ to Python 3.10-3.13, since that's more accurate and useful info.

Here is suggested content to describe free-threading support: https://github.com/rgommers/scipy/tree/relnotes-115-freethreading. Could you please pull that in @tylerjereddy?

I added a few other high-level comments; no detailed line by line review yet since this is draft.

doc/source/release/1.15.0-notes.rst Show resolved Hide resolved
doc/source/release/1.15.0-notes.rst Outdated Show resolved Hide resolved
doc/source/release/1.15.0-notes.rst Outdated Show resolved Hide resolved
doc/source/release/1.15.0-notes.rst Outdated Show resolved Hide resolved
@tylerjereddy
Copy link
Contributor Author

tylerjereddy commented Dec 4, 2024

Ok, I did the manual read through of all 123 ENH-labelled PRs that were merged under the 1.15.0 milestone and filled in some things that were missing. I cast a fairly wide net, adding a few things I thought were borderline, since I figured it would be easier to remove them than to have others also scanning through the pages of PRs.

I also added the array API support section--somehow I forgot that in the first pass at transcribing from the wiki page a few days ago.

Caution: I caught some deprecations that were not mentioned--particularly messy was stuff surrounding special like assoc_legendre_p_all -- I was about to add a release note saying that lpn et al. had a new keyword/improvement, and then when I looked at them, they were deprecated in the same release as the improvement!

@jorenham
Copy link
Contributor

jorenham commented Dec 4, 2024

I'm also compiling a similar 1.15.0 checklist at jorenham/scipy-stubs#250. So in case anyone is willing to see if I missed something, that would be certainly appreciated!

@tylerjereddy tylerjereddy marked this pull request as ready for review December 4, 2024 21:41
@tylerjereddy
Copy link
Contributor Author

I've attempted to catch up with all the reviewer comments now a well--I'll leave them "unresolved" for a bit in case folks prefer to check that I've genuinely resolved their concern.

I'm also lifting the draft status and allowing the docs-only CI to flush through now since it looks pretty nice locally. Of course, this will remain informally WIP until branching proper.

@tylerjereddy
Copy link
Contributor Author

I'd also be "ok" with mentioning scipy-stubs in the "Other" section perhaps? Looks like Joren is pretty organized there and that's at least of peripheral interest to consumers.

@jorenham
Copy link
Contributor

jorenham commented Dec 4, 2024

I'd also be "ok" with mentioning scipy-stubs in the "Other" section perhaps?

That'd be awesome 😄.

I'll try my best to release scipy-stubs 1.15.0.0 before the final scipy 1.15.0. Assuming that there won't be any API changes after the RC has been released, then those 3-4 weeks should be more than enough time for me to achieve that 👌🏻.

@jorenham
Copy link
Contributor

jorenham commented Dec 4, 2024

I'd also be "ok" with mentioning scipy-stubs in the "Other" section perhaps?

That'd be awesome 😄.

I'll try my best to release scipy-stubs 1.15.0.0 before the final scipy 1.15.0. Assuming that there won't be any API changes after the RC has been released, then those 3-4 weeks should be more than enough time for me to achieve that 👌🏻.

So in case it wasn't clear yet; I don't plan on releasing a scipy-stubs 1.15.0rc0.0, and jump right to 1.15.0.0. And even if it turns out that I messed up something, then scipy-stubs isn't bound to any release schedule, so I'll be able to fix it right away.

doc/source/release/1.15.0-notes.rst Outdated Show resolved Hide resolved
doc/source/release/1.15.0-notes.rst Show resolved Hide resolved
doc/source/release/1.15.0-notes.rst Outdated Show resolved Hide resolved
doc/source/release/1.15.0-notes.rst Outdated Show resolved Hide resolved
@@ -17,96 +17,1431 @@ run your code with ``python -Wd`` and check for ``DeprecationWarning`` s).
Our development attention will now shift to bug-fix releases on the
1.15.x branch, and on adding new features on the main branch.

This release requires Python 3.10+ and NumPy 1.23.5 or greater.
This release requires Python 3.10-3.13 and NumPy 1.23.5 or greater.
Copy link
Contributor

@jorenham jorenham Dec 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given SPEC 0, I expected this to be numpy>=1.24(.4) 🤔

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh and I see that SPEC 0 also recommends dropping support for Python 3.10 in Q4 2024.

Copy link
Contributor

@mdhaber mdhaber Dec 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re: Python version, it sounds fine to stay in sync with NumPy as long as they don't get too far behind.

Re: #21982 (comment), note that we will actually be past the SPEC 0 recommended support window for NumPy 1.24 when SciPy 1.15 is released, but we tend to lag behind by a version or so, perhaps to avoid ending up on the borderline. However, I would like to get to 2.1+ earlier rather than later.

If we do NumPy 1.24 for 1.15 (Jan 2025), NumPy 1.25 for SciPy 1.16 (June 2025), and NumPy 1.26 for SciPy 1.17 (Jan 2026), we could probably skip to NumPy 2.1 for SciPy 1.18 (Jun 2026).

@tylerjereddy
Copy link
Contributor Author

tylerjereddy commented Dec 5, 2024

I pushed in a commit that aims to address the latest round of reviewer comments. Python 3.10 will be supported by the upcoming NumPy release, so I suspect we may want to drop 3.10 in sync with them (for the release after that). It may cause a bit of last-minute chaos to bump the lowest NumPy version, maybe do it just after branching?

Edit: there was no need to flush the full CI, forgot [docs only].

@tylerjereddy
Copy link
Contributor Author

I included Matt's addition of ttest_ind-related argument deprecations and a hyperlink to Joren's install instructions section for scipy-stubs (I figure even if the instructions change a bit per above, Joren could always just update that section in the stubs repo a bit).

@jorenham
Copy link
Contributor

jorenham commented Dec 6, 2024

I included Matt's addition of ttest_ind-related argument deprecations and a hyperlink to Joren's install instructions section for scipy-stubs (I figure even if the instructions change a bit per above, Joren could always just update that section in the stubs repo a bit).

I'll make sure that the URL will remain valid for the foreseeable future 👌🏻.

@tylerjereddy
Copy link
Contributor Author

tylerjereddy commented Dec 7, 2024

@mdhaber stats has been moving pretty fast last few days--are we "green light" from your side to branch around tomorrow?

@h-vetinari how about you? any concerns on your end/conda-forge side?

[skip ci] [ci skip] [skip cirrus] [skip circle]
* Add a release note entry for `lombscargle` changes
based on reviewer feedback.

[skip ci] [ci skip] [skip cirrus] [skip circle]
* Update the release notes for SciPy `1.15.0` following
manual inspection of all 123 merged enhancement-labeled PRs with
this milestone.

* Add forgotten array API support section from the release notes
wiki to the RST release notes.

[skip ci] [ci skip] [skip cirrus] [skip circle]
* Update the `.mailmap` file according to scipygh-21940, then
update the authors list in the release notes for SciPy `1.15.0`
accordingly.

[skip ci] [ci skip] [skip cirrus] [skip circle]
* Add issue/PR lists to SciPy `1.15.0` release notes.

[skip ci] [ci skip] [skip cirrus] [skip circle]
* Remove unused subsections in the SciPy `1.15.0` release notes.

[skip ci] [ci skip] [skip cirrus] [skip circle]
* Address the first batch of reviewer comments related
to the draft SciPy `1.15.0` release notes.

[docs only]
* Mention the `scipy-stubs` package in the
release notes.

* Mention `ndonnx` in the array API support section.

* Mention the removal of the functions in `scipy.misc` and the
renaming of `sosfreqz` to `freqz_sos`.

* Remove an undesirable escape character from the release notes
formatting for issue 21044.

* Mention the deprecations of `distance.kulczynski1` and
`distance.sokalmichener`.

* Clarified the current status of batching support for
`scipy.linalg.toeplitz`.
* Mention argument deprecations in `ttest_ind` in the
SciPy `1.15.0` release notes.

* A link to the `scipy-stubs` installation instructions has
been added to the SciPy `1.15.0` release notes.

[docs only]
distribution (e.g. `scipy.stats.norm`) with the new infrastructure.
This can improve the speed and accuracy of existing distributions,
especially for methods not overridden with custom formulas in the
implementation.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#21746 added scipy.stats.{exp,log}

Copy link
Contributor

@jorenham jorenham Dec 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#21752 added scipy.stats.{abs,truncate}

Copy link
Contributor

@mdhaber mdhaber Dec 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tylerjereddy
Copy link
Contributor Author

Ok, I'm doing some fairly fast updating of the relnotes from recently-merged PRs locally for now..

* Deal with scipygh-22020.

* Update the author list one more time in the
SciPy `1.15.0` release notes, before branching.

* Update the issue/PR lists before branching,
and update the release notes again for ENH PRs
that were recently merged.

* Remove the tentative mention of docs relating
to new stats distn infra for now.

[docs only]
@tylerjereddy
Copy link
Contributor Author

I've pushed in the latest revisions/relnotes additions--we'll see if the docs build in CI agrees with the fix for gh-22020.

Comment on lines 302 to 303
- Added ``OrderStatisticDistribution``, a class representing the distribution
of order statistics of samples drawn from another distribution.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OrderStatisticDistribution isn't public, and the scipy.stats.order_statistic factory function is what should be used for this:
https://github.com/scipy/scipy/pull/21790/files

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- Added ``OrderStatisticDistribution``, a class representing the distribution
of order statistics of samples drawn from another distribution.

I'll add this (order_statistic) up above.

other than ``(2, 2)``.
- `scipy.stats.truncate` adds the ability to truncate the distribution
underlying a random variable and `scipy.stats.abs` provides the ability to
take the absolute value of a random variable. The new ``FoldedDistribution``
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think the FoldedDistribution is exposed in scipy.stats, or am I wrong here, @mdhaber ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, thanks! I will read through this again now.

Comment on lines 302 to 303
- Added ``OrderStatisticDistribution``, a class representing the distribution
of order statistics of samples drawn from another distribution.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- Added ``OrderStatisticDistribution``, a class representing the distribution
of order statistics of samples drawn from another distribution.

Comment on lines 304 to 305
- It is now possible to generate quasi-random samples from statistical
distributions by providing a ``QMCEngine`` object to the ``rng`` argument.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- It is now possible to generate quasi-random samples from statistical
distributions by providing a ``QMCEngine`` object to the ``rng`` argument.

This is only part of the sample method of the new RV objects, so it's not really an enhancement that needs to be covered separately.

Comment on lines 302 to 303
- Added ``OrderStatisticDistribution``, a class representing the distribution
of order statistics of samples drawn from another distribution.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- Added ``OrderStatisticDistribution``, a class representing the distribution
of order statistics of samples drawn from another distribution.

I'll add this (order_statistic) up above.

Comment on lines 297 to 301
- `scipy.stats.truncate` adds the ability to truncate the distribution
underlying a random variable and `scipy.stats.abs` provides the ability to
take the absolute value of a random variable. The new ``FoldedDistribution``
represents the distribution underlying the absolute value of a random
variable.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- `scipy.stats.truncate` adds the ability to truncate the distribution
underlying a random variable and `scipy.stats.abs` provides the ability to
take the absolute value of a random variable. The new ``FoldedDistribution``
represents the distribution underlying the absolute value of a random
variable.


- `scipy.stats.Mixture` has been added to represent mixture distributions.
Copy link
Contributor

@mdhaber mdhaber Dec 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- `scipy.stats.Mixture` has been added to represent mixture distributions.
- `scipy.stats.Mixture` has been added to represent mixture distributions.
- Instances of `scipy.stats.Normal` and the classes returned by
`scipy.stats.make_distribution` are supported by several new
mathematical transformations.
- `scipy.stats.truncate` for truncation of the support.
- `scipy.stats.order_statistic` for the order statistics of a specified number of
IID random variables.
- `scipy.stats.abs`, `scipy.stats.exp`, and `scipy.stats.log`. For example,
``scipy.stats.abs(Normal())`` is distributed according to the folded normal
and ``scipy.stats.exp(Normal())`` is lognormally distributed.

Copy link
Contributor

@mdhaber mdhaber left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this will LGTM after these last few suggestions.

Comment on lines 28 to 29
that all new code use sparse arrays and that developers start to migrate
their existing code from sparse matrix to sparse array:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
that all new code use sparse arrays and that developers start to migrate
their existing code from sparse matrix to sparse array:
that all new code use sparse arrays instead of sparse matrices and that developers
start to migrate their existing code from sparse matrix to sparse array:

Comment on lines 38 to 39
- A new distribution infrastructure in `scipy.stats` that includes speed
and accuracy improvements to existing distributions.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- A new distribution infrastructure in `scipy.stats` that includes speed
and accuracy improvements to existing distributions.
- New probability distribution features in `scipy.stats` can be used to improve the
speed and accuracy of existing continuous distributions and perform new
probability calculations.

"can be used to" because they don't automatically change the existing instances of rv_continous; rather, they transform them into subclasses of the new ContinuousDistribution.
"perform new probability calculations" refers to the new methods of ContinuousDistribution (e.g. several new log-methods, separate raw, central, and standardized moments, etc.)

Comment on lines 76 to 78
logarithms. To facilitate batch computation, the function is vectorized and
supports several Array API compatible array libraries in addition to
NumPy (see "Array API Standard Support" below).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
logarithms. To facilitate batch computation, the function is vectorized and
supports several Array API compatible array libraries in addition to
NumPy (see "Array API Standard Support" below).
logarithms.

This is true, but it's mentioned above. Rather than including this sort of phrasing everywhere (e.g. the next two bullets, too), it's fine with me for it to just appear in select places.


- `scipy.stats.Normal` represents the normal distribution with the new
interface. In most cases, its methods are faster and more accurate than
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
interface. In most cases, its methods are faster and more accurate than
interface. In typical cases, its methods are faster and more accurate than

@h-vetinari
Copy link
Member

@h-vetinari how about you? any concerns on your end/conda-forge side?

I started a testrun based on current main in conda-forge/scipy-feedstock#284, and it looks like everything is passing (emulated tests are still running, but not expecting any issues there)

* Apply final round of suggestion from code review
of SciPy `1.15.0` release notes.

[docs only]
@tylerjereddy
Copy link
Contributor Author

Alright, thanks Matt and Joren for helping with last-minute cleanups. I think got most of the recent round of comments dealt with now and if the docs CI passes again I'll likely squash merge and branch.

@tylerjereddy tylerjereddy changed the title WIP, DOC: SciPy 1.15.0 relnotes DOC: SciPy 1.15.0 relnotes Dec 8, 2024
@tylerjereddy tylerjereddy merged commit a3e07fb into scipy:main Dec 8, 2024
33 of 34 checks passed
@tylerjereddy tylerjereddy deleted the treddy_115_relnotes branch December 8, 2024 05:18
@h-vetinari
Copy link
Member

@h-vetinari how about you? any concerns on your end/conda-forge side?

I started a testrun based on current main in conda-forge/scipy-feedstock#284, and it looks like everything is passing (emulated tests are still running, but not expecting any issues there)

Quick update: I opened #22022 (non-critical), and there's some new timeouts in emulation test_quad_vec_pool and test_quad_vec_pool_args.

More interestingly, after working around a conda-build issue, it also looks like we'll be able to add free-threading builds for scipy right away (it builds & test suite passes 🥳).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Documentation Issues related to the SciPy documentation. Also check https://github.com/scipy/scipy.org
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants