Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Close HDF5 dual migration; stay on 1.14.3 until 2.0 is out #6900

Open
h-vetinari opened this issue Jan 6, 2025 · 8 comments
Open

Close HDF5 dual migration; stay on 1.14.3 until 2.0 is out #6900

h-vetinari opened this issue Jan 6, 2025 · 8 comments

Comments

@h-vetinari
Copy link
Member

In #6758, we started migrating hdf5 twice, due to a nasty regression in 1.14.4 on windows that upstream refuses to address until a putative v2.0.

Since HDF5 is part of some very heavy feedstocks, the simple fact of hitting the dual migration on hdf5 causes massive resource starvation in our CI fleet. This is not related to the migration itself though - once merged, all other PRs to such feedstocks will be similarly blown up in impact. Examples from today:

Given that upstream hdf5 plans to only fix this issue in 2.0, the practical reality is that we IMO need to drop everything post 1.14.3, then then bite the bullet to do a monster 2.0 migration if/once that is released and the issue is actually fixed.

CC @conda-forge/hdf5 @conda-forge/core

@hmaarrfk
Copy link
Contributor

hmaarrfk commented Jan 6, 2025

Honestly given their responses that seemed to ignore the urgency of our requests I was going to suggest torevert their changes.

@h-vetinari
Copy link
Member Author

You mean that we carry HDFGroup/hdf5#5197? That seems problematic IMO - we then become responsible for a conda-forge-specific variant of hdf5>1.14.3 that's not found anywhere else. CC @takluyver for the h5py-side

I mean, all current available options are bad. I won't object to carrying a patch1, if other affected parties agree that this is the least bad option. But doing that in direct opposition to upstream needs to clear a very high bar in terms of consensus.

Footnotes

  1. if it allows us to keep migrating for a single current hdf5; staying up-to-date is apparently even CVE-relevant - fun!

@hmaarrfk
Copy link
Contributor

hmaarrfk commented Jan 6, 2025

I had an issue open on the hdf5 feedstock, could we discuss there instead of here?

I feel like this is an hdf5 issue and not a conda-forge issue.

@h-vetinari
Copy link
Member Author

I'm fine to discuss the details in conda-forge/hdf5-feedstock#240.

OTOH, the resource starvation caused by the current dual migration (which looks like it won't change for many months) is an issue for this repo, and IMO we need to completely delete (not just pause) that dual migrator until there's a sustainable solution for hdf5>1.14.3 on windows.

@hmaarrfk
Copy link
Contributor

hmaarrfk commented Jan 6, 2025

The dual migration intent was to give us time to speak with upstream. But they aren’t willing to prioritize this.

I’m ok deleting it and reconsidering our patching options more slowly.

I wasn’t aware there were resources starvation issues

@h-vetinari
Copy link
Member Author

h-vetinari commented Jan 6, 2025

I wasn’t aware there were resources starvation issues

Check out the PRs I linked in the OP - I didn't take all of them, just the biggest ones. For example, fenics got 2 PRs which >60 jobs each (each one taking ~1h).

@hmaarrfk
Copy link
Contributor

hmaarrfk commented Jan 6, 2025

I mean the python matrix and openmpi matrix don’t help either…..

@hmaarrfk hmaarrfk closed this as completed Jan 6, 2025
@h-vetinari
Copy link
Member Author

I mean the python matrix and openmpi matrix don’t help either…..

Definitely. Though those are somewhat irreducible, whereas the hdf5 would be avoidable.

I'm reopening this issue in case other people still want to chime in (easier to see open rather than closed issues). We can close in 1-2 days if there's no further input.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants