Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hdf5: make possible to use the v1.10 API #118454

Merged
merged 4 commits into from
Apr 8, 2021

Conversation

LeSuisse
Copy link
Contributor

@LeSuisse LeSuisse commented Apr 4, 2021

Motivation for this change

hdf5 v1.12 has introduced breaking changes to the API and some dependent packages are not yet compatible with it. This contribution adds a way to build the hdf5 library with the v1.10 API.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@LeSuisse LeSuisse requested a review from FRidh April 4, 2021 11:12
@LeSuisse
Copy link
Contributor Author

LeSuisse commented Apr 4, 2021

@FRidh you mentioned an issue with pandas but I was not able to find the failure on Hydra, what did I miss?

@FRidh
Copy link
Member

FRidh commented Apr 4, 2021

This needs to target staging-next.

pandas segfaults in a tables test. tables is the pytables package that uses hdf5. https://hydra.nixos.org/build/140676472

@LeSuisse
Copy link
Contributor Author

LeSuisse commented Apr 4, 2021

It looks like this solves the issue for vigra but pandas still segfault :/

@r-rmcgibbo

This comment has been minimized.

@FRidh
Copy link
Member

FRidh commented Apr 4, 2021

It looks like this solves the issue for vigra but pandas still segfault :/

Good vigra is fixed. Pandas could have another cause then, or it is the update itself despite using the old api. To check, it is worthwhile building pandas against the old hdf5.

@LeSuisse
Copy link
Contributor Author

LeSuisse commented Apr 4, 2021

Good vigra is fixed. Pandas could have another cause then, or it is the update itself despite using the old api. To check, it is worthwhile building pandas against the old hdf5.

It tried to revert the upgrade and build pandas and the segfault does not happen.

I'm trying to see what's wrong with the upgrade.

@FRidh
Copy link
Member

FRidh commented Apr 4, 2021

In case we won't figure out (within a day or two, so staging-next can proceed), the alternative is to keep 1.11 in just for pandas.

@SuperSandro2000
Copy link
Member

SuperSandro2000 commented Apr 5, 2021

@FRidh you mentioned an issue with pandas but I was not able to find the failure on Hydra, what did I miss?

I did a dirty fix for pandas in #118512

Edit: Please revert when you have decided what to do with pandas b86327e

SuperSandro2000 added a commit to SuperSandro2000/nixpkgs that referenced this pull request Apr 5, 2021
This reverts commit b86327e.

This is a temporary hack which can't be merged into master at some point
because the package is actually broken. The real issue will likely be fixed
in NixOS#118454.
@LeSuisse LeSuisse force-pushed the hdf5-use-v110-api branch from 6ef6455 to abb31df Compare April 6, 2021 17:42
@LeSuisse
Copy link
Contributor Author

LeSuisse commented Apr 6, 2021

I have not made significant progress on this topic and I'm not sure I will in a timely manner. Maybe someone with more experience of the Python ecosystem/pandas could take a look and have more success than me.

In the meantime to unblock the situation I have updated the PR to introduce hdf5_1_10 so it can be used with python3Packages.tables to avoid segfault in the test suite of python3Packages.pandas.

@ofborg ofborg bot added the 8.has: package (new) This PR adds a new package label Apr 6, 2021
@ofborg ofborg bot requested review from 7c6f434c and knedlsepp April 6, 2021 17:51
pkgs/tools/misc/hdf5/1.10.nix Outdated Show resolved Hide resolved
pkgs/tools/misc/hdf5/1.10.nix Outdated Show resolved Hide resolved
pkgs/development/python-modules/pandas/default.nix Outdated Show resolved Hide resolved
LeSuisse added 2 commits April 8, 2021 08:35
Introduce hdf5_1_10 so it can be used by packages encountering issues
hdf5 1.12.0.
Without that, the test suite of python3Packages.pandas segfault.
@LeSuisse LeSuisse force-pushed the hdf5-use-v110-api branch from abb31df to 50d6043 Compare April 8, 2021 06:37
@SuperSandro2000

This comment has been minimized.

@SuperSandro2000
Copy link
Member

SuperSandro2000 commented Apr 8, 2021

This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.

Result of nixpkgs-review pr 118454 run on x86_64-linux 1

37 packages marked as broken and skipped:
  • mlflow-server
  • python38Packages.arviz
  • python38Packages.clifford
  • python38Packages.dask-jobqueue
  • python38Packages.dask-xgboost
  • python38Packages.dftfit
  • python38Packages.geopandas
  • python38Packages.lammps-cython
  • python38Packages.mlflow
  • python38Packages.mlxtend
  • python38Packages.optuna
  • python38Packages.osmnx
  • python38Packages.pymatgen-lammps
  • python38Packages.pymc3
  • python38Packages.rl-coach
  • python38Packages.shap
  • python38Packages.skorch
  • python38Packages.textacy
  • python38Packages.tiledb
  • python38Packages.trackpy
  • python39Packages.dask-jobqueue
  • python39Packages.dask-xgboost
  • python39Packages.geopandas
  • python39Packages.google-cloud-automl
  • python39Packages.google-cloud-bigquery
  • python39Packages.google-cloud-monitoring
  • python39Packages.lammps-cython
  • python39Packages.mlflow
  • python39Packages.mlxtend
  • python39Packages.optuna
  • python39Packages.osmnx
  • python39Packages.pymatgen-lammps
  • python39Packages.rl-coach
  • python39Packages.skorch
  • python39Packages.textacy
  • python39Packages.tiledb
  • python39Packages.trackpy
18 packages failed to build and already failed to build on hydra master:
  • apache-airflow: log was empty
  • python38Packages.cnvkit: log was empty
  • python38Packages.dask-glm: log was empty
  • python38Packages.dask-ml: log was empty
  • python38Packages.imgaug: log was empty
  • python38Packages.nilearn: log was empty
  • python38Packages.pywick: log was empty
  • python38Packages.tablib: log was empty
  • python38Packages.transformers: log was empty
  • python38Packages.xgboost: log was empty
  • python39Packages.cnvkit: log was empty
  • python39Packages.dask-glm: log was empty
  • python39Packages.imgaug: log was empty
  • python39Packages.nilearn: log was empty
  • python39Packages.pywick: log was empty
  • python39Packages.tablib: log was empty
  • python39Packages.transformers: log was empty
  • python39Packages.xgboost: log was empty
2 packages failed to build and are new build failures:
  • python38Packages.spyder-kernels_0_5: log was empty
  • python38Packages.spyder_3: log was empty
336 packages built:
  • aiodnsbrute
  • backblaze-b2
  • check_systemd
  • checkov
  • conan
  • cq-editor
  • csvs-to-sqlite (python38Packages.csvs-to-sqlite)
  • deja-dup
  • dpt-rp1-py
  • duplicity
  • duply
  • enblend-enfuse
  • evdevremapkeys
  • evillimiter
  • ffmpeg-normalize
  • gdown (python38Packages.gdown)
  • gnome-keysign
  • graph-cli
  • hdf5_1_10
  • home-assistant
  • hugin
  • jellyfin-mpv-shim
  • kaggle (python38Packages.kaggle)
  • keepkey_agent (python38Packages.keepkey_agent)
  • ldeep
  • ledger_agent (python38Packages.ledger_agent)
  • lieer
  • luigi
  • magic-wormhole (python38Packages.magic-wormhole)
  • manim
  • mathlibtools (python38Packages.mathlibtools)
  • mnemosyne
  • noto-fonts-emoji
  • nrfutil
  • ocrmypdf
  • papis (python38Packages.papis)
  • past-time
  • plex-mpv-shim
  • python38Packages.Quandl (python38Packages.quandl)
  • python38Packages.acoustics
  • python38Packages.afdko
  • python38Packages.aioinflux
  • python38Packages.aiosmb
  • python38Packages.altair
  • python38Packages.ansible-kernel
  • python38Packages.ansible-runner
  • python38Packages.aplpy
  • python38Packages.apptools
  • python38Packages.asdf
  • python38Packages.atomman
  • python38Packages.b2sdk
  • python38Packages.batchgenerators
  • python38Packages.bkcharts
  • python38Packages.caffe
  • python38Packages.catboost
  • python38Packages.cirq
  • python38Packages.cufflinks
  • python38Packages.dask
  • python38Packages.dask-gateway
  • python38Packages.dask-image
  • python38Packages.dask-mpi
  • python38Packages.datasets
  • python38Packages.datashader
  • python38Packages.dipy
  • python38Packages.distributed
  • python38Packages.dnachisel
  • python38Packages.drms
  • python38Packages.duckdb
  • python38Packages.fastparquet
  • python38Packages.flammkuchen
  • python38Packages.genome-collector
  • python38Packages.glymur
  • python38Packages.google-cloud-automl
  • python38Packages.google-cloud-bigquery
  • python38Packages.google-cloud-monitoring
  • python38Packages.gradient
  • python38Packages.gradient-utils
  • python38Packages.graphtage
  • python38Packages.graspologic
  • python38Packages.gtts
  • python38Packages.holoviews
  • python38Packages.huggingface-hub
  • python38Packages.hvplot
  • python38Packages.hyperopt
  • python38Packages.ignite
  • python38Packages.image-match
  • python38Packages.imagecorruptions
  • python38Packages.imbalanced-learn
  • python38Packages.influxdb
  • python38Packages.influxgraph
  • python38Packages.intake
  • python38Packages.internetarchive
  • python38Packages.ipydatawidgets
  • python38Packages.labelbox
  • python38Packages.libagent
  • python38Packages.lsassy
  • python38Packages.mayavi
  • python38Packages.mesa
  • python38Packages.moviepy
  • python38Packages.mplfinance
  • python38Packages.msldap
  • python38Packages.nagiosplugin
  • python38Packages.nbsmoke
  • python38Packages.ndjson
  • python38Packages.nipype
  • python38Packages.nltk
  • python38Packages.nototools
  • python38Packages.osmpythontools
  • python38Packages.pandas
  • python38Packages.pandas-datareader
  • python38Packages.panel
  • python38Packages.papermill
  • python38Packages.parfive
  • python38Packages.partd
  • python38Packages.perfplot
  • python38Packages.phik
  • python38Packages.pims
  • python38Packages.plexapi
  • python38Packages.ppscore
  • python38Packages.proglog
  • python38Packages.pvlib
  • python38Packages.pyarrow
  • python38Packages.pybids
  • python38Packages.pyfftw
  • python38Packages.pygmt
  • python38Packages.pymatgen
  • python38Packages.pyppeteer
  • python38Packages.pypykatz
  • python38Packages.pyregion
  • python38Packages.pytest-astropy
  • python38Packages.pytest-astropy-header
  • python38Packages.python-daemon
  • python38Packages.python-miio
  • python38Packages.python-mpv-jsonipc
  • python38Packages.pytrends
  • python38Packages.qiskit
  • python38Packages.qiskit-aer
  • python38Packages.qiskit-aqua
  • python38Packages.qiskit-ibmq-provider
  • python38Packages.qiskit-ignis
  • python38Packages.qiskit-terra
  • python38Packages.radio_beam
  • python38Packages.rapidfuzz
  • python38Packages.rbtools
  • python38Packages.rdkit
  • python38Packages.reproject
  • python38Packages.rising
  • python38Packages.rpy2
  • python38Packages.sacremoses
  • python38Packages.salmon-mail
  • python38Packages.scikit-bio
  • python38Packages.scikitimage
  • python38Packages.seaborn
  • python38Packages.sfepy
  • python38Packages.slicedimage
  • python38Packages.slicer
  • python38Packages.spacy
  • python38Packages.sparse
  • python38Packages.spectral-cube
  • python38Packages.statsmodels
  • python38Packages.streamz
  • python38Packages.stumpy
  • python38Packages.stytra
  • python38Packages.sumo
  • python38Packages.sunpy
  • python38Packages.tables
  • python38Packages.tensorly
  • python38Packages.test-tube
  • python38Packages.thinc
  • python38Packages.toggl-cli
  • python38Packages.tqdm
  • python38Packages.traittypes
  • trezor_agent (python38Packages.trezor_agent)
  • python38Packages.tubeup
  • python38Packages.uproot3
  • python38Packages.vega
  • python38Packages.vega_datasets
  • python38Packages.vidstab
  • python38Packages.wrf-python
  • python38Packages.xarray
  • python38Packages.yfinance
  • python39Packages.Quandl (python39Packages.quandl)
  • python39Packages.acoustics
  • python39Packages.afdko
  • python39Packages.aioinflux
  • python39Packages.aiosmb
  • python39Packages.altair
  • python39Packages.ansible-kernel
  • python39Packages.ansible-runner
  • python39Packages.aplpy
  • python39Packages.apptools
  • python39Packages.asdf
  • python39Packages.atomman
  • python39Packages.b2sdk
  • python39Packages.batchgenerators
  • python39Packages.bkcharts
  • python39Packages.caffe
  • python39Packages.catboost
  • python39Packages.cirq
  • python39Packages.csvs-to-sqlite
  • python39Packages.cufflinks
  • python39Packages.dask
  • python39Packages.dask-gateway
  • python39Packages.dask-image
  • python39Packages.dask-mpi
  • python39Packages.datasets
  • python39Packages.dipy
  • python39Packages.distributed
  • python39Packages.dnachisel
  • python39Packages.drms
  • python39Packages.duckdb
  • python39Packages.flammkuchen
  • python39Packages.gdown
  • python39Packages.genome-collector
  • python39Packages.glymur
  • python39Packages.gradient
  • python39Packages.gradient-utils
  • python39Packages.graphtage
  • python39Packages.gtts
  • python39Packages.holoviews
  • python39Packages.huggingface-hub
  • python39Packages.hvplot
  • python39Packages.hyperopt
  • python39Packages.ignite
  • python39Packages.image-match
  • python39Packages.imagecorruptions
  • python39Packages.imbalanced-learn
  • python39Packages.influxdb
  • python39Packages.influxgraph
  • python39Packages.intake
  • python39Packages.internetarchive
  • python39Packages.ipydatawidgets
  • python39Packages.kaggle
  • python39Packages.keepkey_agent
  • python39Packages.labelbox
  • python39Packages.ledger_agent
  • python39Packages.libagent
  • python39Packages.lsassy
  • python39Packages.magic-wormhole
  • python39Packages.mathlibtools
  • python39Packages.mayavi
  • python39Packages.mesa
  • python39Packages.moviepy
  • python39Packages.mplfinance
  • python39Packages.msldap
  • python39Packages.nagiosplugin
  • python39Packages.nbsmoke
  • python39Packages.ndjson
  • python39Packages.nipype
  • python39Packages.nltk
  • python39Packages.nototools
  • python39Packages.osmpythontools
  • python39Packages.pandas
  • python39Packages.pandas-datareader
  • python39Packages.panel
  • python39Packages.papermill
  • python39Packages.papis
  • python39Packages.parfive
  • python39Packages.partd
  • python39Packages.perfplot
  • python39Packages.pims
  • python39Packages.plexapi
  • python39Packages.ppscore
  • python39Packages.proglog
  • python39Packages.pvlib
  • python39Packages.pyarrow
  • python39Packages.pybids
  • python39Packages.pyfftw
  • python39Packages.pygmt
  • python39Packages.pymatgen
  • python39Packages.pyppeteer
  • python39Packages.pypykatz
  • python39Packages.pyregion
  • python39Packages.pytest-astropy
  • python39Packages.pytest-astropy-header
  • python39Packages.python-daemon
  • python39Packages.python-miio
  • python39Packages.python-mpv-jsonipc
  • python39Packages.pytrends
  • python39Packages.qiskit
  • python39Packages.qiskit-aer
  • python39Packages.qiskit-aqua
  • python39Packages.qiskit-ibmq-provider
  • python39Packages.qiskit-ignis
  • python39Packages.qiskit-terra
  • python39Packages.radio_beam
  • python39Packages.rapidfuzz
  • python39Packages.rbtools
  • python39Packages.rdkit
  • python39Packages.reproject
  • python39Packages.rising
  • python39Packages.rpy2
  • python39Packages.sacremoses
  • python39Packages.salmon-mail
  • python39Packages.scikit-bio
  • python39Packages.scikitimage
  • python39Packages.seaborn
  • python39Packages.sfepy
  • python39Packages.slicedimage
  • python39Packages.slicer
  • python39Packages.spacy
  • python39Packages.spectral-cube
  • python39Packages.spyder-kernels_0_5
  • python39Packages.spyder_3
  • python39Packages.statsmodels
  • python39Packages.streamz
  • python39Packages.sumo
  • python39Packages.sunpy
  • python39Packages.tables
  • python39Packages.test-tube
  • python39Packages.thinc
  • python39Packages.tqdm
  • python39Packages.traittypes
  • python39Packages.trezor_agent
  • python39Packages.tubeup
  • python39Packages.uproot3
  • python39Packages.vega
  • python39Packages.vega_datasets
  • python39Packages.vidstab
  • python39Packages.wrf-python
  • python39Packages.xarray
  • python39Packages.yfinance
  • r128gain
  • sacad
  • saga
  • spotdl
  • streamlit
  • tartube
  • tebreak
  • theharvester
  • tinyprog
  • truvari
  • twitter-color-emoji
  • urlwatch
  • vigra
  • visidata

the two build failures are probably not reproducible.

Merging right now because I want to test home-assistant on staging/staging-next.

@SuperSandro2000 SuperSandro2000 merged commit 9f7f61b into NixOS:staging-next Apr 8, 2021
@LeSuisse LeSuisse deleted the hdf5-use-v110-api branch April 8, 2021 09:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants