[Bug]: numpy 2.0 causes pynwb ZarrIO tutorial to fail

[pauladkisson]

What happened?

Tried to follow pynwb ZarrIO tutorial as a base case for working with ZarrIO, but it fails with the latest hdmf_zarr from github.

Tried downgrading numpy from 2.0 --> 1.26.4 and the error was resolved.

Steps to Reproduce

# Create the NWBFile. Substitute your NWBFile generation here.
nwbfile = NWBFile(
    session_description="my first synthetic recording",
    identifier="EXAMPLE_ID",
    session_start_time=datetime.now(tzlocal()),
    session_id="LONELYMTN",
)

data_with_zarr_data_io = ZarrDataIO(
    data=np.random.randn(100, 100),
    chunks=(10, 10),
    fillvalue=0,
    compressor=Blosc(cname='zstd', clevel=3, shuffle=Blosc.SHUFFLE)
)
nwbfile.add_acquisition(
    TimeSeries(
        name="synthetic_timeseries",
        data=data_with_zarr_data_io,
        unit="m",
        rate=10e3,
    )
)

path = "zarr_tutorial.nwb.zarr"
with NWBZarrIO(path=path, mode="w") as io:
    io.write(nwbfile)

Traceback

Traceback (most recent call last):
  File "/Users/pauladkisson/Documents/CatalystNeuro/HDMF/hdmf-zarr/temp_test.py", line 5, in <module>
    from pynwb import NWBFile, TimeSeries
  File "/opt/anaconda3/envs/hdmf_zarr_env/lib/python3.9/site-packages/pynwb/__init__.py", line 8, in <module>
    import h5py
  File "/opt/anaconda3/envs/hdmf_zarr_env/lib/python3.9/site-packages/h5py/__init__.py", line 45, in <module>
    from ._conv import register_converters as _register_converters, \
  File "h5py/_conv.pyx", line 1, in init h5py._conv
  File "h5py/h5r.pyx", line 1, in init h5py.h5r
  File "h5py/h5p.pyx", line 1, in init h5py.h5p
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

Operating System

macOS

Python Executable

Conda

Python Version

3.9

Package Versions

For the environment, I followed installation instructions here: https://hdmf-zarr.readthedocs.io/en/latest/installation.html

Code of Conduct

• I agree to follow this project's Code of Conduct
• Have you checked the Contributing document?
• Have you ensured this bug was not already reported?

[mavaylon1]

Hmmm that is weird. Thank you for the issue ticket. I will take a look today to see what is going on!

I will let you of a timeline then.

[mavaylon1]

Hmmm the gallery tests are passing on the scheduler.

[mavaylon1]

Ah the tests are using the numpy v1. I'm not sure why though as the requirements files state numpy 2.

[mavaylon1]

I also tried your example with numpy 2.0 and it works. Can you try creating a clean environment and both trying it out again and show me your pip list

[pauladkisson]

I was already using a clean env, but I tried it out again just in case (same error).

Here is pip list output:

Package                       Version             Editable project location
----------------------------- ------------------- ----------------------------------------------------------
airium                        0.2.6
alabaster                     0.7.16
annotated-types               0.7.0
antlr4-python3-runtime        4.9.3
appdirs                       1.4.4
arrow                         1.3.0
asciitree                     0.3.3
attrs                         24.2.0
babel                         2.16.0
bcp47                         0.1.0
beautifulsoup4                4.12.3
bioregistry                   0.11.14
black                         24.3.0
cachetools                    5.5.0
cattrs                        23.2.3
certifi                       2024.7.4
CFGraph                       0.2.1
cfgv                          3.4.0
chardet                       5.2.0
charset-normalizer            3.3.2
class_resolver                0.4.3
click                         8.1.7
codespell                     2.2.6
colorama                      0.4.6
contourpy                     1.2.1
coverage                      7.3.2
curies                        0.7.10
cycler                        0.12.1
daff                          1.3.46
defusedxml                    0.7.1
Deprecated                    1.2.14
deprecation                   2.1.0
distlib                       0.3.8
docutils                      0.20.1
et-xmlfile                    1.1.0
eutils                        0.6.0
exceptiongroup                1.2.2
fasteners                     0.19
fastobo                       0.12.3
filelock                      3.15.4
fonttools                     4.53.1
fqdn                          1.5.1
funowl                        0.2.3
google                        3.0.0
google-api-core               2.19.1
google-api-python-client      2.142.0
google-auth                   2.34.0
google-auth-httplib2          0.2.0
google-auth-oauthlib          1.2.1
googleapis-common-protos      1.63.2
graphviz                      0.20.3
greenlet                      3.0.3
gspread                       6.1.2
gspread-formatting            1.2.0
h5py                          3.11.0
hbreader                      0.9.1
hdf5plugin                    4.3.0
hdmf                          3.14.3
hdmf_zarr                     0.8.1.dev6+g82c993c /Users/pauladkisson/Documents/CatalystNeuro/HDMF/hdmf-zarr
httplib2                      0.22.0
identify                      2.6.0
idna                          3.7
ijson                         3.3.0
imagesize                     1.4.1
importlib_metadata            8.4.0
importlib_resources           6.4.3
iniconfig                     2.0.0
isodate                       0.6.1
isoduration                   20.11.0
Jinja2                        3.1.4
json-flattener                0.1.9
jsonasobj                     1.3.1
jsonasobj2                    1.0.4
jsonlines                     4.0.0
jsonpatch                     1.33
jsonpath-ng                   1.6.1
jsonpointer                   3.0.0
jsonschema                    4.22.0
jsonschema-specifications     2023.12.1
kgcl-rdflib                   0.5.0
kgcl_schema                   0.6.9
kiwisolver                    1.4.5
lark                          1.2.2
linkml                        1.7.10
linkml-dataops                0.1.0
linkml-renderer               0.3.1
linkml-runtime                1.7.7
lxml                          5.3.0
MarkupSafe                    2.1.5
matplotlib                    3.9.2
mkl-fft                       1.3.8
mkl-random                    1.2.4
mkl-service                   2.4.0
more-click                    0.1.2
mypy-extensions               1.0.0
ndex2                         3.9.0
networkx                      3.2.1
nodeenv                       1.9.1
numcodecs                     0.12.1
numpy                         2.0.0
oaklib                        0.6.10
oauthlib                      3.2.2
ols-client                    0.1.4
ontodev-cogs                  0.3.3
ontoportal-client             0.0.4
openpyxl                      3.1.5
packaging                     24.1
pandas                        2.2.2
pansql                        0.0.1
parse                         1.20.2
pathspec                      0.12.1
pillow                        10.4.0
pip                           24.2
platformdirs                  4.2.2
pluggy                        1.5.0
ply                           3.11
pre-commit                    3.5.0
prefixcommons                 0.1.12
prefixmaps                    0.2.5
pronto                        2.5.7
proto-plus                    1.24.0
protobuf                      5.27.3
pyasn1                        0.6.0
pyasn1_modules                0.4.0
pydantic                      2.8.2
pydantic_core                 2.20.1
Pygments                      2.18.0
PyJSG                         0.11.10
pynwb                         2.5.0
pyparsing                     3.1.2
pyproject-api                 1.7.1
PyShEx                        0.8.1
PyShExC                       0.9.1
pysolr                        3.9.0
pystow                        0.5.4
pytest                        7.4.3
pytest-cov                    4.1.0
pytest-logging                2015.11.4
python-dateutil               2.8.2
PyTrie                        0.4.0
pytz                          2024.1
PyYAML                        6.0.2
ratelimit                     2.2.1
rdflib                        7.0.0
rdflib-jsonld                 0.6.1
rdflib-shim                   1.0.3
referencing                   0.35.1
requests                      2.32.3
requests-cache                1.2.1
requests-oauthlib             2.0.0
requests-toolbelt             1.0.0
rfc3339-validator             0.1.4
rfc3987                       1.3.8
rpds-py                       0.20.0
rsa                           4.9
ruamel.yaml                   0.18.2
ruamel.yaml.clib              0.2.8
ruff                          0.1.3
schemasheets                  0.2.1
scipy                         1.13.1
semsql                        0.3.3
setuptools                    72.1.0
ShExJSG                       0.8.2
six                           1.16.0
snowballstemmer               2.2.0
sortedcontainers              2.4.0
soupsieve                     2.6
sparqlslurper                 0.5.1
SPARQLWrapper                 2.0.0
Sphinx                        7.4.7
sphinx-copybutton             0.5.2
sphinx-gallery                0.17.1
sphinx-rtd-theme              2.0.0
sphinxcontrib-applehelp       2.0.0
sphinxcontrib-devhelp         2.0.0
sphinxcontrib-htmlhelp        2.1.0
sphinxcontrib-jquery          4.1
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          2.0.0
sphinxcontrib-serializinghtml 2.0.0
SQLAlchemy                    2.0.32
SQLAlchemy-Utils              0.38.3
sssom                         0.4.11
sssom-schema                  0.15.2
tabulate                      0.9.0
tenacity                      8.5.0
termcolor                     2.4.0
threadpoolctl                 3.2.0
tomli                         2.0.1
tox                           4.11.3
tqdm                          4.66.4
types-python-dateutil         2.9.0.20240821
typing_extensions             4.12.2
tzdata                        2024.1
uri-template                  1.3.0
uritemplate                   4.1.1
url-normalize                 1.4.3
urllib3                       2.2.2
validators                    0.33.0
virtualenv                    20.26.3
watchdog                      4.0.2
webcolors                     24.8.0
wheel                         0.43.0
wrapt                         1.16.0
zarr                          2.16.1
zipp                          3.20.0

[mavaylon1]

@pauladkisson could you do the following:

1. activate the env
2. touch environment.yml
3. conda env export > environment.yml
4. paste the content of that file here

[mavaylon1]

With this I can recreate the exact env on my end to try it out.

[pauladkisson]

name: hdmf_zarr_env2
channels:
  - defaults
dependencies:
  - blas=1.0=mkl
  - c-ares=1.19.1=h6c40b1e_0
  - ca-certificates=2024.7.2=hecd8cb5_0
  - h5py=3.11.0=py39h8cd6c6c_0
  - hdf5=1.12.1=ha01d115_3
  - intel-openmp=2023.1.0=ha357a0b_43548
  - krb5=1.20.1=h428f121_1
  - libcurl=8.7.1=hf20ceda_0
  - libcxx=14.0.6=h9765a3e_0
  - libedit=3.1.20230828=h6c40b1e_0
  - libev=4.33=h9ed2024_1
  - libffi=3.4.4=hecd8cb5_1
  - libgfortran=5.0.0=11_3_0_hecd8cb5_28
  - libgfortran5=11.3.0=h9dfd629_28
  - libnghttp2=1.57.0=h9beae6a_0
  - libssh2=1.11.0=hf20ceda_0
  - llvm-openmp=14.0.6=h0dcd299_0
  - mkl=2023.1.0=h8e150cf_43560
  - mkl-service=2.4.0=py39h6c40b1e_1
  - mkl_fft=1.3.8=py39h6c40b1e_0
  - mkl_random=1.2.4=py39ha357a0b_0
  - ncurses=6.4=hcec6c5f_0
  - openssl=3.0.14=h46256e1_0
  - pip=24.2=py39hecd8cb5_0
  - python=3.9.19=h5ee71fb_1
  - readline=8.2=hca72f7f_0
  - setuptools=72.1.0=py39hecd8cb5_0
  - sqlite=3.45.3=h6c40b1e_0
  - tbb=2021.8.0=ha357a0b_0
  - tk=8.6.14=h4d00af3_0
  - wheel=0.43.0=py39hecd8cb5_0
  - xz=5.4.6=h6c40b1e_1
  - zlib=1.2.13=h4b97444_1
  - pip:
      - airium==0.2.6
      - alabaster==0.7.16
      - annotated-types==0.7.0
      - antlr4-python3-runtime==4.9.3
      - appdirs==1.4.4
      - arrow==1.3.0
      - asciitree==0.3.3
      - attrs==24.2.0
      - babel==2.16.0
      - bcp47==0.1.0
      - beautifulsoup4==4.12.3
      - bioregistry==0.11.14
      - black==24.3.0
      - cachetools==5.5.0
      - cattrs==23.2.3
      - certifi==2024.7.4
      - cfgraph==0.2.1
      - cfgv==3.4.0
      - chardet==5.2.0
      - charset-normalizer==3.3.2
      - class-resolver==0.4.3
      - click==8.1.7
      - codespell==2.2.6
      - colorama==0.4.6
      - contourpy==1.2.1
      - coverage==7.3.2
      - curies==0.7.10
      - cycler==0.12.1
      - daff==1.3.46
      - defusedxml==0.7.1
      - deprecated==1.2.14
      - deprecation==2.1.0
      - distlib==0.3.8
      - docutils==0.20.1
      - et-xmlfile==1.1.0
      - eutils==0.6.0
      - exceptiongroup==1.2.2
      - fasteners==0.19
      - fastobo==0.12.3
      - filelock==3.15.4
      - fonttools==4.53.1
      - fqdn==1.5.1
      - funowl==0.2.3
      - google==3.0.0
      - google-api-core==2.19.1
      - google-api-python-client==2.142.0
      - google-auth==2.34.0
      - google-auth-httplib2==0.2.0
      - google-auth-oauthlib==1.2.1
      - googleapis-common-protos==1.63.2
      - greenlet==3.0.3
      - gspread==6.1.2
      - gspread-formatting==1.2.0
      - hbreader==0.9.1
      - hdf5plugin==4.3.0
      - hdmf==3.14.3
      - hdmf-zarr==0.8.1.dev6+g82c993c
      - httplib2==0.22.0
      - identify==2.6.0
      - idna==3.7
      - ijson==3.3.0
      - imagesize==1.4.1
      - importlib-metadata==8.4.0
      - importlib-resources==6.4.3
      - iniconfig==2.0.0
      - isodate==0.6.1
      - isoduration==20.11.0
      - jinja2==3.1.4
      - json-flattener==0.1.9
      - jsonasobj==1.3.1
      - jsonasobj2==1.0.4
      - jsonlines==4.0.0
      - jsonpatch==1.33
      - jsonpath-ng==1.6.1
      - jsonpointer==3.0.0
      - jsonschema==4.22.0
      - jsonschema-specifications==2023.12.1
      - kgcl-rdflib==0.5.0
      - kgcl-schema==0.6.9
      - kiwisolver==1.4.5
      - lark==1.2.2
      - linkml==1.7.10
      - linkml-dataops==0.1.0
      - linkml-renderer==0.3.1
      - linkml-runtime==1.7.7
      - lxml==5.3.0
      - markupsafe==2.1.5
      - matplotlib==3.9.2
      - more-click==0.1.2
      - mypy-extensions==1.0.0
      - ndex2==3.9.0
      - networkx==3.2.1
      - nodeenv==1.9.1
      - numcodecs==0.12.1
      - numpy==2.0.0
      - oaklib==0.6.10
      - oauthlib==3.2.2
      - ols-client==0.1.4
      - ontodev-cogs==0.3.3
      - ontoportal-client==0.0.4
      - openpyxl==3.1.5
      - packaging==24.1
      - pandas==2.2.2
      - pansql==0.0.1
      - parse==1.20.2
      - pathspec==0.12.1
      - pillow==10.4.0
      - platformdirs==4.2.2
      - pluggy==1.5.0
      - ply==3.11
      - pre-commit==3.5.0
      - prefixcommons==0.1.12
      - prefixmaps==0.2.5
      - pronto==2.5.7
      - proto-plus==1.24.0
      - protobuf==5.27.3
      - pyasn1==0.6.0
      - pyasn1-modules==0.4.0
      - pydantic==2.8.2
      - pydantic-core==2.20.1
      - pygments==2.18.0
      - pyjsg==0.11.10
      - pynwb==2.5.0
      - pyparsing==3.1.2
      - pyproject-api==1.7.1
      - pyshex==0.8.1
      - pyshexc==0.9.1
      - pysolr==3.9.0
      - pystow==0.5.4
      - pytest==7.4.3
      - pytest-cov==4.1.0
      - pytest-logging==2015.11.4
      - python-dateutil==2.8.2
      - python-graphviz==0.20.3
      - pytrie==0.4.0
      - pytz==2024.1
      - pyyaml==6.0.2
      - ratelimit==2.2.1
      - rdflib==7.0.0
      - rdflib-jsonld==0.6.1
      - rdflib-shim==1.0.3
      - referencing==0.35.1
      - requests==2.32.3
      - requests-cache==1.2.1
      - requests-oauthlib==2.0.0
      - requests-toolbelt==1.0.0
      - rfc3339-validator==0.1.4
      - rfc3987==1.3.8
      - rpds-py==0.20.0
      - rsa==4.9
      - ruamel-yaml==0.18.2
      - ruamel-yaml-clib==0.2.8
      - ruff==0.1.3
      - schemasheets==0.2.1
      - scipy==1.13.1
      - semsql==0.3.3
      - shexjsg==0.8.2
      - six==1.16.0
      - snowballstemmer==2.2.0
      - sortedcontainers==2.4.0
      - soupsieve==2.6
      - sparqlslurper==0.5.1
      - sparqlwrapper==2.0.0
      - sphinx==7.4.7
      - sphinx-copybutton==0.5.2
      - sphinx-gallery==0.17.1
      - sphinx-rtd-theme==2.0.0
      - sphinxcontrib-applehelp==2.0.0
      - sphinxcontrib-devhelp==2.0.0
      - sphinxcontrib-htmlhelp==2.1.0
      - sphinxcontrib-jquery==4.1
      - sphinxcontrib-jsmath==1.0.1
      - sphinxcontrib-qthelp==2.0.0
      - sphinxcontrib-serializinghtml==2.0.0
      - sqlalchemy==2.0.32
      - sqlalchemy-utils==0.38.3
      - sssom==0.4.11
      - sssom-schema==0.15.2
      - tabulate==0.9.0
      - tenacity==8.5.0
      - termcolor==2.4.0
      - threadpoolctl==3.2.0
      - tomli==2.0.1
      - tox==4.11.3
      - tqdm==4.66.4
      - types-python-dateutil==2.9.0.20240821
      - typing-extensions==4.12.2
      - tzdata==2024.1
      - uri-template==1.3.0
      - uritemplate==4.1.1
      - url-normalize==1.4.3
      - urllib3==2.2.2
      - validators==0.33.0
      - virtualenv==20.26.3
      - watchdog==4.0.2
      - webcolors==24.8.0
      - wrapt==1.16.0
      - zarr==2.16.1
      - zipp==3.20.0
prefix: /opt/anaconda3/envs/hdmf_zarr_env2

[mavaylon1]

Hey @pauladkisson here are a few updates and a rough timeline:

1. I see the error comes from just importing hdf5. I installed hdf5 (your version) and swapped between numpy version and it still works.
2. PyNWB restricts numpy to below 2, but that isn't affecting the workflow and even when I do it locally it works fine.
3. When I try creating your environment, it fails. I see you have packages like linkml in there. In that env, did you just run "pip install hdmf-zarr". I'm curious to how you have linkml because that is an optional package in hdmf.

Since downgrading numpy is essentially working, I will take a look again next week.

[pauladkisson]

Thanks for looking into this.

In that env, did you just run "pip install hdmf-zarr"?

I was following the dev instructions here, so I installed all the requirements as well:

conda create --name hdmf-zarr-test python=3.9
conda activate hdmf-zarr-test
conda install h5py

git clone --recurse-submodules https://github.com/hdmf-dev/hdmf.git
cd hdmf
pip install -r requirements.txt -r requirements-dev.txt -r requirements-doc.txt -r requirements-opt.txt
pip install -e .
cd ..

git clone https://github.com/hdmf-dev/hdmf-zarr.git
cd hdmf-zarr
pip install -r requirements.txt -r requirements-dev.txt -r requirements-doc.txt
pip install -e .

[rly]

Pynwb 2.8.2 was released today which adds support for numpy 2. Gallery tests are now failing: https://github.com/hdmf-dev/hdmf-zarr/actions/runs/10786265512. The two may be related. @mavaylon1 can you take a look?

[rly]

FYI Gallery tests are now passing

[mavaylon1]

Can you try again now? It should be working now due to updates from both pynwb and zarr packages. @pauladkisson

[pauladkisson]

Still failing with new env and latest changes from github.

[pauladkisson]

Here's the environment for reference:

name: hdmf_zarr_env
channels:
  - defaults
dependencies:
  - blas=1.0=mkl
  - c-ares=1.19.1=h6c40b1e_0
  - ca-certificates=2024.9.24=hecd8cb5_0
  - h5py=3.11.0=py39h8cd6c6c_0
  - hdf5=1.12.1=ha01d115_3
  - intel-openmp=2023.1.0=ha357a0b_43548
  - krb5=1.20.1=h428f121_1
  - libcurl=8.9.1=h3a17b82_0
  - libcxx=14.0.6=h9765a3e_0
  - libedit=3.1.20230828=h6c40b1e_0
  - libev=4.33=h9ed2024_1
  - libffi=3.4.4=hecd8cb5_1
  - libgfortran=5.0.0=11_3_0_hecd8cb5_28
  - libgfortran5=11.3.0=h9dfd629_28
  - libnghttp2=1.57.0=h9beae6a_0
  - libssh2=1.11.0=hf20ceda_0
  - llvm-openmp=14.0.6=h0dcd299_0
  - mkl=2023.1.0=h8e150cf_43560
  - mkl-service=2.4.0=py39h6c40b1e_1
  - mkl_fft=1.3.8=py39h6c40b1e_0
  - mkl_random=1.2.4=py39ha357a0b_0
  - ncurses=6.4=hcec6c5f_0
  - openssl=3.0.15=h46256e1_0
  - pip=24.2=py39hecd8cb5_0
  - python=3.9.20=hce00570_1
  - readline=8.2=hca72f7f_0
  - setuptools=75.1.0=py39hecd8cb5_0
  - sqlite=3.45.3=h6c40b1e_0
  - tbb=2021.8.0=ha357a0b_0
  - tk=8.6.14=h4d00af3_0
  - wheel=0.44.0=py39hecd8cb5_0
  - xz=5.4.6=h6c40b1e_1
  - zlib=1.2.13=h4b97444_1
  - pip:
      - airium==0.2.6
      - alabaster==0.7.16
      - annotated-types==0.7.0
      - antlr4-python3-runtime==4.9.3
      - appdirs==1.4.4
      - arrow==1.3.0
      - asciitree==0.3.3
      - attrs==24.2.0
      - babel==2.16.0
      - bcp47==0.1.0
      - beautifulsoup4==4.12.3
      - bioregistry==0.11.18
      - black==24.3.0
      - cachetools==5.5.0
      - cattrs==24.1.2
      - certifi==2024.8.30
      - cfgraph==0.2.1
      - cfgv==3.4.0
      - chardet==5.2.0
      - charset-normalizer==3.4.0
      - class-resolver==0.5.2
      - click==8.1.7
      - codespell==2.2.6
      - colorama==0.4.6
      - contourpy==1.3.0
      - coverage==7.3.2
      - curies==0.8.0
      - cycler==0.12.1
      - daff==1.3.46
      - defusedxml==0.7.1
      - deprecated==1.2.14
      - deprecation==2.1.0
      - distlib==0.3.9
      - docutils==0.21.2
      - et-xmlfile==1.1.0
      - eutils==0.6.0
      - exceptiongroup==1.2.2
      - fasteners==0.19
      - fastobo==0.12.3
      - filelock==3.16.1
      - fonttools==4.54.1
      - fqdn==1.5.1
      - funowl==0.2.3
      - google==3.0.0
      - google-api-core==2.21.0
      - google-api-python-client==2.149.0
      - google-auth==2.35.0
      - google-auth-httplib2==0.2.0
      - google-auth-oauthlib==1.2.1
      - googleapis-common-protos==1.65.0
      - greenlet==3.1.1
      - gspread==6.1.4
      - gspread-formatting==1.2.0
      - hbreader==0.9.1
      - hdf5plugin==4.3.0
      - hdmf==3.14.5
      - hdmf-zarr==0.9.1.dev1+g46bdb77
      - html5lib-modern==1.2
      - httplib2==0.22.0
      - identify==2.6.1
      - idna==3.10
      - ijson==3.3.0
      - imagesize==1.4.1
      - importlib-metadata==8.5.0
      - importlib-resources==6.4.5
      - iniconfig==2.0.0
      - isodate==0.7.2
      - isoduration==20.11.0
      - jinja2==3.1.4
      - json-flattener==0.1.9
      - jsonasobj==1.3.1
      - jsonasobj2==1.0.4
      - jsonlines==4.0.0
      - jsonpatch==1.33
      - jsonpath-ng==1.7.0
      - jsonpointer==3.0.0
      - jsonschema==4.22.0
      - jsonschema-specifications==2024.10.1
      - kgcl-rdflib==0.5.0
      - kgcl-schema==0.6.9
      - kiwisolver==1.4.7
      - lark==1.2.2
      - linkml==1.7.10
      - linkml-dataops==0.1.0
      - linkml-renderer==0.3.1
      - linkml-runtime==1.7.7
      - lxml==5.3.0
      - markupsafe==3.0.2
      - matplotlib==3.9.2
      - more-click==0.1.2
      - mypy-extensions==1.0.0
      - ndex2==3.9.0
      - networkx==3.2.1
      - nodeenv==1.9.1
      - numcodecs==0.12.1
      - numpy==2.0.0
      - oaklib==0.6.10
      - oauthlib==3.2.2
      - ols-client==0.1.4
      - ontodev-cogs==0.3.3
      - ontoportal-client==0.0.4
      - openpyxl==3.1.5
      - packaging==24.1
      - pandas==2.2.2
      - pansql==0.0.1
      - parse==1.20.2
      - pathspec==0.12.1
      - pillow==11.0.0
      - platformdirs==4.3.6
      - pluggy==1.5.0
      - ply==3.11
      - pre-commit==3.5.0
      - prefixcommons==0.1.12
      - prefixmaps==0.2.6
      - pronto==2.5.8
      - proto-plus==1.24.0
      - protobuf==5.28.2
      - pyasn1==0.6.1
      - pyasn1-modules==0.4.1
      - pydantic==2.9.2
      - pydantic-core==2.23.4
      - pygments==2.18.0
      - pyjsg==0.11.10
      - pynwb==2.5.0
      - pyparsing==3.2.0
      - pyproject-api==1.8.0
      - pyshex==0.8.1
      - pyshexc==0.9.1
      - pysolr==3.10.0
      - pystow==0.5.6
      - pytest==7.4.3
      - pytest-cov==4.1.0
      - pytest-logging==2015.11.4
      - python-dateutil==2.8.2
      - python-graphviz==0.20.3
      - pytrie==0.4.0
      - pytz==2024.2
      - pyyaml==6.0.2
      - ratelimit==2.2.1
      - rdflib==7.1.0
      - rdflib-jsonld==0.6.1
      - rdflib-shim==1.0.3
      - referencing==0.35.1
      - requests==2.32.3
      - requests-cache==1.2.1
      - requests-oauthlib==2.0.0
      - requests-toolbelt==1.0.0
      - rfc3339-validator==0.1.4
      - rfc3987==1.3.8
      - rpds-py==0.20.0
      - rsa==4.9
      - ruamel-yaml==0.18.2
      - ruamel-yaml-clib==0.2.12
      - ruff==0.1.3
      - schemasheets==0.2.1
      - scipy==1.13.1
      - semsql==0.3.3
      - shexjsg==0.8.2
      - six==1.16.0
      - snowballstemmer==2.2.0
      - sortedcontainers==2.4.0
      - soupsieve==2.6
      - sparqlslurper==0.5.1
      - sparqlwrapper==2.0.0
      - sphinx==7.4.7
      - sphinx-copybutton==0.5.2
      - sphinx-gallery==0.18.0
      - sphinx-rtd-theme==3.0.1
      - sphinxcontrib-applehelp==2.0.0
      - sphinxcontrib-devhelp==2.0.0
      - sphinxcontrib-htmlhelp==2.1.0
      - sphinxcontrib-jquery==4.1
      - sphinxcontrib-jsmath==1.0.1
      - sphinxcontrib-qthelp==2.0.0
      - sphinxcontrib-serializinghtml==2.0.0
      - sqlalchemy==2.0.36
      - sqlalchemy-utils==0.38.3
      - sssom==0.4.11
      - sssom-schema==0.15.2
      - tabulate==0.9.0
      - tenacity==8.5.0
      - termcolor==2.5.0
      - threadpoolctl==3.2.0
      - tomli==2.0.2
      - tox==4.11.3
      - tqdm==4.66.4
      - types-python-dateutil==2.9.0.20241003
      - typing-extensions==4.12.2
      - tzdata==2024.2
      - uri-template==1.3.0
      - uritemplate==4.1.1
      - url-normalize==1.4.3
      - urllib3==2.2.3
      - validators==0.34.0
      - virtualenv==20.27.0
      - watchdog==5.0.3
      - webcolors==24.8.0
      - wrapt==1.16.0
      - zarr==2.16.1
      - zipp==3.20.2
prefix: /opt/anaconda3/envs/hdmf_zarr_env

[mavaylon1]

Here's the environment for reference:

name: hdmf_zarr_env

channels:

  - defaults

dependencies:

  - blas=1.0=mkl

  - c-ares=1.19.1=h6c40b1e_0

  - ca-certificates=2024.9.24=hecd8cb5_0

  - h5py=3.11.0=py39h8cd6c6c_0

  - hdf5=1.12.1=ha01d115_3

  - intel-openmp=2023.1.0=ha357a0b_43548

  - krb5=1.20.1=h428f121_1

  - libcurl=8.9.1=h3a17b82_0

  - libcxx=14.0.6=h9765a3e_0

  - libedit=3.1.20230828=h6c40b1e_0

  - libev=4.33=h9ed2024_1

  - libffi=3.4.4=hecd8cb5_1

  - libgfortran=5.0.0=11_3_0_hecd8cb5_28

  - libgfortran5=11.3.0=h9dfd629_28

  - libnghttp2=1.57.0=h9beae6a_0

  - libssh2=1.11.0=hf20ceda_0

  - llvm-openmp=14.0.6=h0dcd299_0

  - mkl=2023.1.0=h8e150cf_43560

  - mkl-service=2.4.0=py39h6c40b1e_1

  - mkl_fft=1.3.8=py39h6c40b1e_0

  - mkl_random=1.2.4=py39ha357a0b_0

  - ncurses=6.4=hcec6c5f_0

  - openssl=3.0.15=h46256e1_0

  - pip=24.2=py39hecd8cb5_0

  - python=3.9.20=hce00570_1

  - readline=8.2=hca72f7f_0

  - setuptools=75.1.0=py39hecd8cb5_0

  - sqlite=3.45.3=h6c40b1e_0

  - tbb=2021.8.0=ha357a0b_0

  - tk=8.6.14=h4d00af3_0

  - wheel=0.44.0=py39hecd8cb5_0

  - xz=5.4.6=h6c40b1e_1

  - zlib=1.2.13=h4b97444_1

  - pip:

      - airium==0.2.6

      - alabaster==0.7.16

      - annotated-types==0.7.0

      - antlr4-python3-runtime==4.9.3

      - appdirs==1.4.4

      - arrow==1.3.0

      - asciitree==0.3.3

      - attrs==24.2.0

      - babel==2.16.0

      - bcp47==0.1.0

      - beautifulsoup4==4.12.3

      - bioregistry==0.11.18

      - black==24.3.0

      - cachetools==5.5.0

      - cattrs==24.1.2

      - certifi==2024.8.30

      - cfgraph==0.2.1

      - cfgv==3.4.0

      - chardet==5.2.0

      - charset-normalizer==3.4.0

      - class-resolver==0.5.2

      - click==8.1.7

      - codespell==2.2.6

      - colorama==0.4.6

      - contourpy==1.3.0

      - coverage==7.3.2

      - curies==0.8.0

      - cycler==0.12.1

      - daff==1.3.46

      - defusedxml==0.7.1

      - deprecated==1.2.14

      - deprecation==2.1.0

      - distlib==0.3.9

      - docutils==0.21.2

      - et-xmlfile==1.1.0

      - eutils==0.6.0

      - exceptiongroup==1.2.2

      - fasteners==0.19

      - fastobo==0.12.3

      - filelock==3.16.1

      - fonttools==4.54.1

      - fqdn==1.5.1

      - funowl==0.2.3

      - google==3.0.0

      - google-api-core==2.21.0

      - google-api-python-client==2.149.0

      - google-auth==2.35.0

      - google-auth-httplib2==0.2.0

      - google-auth-oauthlib==1.2.1

      - googleapis-common-protos==1.65.0

      - greenlet==3.1.1

      - gspread==6.1.4

      - gspread-formatting==1.2.0

      - hbreader==0.9.1

      - hdf5plugin==4.3.0

      - hdmf==3.14.5

      - hdmf-zarr==0.9.1.dev1+g46bdb77

      - html5lib-modern==1.2

      - httplib2==0.22.0

      - identify==2.6.1

      - idna==3.10

      - ijson==3.3.0

      - imagesize==1.4.1

      - importlib-metadata==8.5.0

      - importlib-resources==6.4.5

      - iniconfig==2.0.0

      - isodate==0.7.2

      - isoduration==20.11.0

      - jinja2==3.1.4

      - json-flattener==0.1.9

      - jsonasobj==1.3.1

      - jsonasobj2==1.0.4

      - jsonlines==4.0.0

      - jsonpatch==1.33

      - jsonpath-ng==1.7.0

      - jsonpointer==3.0.0

      - jsonschema==4.22.0

      - jsonschema-specifications==2024.10.1

      - kgcl-rdflib==0.5.0

      - kgcl-schema==0.6.9

      - kiwisolver==1.4.7

      - lark==1.2.2

      - linkml==1.7.10

      - linkml-dataops==0.1.0

      - linkml-renderer==0.3.1

      - linkml-runtime==1.7.7

      - lxml==5.3.0

      - markupsafe==3.0.2

      - matplotlib==3.9.2

      - more-click==0.1.2

      - mypy-extensions==1.0.0

      - ndex2==3.9.0

      - networkx==3.2.1

      - nodeenv==1.9.1

      - numcodecs==0.12.1

      - numpy==2.0.0

      - oaklib==0.6.10

      - oauthlib==3.2.2

      - ols-client==0.1.4

      - ontodev-cogs==0.3.3

      - ontoportal-client==0.0.4

      - openpyxl==3.1.5

      - packaging==24.1

      - pandas==2.2.2

      - pansql==0.0.1

      - parse==1.20.2

      - pathspec==0.12.1

      - pillow==11.0.0

      - platformdirs==4.3.6

      - pluggy==1.5.0

      - ply==3.11

      - pre-commit==3.5.0

      - prefixcommons==0.1.12

      - prefixmaps==0.2.6

      - pronto==2.5.8

      - proto-plus==1.24.0

      - protobuf==5.28.2

      - pyasn1==0.6.1

      - pyasn1-modules==0.4.1

      - pydantic==2.9.2

      - pydantic-core==2.23.4

      - pygments==2.18.0

      - pyjsg==0.11.10

      - pynwb==2.5.0

      - pyparsing==3.2.0

      - pyproject-api==1.8.0

      - pyshex==0.8.1

      - pyshexc==0.9.1

      - pysolr==3.10.0

      - pystow==0.5.6

      - pytest==7.4.3

      - pytest-cov==4.1.0

      - pytest-logging==2015.11.4

      - python-dateutil==2.8.2

      - python-graphviz==0.20.3

      - pytrie==0.4.0

      - pytz==2024.2

      - pyyaml==6.0.2

      - ratelimit==2.2.1

      - rdflib==7.1.0

      - rdflib-jsonld==0.6.1

      - rdflib-shim==1.0.3

      - referencing==0.35.1

      - requests==2.32.3

      - requests-cache==1.2.1

      - requests-oauthlib==2.0.0

      - requests-toolbelt==1.0.0

      - rfc3339-validator==0.1.4

      - rfc3987==1.3.8

      - rpds-py==0.20.0

      - rsa==4.9

      - ruamel-yaml==0.18.2

      - ruamel-yaml-clib==0.2.12

      - ruff==0.1.3

      - schemasheets==0.2.1

      - scipy==1.13.1

      - semsql==0.3.3

      - shexjsg==0.8.2

      - six==1.16.0

      - snowballstemmer==2.2.0

      - sortedcontainers==2.4.0

      - soupsieve==2.6

      - sparqlslurper==0.5.1

      - sparqlwrapper==2.0.0

      - sphinx==7.4.7

      - sphinx-copybutton==0.5.2

      - sphinx-gallery==0.18.0

      - sphinx-rtd-theme==3.0.1

      - sphinxcontrib-applehelp==2.0.0

      - sphinxcontrib-devhelp==2.0.0

      - sphinxcontrib-htmlhelp==2.1.0

      - sphinxcontrib-jquery==4.1

      - sphinxcontrib-jsmath==1.0.1

      - sphinxcontrib-qthelp==2.0.0

      - sphinxcontrib-serializinghtml==2.0.0

      - sqlalchemy==2.0.36

      - sqlalchemy-utils==0.38.3

      - sssom==0.4.11

      - sssom-schema==0.15.2

      - tabulate==0.9.0

      - tenacity==8.5.0

      - termcolor==2.5.0

      - threadpoolctl==3.2.0

      - tomli==2.0.2

      - tox==4.11.3

      - tqdm==4.66.4

      - types-python-dateutil==2.9.0.20241003

      - typing-extensions==4.12.2

      - tzdata==2024.2

      - uri-template==1.3.0

      - uritemplate==4.1.1

      - url-normalize==1.4.3

      - urllib3==2.2.3

      - validators==0.34.0

      - virtualenv==20.27.0

      - watchdog==5.0.3

      - webcolors==24.8.0

      - wrapt==1.16.0

      - zarr==2.16.1

      - zipp==3.20.2

prefix: /opt/anaconda3/envs/hdmf_zarr_env

I will look into this

[rly]

Hi @pauladkisson I can reproduce the error but I think the issue is ultimately due to an incompatibility between your h5py/hdf5 installation and your numpy installation. The error comes from import h5py. I suspect that in this environment, if you open python and run import h5py you will get the same error. That happens for me. I think this might have happened if you installed h5py/hdf5 in conda with numpy < 2, and then later updated numpy >= 2. I recommend that you run conda install "h5py>3.12" to get the latest h5py version that should work with the latest numpy. After that, your example script runs. Let me know if that works for you.

[pauladkisson]

Yep, that was it. I just made a simple change to the dev install instructions and now I get a compatible version of h5py. -- See #233

[rly]

Great, thanks.