h5topng still gives invalid colormap error

[junyi-lee]

Per #1015 and #739, I tried updating h5utils on both conda-forge and simpetus but still got the invalid colormap error (meep imports fine though):

$ h5topng -t 0:332 -R -Zc dkbluered -v -a yarg -A eps-000000.00.h5 ez.h5
Using colormap "dkbluered" in file "/home/xxx/miniconda3/envs/meep/share/h5utils/colormaps/".
h5topng error: invalid colormap file

However the files are in the directory:
~/miniconda3/envs/meep/share/h5utils/colormaps$ ls
BrBG autumn bone cool dkbluered gray hot inferno lines prism summer viridis yarg
RdBu bluered colorcube copper flag green hsv jet pink spring vga winter yellow

The conda list is as follows:

and which gives
$ which h5topng
/home/xxx/miniconda3/envs/meep/bin/h5topng

[ChristopherHogan]

I can reproduce this. The h5utils package has been fixed, but it's fixed only in build number 1106 and for some reason that I can't figure out yet, build 1106 is not compatible with the rest of the meep environment so conda installs build number 1105, which is broken. As a workaround, you can either install h5utils with your package manager and run /usr/bin/h5topng ..., or create a conda environment with just h5utils (it will install build number 1106, which work), and switch to it when you need to use h5topng.

[junyi-lee]

Thanks for looking into this. Installing h5utils build number 1106 in another environment works for me too.

[devrra]

I faced same issue. I created another environment for build number 1106 of h5utils and everything looks okay. Thank you

[stevengj]

@ChristopherHogan, is the problem with build 1106 and Conda fixed yet?

[stevengj]

Another workaround is to pass the absolute path of the colormap file to the -c option of h5topng. That is, instead of passing -c bluered, pass -c /PATH/TO/CONDA/share/h5utils/colormaps/bluered. (You can find the Conda path by typing which h5topng, which should print /PATH/TO/CONDA/bin/h5topng.)

[ChristopherHogan]

is the problem with build 1106 and Conda fixed yet?

@oskooi tried it yesterday and said it is not fixed yet. I'll look into it.

[wizzhop0]

Hi, this still seems to be an issue.

Is there a workaround for using output_png in Conda/Notebook that actually works. Passing the absolute path of colormaps to output_png doesn't work.

[wizzhop0]

OK i managed to get it to run with an absolute path. For me, if I set sim.use_output_directory() with no arguments, it creates a folder "-out" which h5topng is interpreting as an input argument and throws it's help message instead of converting.

The solution was to give use_output_directory an argument (folder name) and also give output_png the absolute path to colormaps.

[scigg]

I have the same issue today.

[sbedi1212]

Hi I am trying to convert my h5 data file to png using following command:- "h5topng -RZc dkbluered -C wg_test_3D-eps-000000.00.h5 wg_test_3D-hz-*.h5". But I get the following error:- "h5topng error: invalid colormap file".
I have gone through the previous comments on this issue but since I recently started using meep, I have the updated h5utils build - 1113.

I tried adding the absolute path also but even that doesn't work. "h5topng -c /home/bedi_meep/install/install/envs/mp/bin/h5topng -C wg_test_3D-eps-000000.00.h5 wg_test_3D-hz-*.h5"
and gives me the same error:- "h5topng error: invalid colormap file".

Can anyone please help?
Thanks.

[Andeloth]

Update and request to reopen this issue, I created a new conda environment with only h5utils installed and I still wasn't able to run h5topng. IMO this is a major problem for trying to debug simulations in meep.

[oskooi]

The h5utils Conda package was updated two weeks ago to use an updated HDF5 package. Perhaps the reason you are not able to use the latest version of h5utils is because you have multiple HDF5 libraries installed in your system which is causing a conflict?

[Andeloth]

Possible, this is on a supercomputer so I don't have root access. The only way I've installed hdf5 is as a requirement installation for the meep (mpich) installation using

conda install -c conda-forge pymeep="*=mpi_mpich*"

# Name                    Version                   Build  Channel
hdf5                      1.12.2          mpi_mpich_h08b82f9_1    conda-forge

# Name                    Version                   Build  Channel
h5utils                   1.13.1          mpi_mpich_hdf58ef8_1014    conda-forge

There should be no other hdf5 installations on the instance. I know that there are hdf5 modules that can be loaded on the network but I don't have any modules loaded.

[Andeloth]

On a fresh installation of meep (newest version, parallel meep) the hdf5 version that is installed is still 1.12.1. Installing pymeep-extras doesn't change the version of hdf5 installed either. @oskooi

(pmp) bash-4.2$ conda install -c conda-forge pymeep="*=mpi_mpich*"
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: done

## Package Plan ##

  environment location: /fslhome/andeloth/miniconda3/envs/pmp

  added / updated specs:
    - pymeep[build=mpi_mpich*]


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    hdf5-1.12.2                |mpi_mpich_h5d83325_1         3.3 MB  conda-forge
    libcurl-7.88.1             |       hdc1c0ab_0         350 KB  conda-forge
    mpi-1.0                    |            mpich           4 KB  conda-forge
    python-3.10.0              |h543edf9_3_cpython        29.8 MB  conda-forge
    python_abi-3.10            |          3_cp310           6 KB  conda-forge
    sqlite-3.40.0              |       h4ff8645_0         801 KB  conda-forge
    ------------------------------------------------------------
                                           Total:        34.3 MB

The following NEW packages will be INSTALLED:

  c-ares             conda-forge/linux-64::c-ares-1.18.1-h7f98852_0 
  cached-property    conda-forge/noarch::cached-property-1.5.2-hd8ed1ab_1 
  cached_property    conda-forge/noarch::cached_property-1.5.2-pyha770c72_1 
  fftw               conda-forge/linux-64::fftw-3.3.10-mpi_mpich_h3e343da_6 
  gsl                conda-forge/linux-64::gsl-2.7-he838d99_0 
  h5py               conda-forge/linux-64::h5py-3.8.0-mpi_mpich_py310h9f271e8_0 
  harminv            conda-forge/linux-64::harminv-1.4.2-hc567f7b_0 
  hdf5               conda-forge/linux-64::hdf5-1.12.2-mpi_mpich_h5d83325_1 
  keyutils           conda-forge/linux-64::keyutils-1.6.1-h166bdaf_0 
  krb5               conda-forge/linux-64::krb5-1.20.1-h81ceb04_0 
  libaec             conda-forge/linux-64::libaec-1.0.6-hcb278e6_1 
  libblas            conda-forge/linux-64::libblas-3.9.0-16_linux64_openblas 
  libcblas           conda-forge/linux-64::libcblas-3.9.0-16_linux64_openblas 
  libctl             conda-forge/linux-64::libctl-4.5.1-h7f98852_0 
  libcurl            conda-forge/linux-64::libcurl-7.88.1-hdc1c0ab_0 
  libedit            conda-forge/linux-64::libedit-3.1.20191231-he28a2e2_2 
  libev              conda-forge/linux-64::libev-4.33-h516909a_1 
  libgdsii           conda-forge/linux-64::libgdsii-0.21-h4bd325d_4 
  libgfortran-ng     conda-forge/linux-64::libgfortran-ng-12.2.0-h69a702a_19 
  libgfortran5       conda-forge/linux-64::libgfortran5-12.2.0-h337968e_19 
  liblapack          conda-forge/linux-64::liblapack-3.9.0-16_linux64_openblas 
  libnghttp2         conda-forge/linux-64::libnghttp2-1.51.0-hff17c54_0 
  libnsl             conda-forge/linux-64::libnsl-2.0.0-h7f98852_0 
  libopenblas        conda-forge/linux-64::libopenblas-0.3.21-pthreads_h78a6416_3 
  libsqlite          conda-forge/linux-64::libsqlite-3.40.0-h753d276_0 
  libssh2            conda-forge/linux-64::libssh2-1.10.0-hf14f497_3 
  libzlib            conda-forge/linux-64::libzlib-1.2.13-h166bdaf_4 
  mpb                conda-forge/linux-64::mpb-1.11.1-mpi_mpich_hb322358_4 
  mpi                conda-forge/linux-64::mpi-1.0-mpich 
  mpi4py             conda-forge/linux-64::mpi4py-3.1.4-py310h37cc914_0 
  mpich              conda-forge/linux-64::mpich-4.0.3-h846660c_100 
  numpy              conda-forge/linux-64::numpy-1.24.2-py310h8deb116_0 
  pymeep             conda-forge/linux-64::pymeep-1.25.0-mpi_mpich_py310h1234567_1 
  python_abi         conda-forge/linux-64::python_abi-3.10-3_cp310 

The following packages will be UPDATED:

  bzip2                   pkgs/main::bzip2-1.0.8-h7b6447c_0 --> conda-forge::bzip2-1.0.8-h7f98852_4 
  ld_impl_linux-64   pkgs/main::ld_impl_linux-64-2.38-h118~ --> conda-forge::ld_impl_linux-64-2.40-h41732ed_0 
  libffi                   pkgs/main::libffi-3.3-he6710b0_2 --> conda-forge::libffi-3.4.2-h7f98852_5 
  libgcc-ng          pkgs/main::libgcc-ng-11.2.0-h1234567_1 --> conda-forge::libgcc-ng-12.2.0-h65d4601_19 
  libgomp              pkgs/main::libgomp-11.2.0-h1234567_1 --> conda-forge::libgomp-12.2.0-h65d4601_19 
  libstdcxx-ng       pkgs/main::libstdcxx-ng-11.2.0-h12345~ --> conda-forge::libstdcxx-ng-12.2.0-h46fd767_19 
  libuuid              pkgs/main::libuuid-1.41.5-h5eee18b_0 --> conda-forge::libuuid-2.32.1-h7f98852_1000 
  openssl              pkgs/main::openssl-1.1.1t-h7f8727e_0 --> conda-forge::openssl-3.0.8-h0b41bf4_0 
  pip                pkgs/main/linux-64::pip-22.3.1-py310h~ --> conda-forge/noarch::pip-23.0.1-pyhd8ed1ab_0 
  setuptools         pkgs/main/linux-64::setuptools-65.6.3~ --> conda-forge/noarch::setuptools-67.3.2-pyhd8ed1ab_0 
  zlib                    pkgs/main::zlib-1.2.13-h5eee18b_0 --> conda-forge::zlib-1.2.13-h166bdaf_4 

The following packages will be SUPERSEDED by a higher-priority channel:

  _libgcc_mutex           pkgs/main::_libgcc_mutex-0.1-main --> conda-forge::_libgcc_mutex-0.1-conda_forge 
  _openmp_mutex          pkgs/main::_openmp_mutex-5.1-1_gnu --> conda-forge::_openmp_mutex-4.5-2_gnu 
  ca-certificates    pkgs/main::ca-certificates-2023.01.10~ --> conda-forge::ca-certificates-2022.12.7-ha878542_0 
  certifi            pkgs/main/linux-64::certifi-2022.12.7~ --> conda-forge/noarch::certifi-2022.12.7-pyhd8ed1ab_0 
  ncurses                 pkgs/main::ncurses-6.4-h6a678d5_0 --> conda-forge::ncurses-6.3-h27087fc_1 
  python                pkgs/main::python-3.10.0-h12debd9_5 --> conda-forge::python-3.10.0-h543edf9_3_cpython 
  readline               pkgs/main::readline-8.2-h5eee18b_0 --> conda-forge::readline-8.1.2-h0f457ee_0 
  sqlite                pkgs/main::sqlite-3.40.1-h5082296_0 --> conda-forge::sqlite-3.40.0-h4ff8645_0 
  tk                        pkgs/main::tk-8.6.12-h1ccaba5_0 --> conda-forge::tk-8.6.12-h27826a3_0 
  tzdata                 pkgs/main::tzdata-2022g-h04d1e81_0 --> conda-forge::tzdata-2022g-h191b570_0 
  wheel              pkgs/main/linux-64::wheel-0.38.4-py31~ --> conda-forge/noarch::wheel-0.38.4-pyhd8ed1ab_0 
  xz                        pkgs/main::xz-5.2.10-h5eee18b_1 --> conda-forge::xz-5.2.6-h166bdaf_0 


Proceed ([y]/n)?

[Andeloth]

Okay, what I found matches the original issue, so having a separate h5utils environment works as a temporary workaround. Installing pymeep after having installed h5utils downgrades both h5utils and hdf5 to the versions which I assume have the colormap issue still.

conda install -c conda-forge pymeep="*=mpi_mpich*"
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /fslhome/andeloth/miniconda3/envs/test

  added / updated specs:
    - pymeep[build=mpi_mpich*]


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    h5py-3.8.0                 |mpi_mpich_py311heefd281_0         1.2 MB  conda-forge
    mpi4py-3.1.4               |  py311h7edb0b5_0         629 KB  conda-forge
    pymeep-1.25.0              |mpi_mpich_py311h1234567_1         2.0 MB  conda-forge
    ------------------------------------------------------------
                                           Total:         3.8 MB

The following NEW packages will be INSTALLED:

  cached-property    conda-forge/noarch::cached-property-1.5.2-hd8ed1ab_1 
  cached_property    conda-forge/noarch::cached_property-1.5.2-pyha770c72_1 
  fftw               conda-forge/linux-64::fftw-3.3.10-mpi_mpich_h3e343da_6 
  gsl                conda-forge/linux-64::gsl-2.7-he838d99_0 
  h5py               conda-forge/linux-64::h5py-3.8.0-mpi_mpich_py311heefd281_0 
  harminv            conda-forge/linux-64::harminv-1.4.2-hc567f7b_0 
  libctl             conda-forge/linux-64::libctl-4.5.1-h7f98852_0 
  libgdsii           conda-forge/linux-64::libgdsii-0.21-h4bd325d_4 
  mpb                conda-forge/linux-64::mpb-1.11.1-mpi_mpich_hb322358_4 
  mpi4py             conda-forge/linux-64::mpi4py-3.1.4-py311h7edb0b5_0 
  pymeep             conda-forge/linux-64::pymeep-1.25.0-mpi_mpich_py311h1234567_1 

The following packages will be DOWNGRADED:

  h5utils                    1.13.1-mpi_mpich_hb56ab3d_1015 --> 1.13.1-mpi_mpich_hdf58ef8_1014 
  hdf5                          1.14.0-mpi_mpich_hcccbd3f_2 --> 1.12.2-mpi_mpich_h5d83325_1 


Proceed ([y]/n)?

I would like to suggest not closing this issue until meep works with the updated hdf5/h5utils.

[Andeloth]

Actually, the same issue happens when you use a separate conda environment with only the h5utils installed using conda.

[stevengj]

You need h5utils 1.13.2 to fix this. Probably the Conda packages have not been updated yet? See NanoComp/h5utils#16 for an explanation and a workaround

[oskooi]

I just updated the h5utils Conda package to 1.13.2 which should be available momentarily: https://anaconda.org/conda-forge/h5utils.

I will also update the PyMeep Conda package with this updated dependency.

[Andeloth]

Thanks Ardavan! Much appreciated :)