no rendering using interactive openPMD-viewer #1572

BenWibking · 2023-12-14T20:23:24Z

I am attempting to use the default interactive notebook to visualize a series of BP5 outputs. It recognizes the names of the components, but nothing shows up. In the console, it shows:

[I 2023-12-14 15:18:13.652 JupyterNotebookApp] 302 GET /tree/openPMD-visualization.ipynb?token=[secret] (1adbbb045dda4c4e9fc49c3a014e6e57@::1) 4.07ms
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
[W 2023-12-14 15:18:15.366 ServerApp] Notebook openPMD-visualization.ipynb is not trusted
[I 2023-12-14 15:18:15.521 ServerApp] Kernel started: 22f6f31c-47fe-4adb-8800-a3b61c452b54
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
[I 2023-12-14 15:18:15.812 ServerApp] Connecting to kernel 22f6f31c-47fe-4adb-8800-a3b61c452b54.
[AbstractIOHandlerImpl] IO Task AVAILABLE_CHUNKS failed with exception. Clearing IO queue and passing on the exception.

Each time I click "Refresh now!", it prints a new line:

[AbstractIOHandlerImpl] IO Task AVAILABLE_CHUNKS failed with exception. Clearing IO queue and passing on the exception.

The text was updated successfully, but these errors were encountered:

ax3l · 2023-12-16T00:17:55Z

Hi @BenWibking ,

Thanks a lot for the report! I moved it over to openPMD-api, because the backend issue in openPMD-viewer is rooted here.

Can you provide us some deetails how openPMD-api was installed for your Jupyter/Python/openPMD-viewer stack?

Package manager or other method details
Version of openPMD-api and openPMD-viewer

Can you print the ls structure and the exact Python commands you use to read?

ax3l · 2023-12-16T00:20:32Z

@BenWibking, for testing your data set, can you do produce a comparable output with .h5?

The .h5 files, you can run through:

openPMD-validator and
open with openPMD-api and openPMD-viewer.

Does the validator show any errors or warnings?
Does openPMD-api and openPMD-viewer consume the HDF5 (or alternatively, .bp4) files well?

BenWibking · 2023-12-16T03:50:26Z

For this case, I installed openPMD-api and openPMD-viewer with conda as follows:

conda create -n openpmd -c conda-forge openpmd-api
conda install -c conda-forge openpmd-viewer openpmd-api
conda install -c conda-forge openpmd-validator

>>> import openpmd_api
>>> openpmd_api.__version__
'0.15.2'
>>> import openpmd_viewer
>>> openpmd_viewer.__version__
'1.9.0'

Then I used openPMD_notebook to create the notebook, and then changed the path in cell 2.

Conda info:

(openpmd) ➜  tests git:(development) ✗ conda info

     active environment : openpmd
    active env location : /opt/homebrew/Caskroom/miniconda/base/envs/openpmd
            shell level : 2
       user config file : /Users/benwibking/.condarc
 populated config files :
          conda version : 23.7.4
    conda-build version : not installed
         python version : 3.11.6.final.0
       virtual packages : __archspec=1=arm64
                          __osx=14.2=0
                          __unix=0=0
       base environment : /opt/homebrew/Caskroom/miniconda/base  (writable)
      conda av data dir : /opt/homebrew/Caskroom/miniconda/base/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/osx-arm64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-arm64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /opt/homebrew/Caskroom/miniconda/base/pkgs
                          /Users/benwibking/.conda/pkgs
       envs directories : /opt/homebrew/Caskroom/miniconda/base/envs
                          /Users/benwibking/.conda/envs
               platform : osx-arm64
             user-agent : conda/23.7.4 requests/2.29.0 CPython/3.11.6 Darwin/23.2.0 OSX/14.2
                UID:GID : 501:20
             netrc file : None
           offline mode : False

@BenWibking, for testing your data set, can you do produce a comparable output with .h5?

The .h5 files, you can run through:

openPMD-validator and

open with openPMD-api and openPMD-viewer.

Does the validator show any errors or warnings? Does openPMD-api and openPMD-viewer consume the HDF5 (or alternatively, .bp4) files well?

The validator shows:

(openpmd) ➜  tests git:(development) ✗ openPMD_check_h5 -i hdf5/plt00000.h5
Warning: Attribute author (recommended) does NOT exist in `/`!
Found 1 iteration(s)
Iteration 0 : found 6 meshes
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniconda/base/envs/openpmd/bin/openPMD_check_h5", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/openpmd/lib/python3.12/site-packages/openpmd_validator/check_h5.py", line 931, in main
    result_array = check_file(file_name, verbose, force_extension_pic)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/openpmd/lib/python3.12/site-packages/openpmd_validator/check_h5.py", line 924, in check_file
    result_array += check_iterations(f, verbose, extensionStates)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/openpmd/lib/python3.12/site-packages/openpmd_validator/check_h5.py", line 495, in check_iterations
    result_array += check_meshes(f, iteration, v, extensionStates)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/openpmd/lib/python3.12/site-packages/openpmd_validator/check_h5.py", line 603, in check_meshes
    result_array += test_record(f[full_meshes_path], field_name)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/openpmd/lib/python3.12/site-packages/openpmd_validator/check_h5.py", line 159, in test_record
    %(r.name) )
      ^^^^^^
AttributeError: 'str' object has no attribute 'name'

and openPMD-viewer shows the same error as before:

[AbstractIOHandlerImpl] IO Task AVAILABLE_CHUNKS failed with exception. Clearing IO queue and passing on the exception.

Here are the files in both HDF5 and BP5 formats:
quokka_openpmd.tar.gz

BenWibking · 2023-12-17T01:59:39Z

This script shows that the data appears to be there, so I'm not sure what's going on...

import openpmd_api as io
import numpy as np
import matplotlib.pyplot as plt

filename = "bp5/plt00758.bp"
#filename = "hdf5/plt00758.h5"
series = io.Series(filename, io.Access.read_only)
my_iter = series.iterations[list(series.iterations)[0]]
mesh = my_iter.meshes

def readComp(varname):
    comp = mesh[varname][io.Mesh_Record_Component.SCALAR][()]
    series.flush()
    return comp

# read gasDensity
density = readComp("gasDensity")
plt.clf()
plt.figure(figsize=(4,4))
im = plt.imshow(density, origin='lower')
plt.colorbar(im)
plt.tight_layout()
plt.savefig("gasDensity.png", dpi=150)

BenWibking · 2023-12-17T02:12:30Z

If I change line 159 of check_h5.py to print r instead of r.name, I get:

$ openPMD_check_h5 -i hdf5/plt00000.h5 
/home/benwibking/openpmd_files/openPMD-validator/openpmd_validator/check_h5.py:147: SyntaxWarning: invalid escape sequence '\w'
  regEx = re.compile("^\w+$") # Python3 only: re.ASCII
/home/benwibking/openpmd_files/openPMD-validator/openpmd_validator/check_h5.py:407: SyntaxWarning: invalid escape sequence '\.'
  result_array += test_attr(f, v, "required", "openPMD", np.string_, "^[0-9]+\.[0-9]+\.[0-9]+$")
/home/benwibking/openpmd_files/openPMD-validator/openpmd_validator/check_h5.py:409: SyntaxWarning: invalid escape sequence '\/'
  result_array += test_attr(f, v, "required", "basePath", np.string_, "^\/data\/\%T\/$")
/home/benwibking/openpmd_files/openPMD-validator/openpmd_validator/check_h5.py:414: SyntaxWarning: invalid escape sequence '\/'
  result_array += test_attr(f, v, "optional", "meshesPath", np.string_, "^.*\/$")
/home/benwibking/openpmd_files/openPMD-validator/openpmd_validator/check_h5.py:415: SyntaxWarning: invalid escape sequence '\/'
  result_array += test_attr(f, v, "optional", "particlesPath", np.string_, "^.*\/$")
/home/benwibking/openpmd_files/openPMD-validator/openpmd_validator/check_h5.py:431: SyntaxWarning: invalid escape sequence '\+'
  "^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2} [\+|-][0-9]{4}$")
Warning: Attribute author (recommended) does NOT exist in `/`!
Found 1 iteration(s)
Iteration 0 : found 6 meshes
Error: Record x-GasMomentum is NOT named properly (a-Z0-9_)!
Error: Record y-GasMomentum is NOT named properly (a-Z0-9_)!
Error: Record z-GasMomentum is NOT named properly (a-Z0-9_)!
Iteration 0 : found 0 particle species
Result: 3 Errors and 1 Warnings.

BenWibking · 2023-12-17T03:13:08Z

I realized I was also missing axisLabels for all of the coordinate dimensions, but the validator didn't catch that. However, that doesn't fix the problem with openPMD-viewer.

BenWibking · 2023-12-17T04:48:50Z

With fixed iteration encoding, fixed axisLabels, and fixed component names:
quokka_openpmd_fixed.tar.gz

Still nothing appears in openPMD-viewer.

franzpoeschel · 2023-12-18T10:06:15Z

I can reproduce this locally with your dataset. Unfortunately, the openPMD-viewer seems to swallow the original exception, so it is not obvious where the error originally stems from.
I'll have a look.

franzpoeschel · 2023-12-18T10:41:25Z

The problem is that your files for some reason encode the wrong iteration encoding:

> bpls -al bp5/plt00000.bp/ -e '/iterationEncoding'
  string    /iterationEncoding                                 attr   = "groupBased"

As a consequence, the available_chunks() call does not know that it needs to open the file before accessing it. How did you create the file? Was this originally group-based output and you renamed it as file-based?

Aside: In https://github.com/openPMD/openPMD-api/pull/1368/files, we removed two warnings when the encoded iteration encoding did not match the filenames. We should reinstate the first of the two removed warnings. I'll prepare a PR.

franzpoeschel · 2023-12-18T11:20:03Z

With #1573:

> python
Python 3.10.12 (main, Jun  6 2023, 22:43:10) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import openpmd_api as io
>>> s = io.Series("bp5/plt%T.bp", io.Access.read_only)
Series constructor called with iteration regex '%T' suggests loading a time series with fileBased iteration encoding. Loaded file is groupBased. Will ignore the encoding stated in the file and continue treating this as file-based. Depending on what data the opened files actually contain, this might not yield correct results.
Series constructor called with iteration regex '%T' suggests loading a time series with fileBased iteration encoding. Loaded file is groupBased. Will ignore the encoding stated in the file and continue treating this as file-based. Depending on what data the opened files actually contain, this might not yield correct results.
>>> s.iterations[0].meshes["gasDensity"][io.Record_Component.SCALAR].available_chunks()
[<openPMD.WrittenChunkInfo of dimensionality 2'>, <openPMD.WrittenChunkInfo of dimensionality 2'>, <openPMD.WrittenChunkInfo of dimensionality 2'>, <openPMD.WrittenChunkInfo of dimensionality 2'>, <openPMD.WrittenChunkInfo of dimensionality 2'>, <openPMD.WrittenChunkInfo of dimensionality 2'>, <openPMD.WrittenChunkInfo of dimensionality 2'>, <openPMD.WrittenChunkInfo of dimensionality 2'>, <openPMD.WrittenChunkInfo of dimensionality 2'>, <openPMD.WrittenChunkInfo of dimensionality 2'>, <openPMD.WrittenChunkInfo of dimensionality 2'>, <openPMD.WrittenChunkInfo of dimensionality 2'>, <openPMD.WrittenChunkInfo of dimensionality 2'>, <openPMD.WrittenChunkInfo of dimensionality 2'>, <openPMD.WrittenChunkInfo of dimensionality 2'>, <openPMD.WrittenChunkInfo of dimensionality 2'>]

Fixing the iteration encoding in your files should still be the preferred solution. Since ADIOS2 does not really support editing files once written, this is only really possible by copying/recreating the file. openpmd-pipe can help you with that (installed along with openPMD-api): openpmd-pipe --infile bp5/plt%T.bp --outfile fixed_%T.bp.

BenWibking · 2023-12-18T17:08:23Z

With fixed iteration encoding, fixed axisLabels, and fixed component names: quokka_openpmd_fixed.tar.gz

Still nothing appears in openPMD-viewer.

I tried to fix the iteration encoding issue in these files. The exception no longer appears in the console, but I still don't see any rendering of the data in openPMD-viewer. Can you reproduce this?

franzpoeschel · 2023-12-18T18:10:44Z

Ah, yep, should have read more carefully. The initial bug was something that needs to be fixed in the openPMD-api anyway, but the remaining issue seems to be an unrelated issue in the openPMD-viewer.
I'll first ask @ax3l if he has an idea what's going on, I'm not very familiar with how the openPMD-viewer works.

What I see is just the GUI of the Viewer, without any actual output, I assume it's the same on your end?

BenWibking · 2023-12-18T18:47:44Z

What I see is just the GUI of the Viewer, without any actual output, I assume it's the same on your end?

Yeah, that's exactly what I see as well.

franzpoeschel · 2023-12-19T11:50:13Z

Huh
I tried this again, and changing %matplotlib notebook to %matplotlib widget did the trick for me. Wonder what's the meaning of that..

BenWibking · 2023-12-19T16:12:29Z

It works!

Maybe that could be changed to the default? When I loaded it with notebook first, then changed it to widget, it gives an error that at first led me to think that widget wouldn't work at all:

However, if I change it to widget before running the first cell, it works fine and produces the same result you show above.

franzpoeschel · 2023-12-19T17:31:27Z

Maybe that could be changed to the default? When I loaded it with notebook first, then changed it to widget, it gives an error that at first led me to think that widget wouldn't work at all:

Let me be honest with you here: I have no idea what either of these two options do, I was just poking things until they worked.

But these observations might be interesting to @RemiLehe (maintainer of the openPMD-viewer) and @ax3l, as this sounds at least like a UX issue of the openPMD-viewer, if not a bug.

BenWibking · 2023-12-19T17:36:47Z

According to the matplotlib docs, it looks like %matplotlib notebook only works for old versions of notebooks, whereas %matplotlib widget works on all modern versions of either Jupyter Notebook or Jupyterlab: https://matplotlib.org/stable/users/explain/figure/interactive.html#jupyter-notebooks-jupyterlab.

ax3l transferred this issue from openPMD/openPMD-viewer Dec 16, 2023

ax3l changed the title ~~no rendering using interactive viewer: "IO Task AVAILABLE_CHUNKS failed"~~ no rendering using interactive openPMD-viewer: "IO Task AVAILABLE_CHUNKS failed" Dec 16, 2023

ax3l added the backend: ADIOS2 label Dec 16, 2023

ax3l added the question label Dec 16, 2023

ax3l assigned ax3l and franzpoeschel Dec 16, 2023

franzpoeschel mentioned this issue Dec 18, 2023

Update for OPENPMD_VERBOSE logging #1574

Merged

BenWibking mentioned this issue Dec 18, 2023

documentation: mention that ParaView (Windows) and VisIt (all platforms) can't read BP5 files #1576

Open

BenWibking changed the title ~~no rendering using interactive openPMD-viewer: "IO Task AVAILABLE_CHUNKS failed"~~ no rendering using interactive openPMD-viewer Dec 19, 2023

BenWibking mentioned this issue Dec 19, 2023

use %matplotlib widget by default openPMD/openPMD-viewer#407

Merged

RemiLehe closed this as completed in openPMD/openPMD-viewer#407 Dec 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

no rendering using interactive openPMD-viewer #1572

no rendering using interactive openPMD-viewer #1572

BenWibking commented Dec 14, 2023

ax3l commented Dec 16, 2023 •

edited

Loading

ax3l commented Dec 16, 2023

BenWibking commented Dec 16, 2023 •

edited

Loading

BenWibking commented Dec 17, 2023

BenWibking commented Dec 17, 2023

BenWibking commented Dec 17, 2023

BenWibking commented Dec 17, 2023

franzpoeschel commented Dec 18, 2023

franzpoeschel commented Dec 18, 2023

franzpoeschel commented Dec 18, 2023

BenWibking commented Dec 18, 2023

franzpoeschel commented Dec 18, 2023

BenWibking commented Dec 18, 2023

franzpoeschel commented Dec 19, 2023

BenWibking commented Dec 19, 2023

franzpoeschel commented Dec 19, 2023

BenWibking commented Dec 19, 2023

no rendering using interactive openPMD-viewer #1572

no rendering using interactive openPMD-viewer #1572

Comments

BenWibking commented Dec 14, 2023

ax3l commented Dec 16, 2023 • edited Loading

ax3l commented Dec 16, 2023

BenWibking commented Dec 16, 2023 • edited Loading

BenWibking commented Dec 17, 2023

BenWibking commented Dec 17, 2023

BenWibking commented Dec 17, 2023

BenWibking commented Dec 17, 2023

franzpoeschel commented Dec 18, 2023

franzpoeschel commented Dec 18, 2023

franzpoeschel commented Dec 18, 2023

BenWibking commented Dec 18, 2023

franzpoeschel commented Dec 18, 2023

BenWibking commented Dec 18, 2023

franzpoeschel commented Dec 19, 2023

BenWibking commented Dec 19, 2023

franzpoeschel commented Dec 19, 2023

BenWibking commented Dec 19, 2023

ax3l commented Dec 16, 2023 •

edited

Loading

BenWibking commented Dec 16, 2023 •

edited

Loading