Plotting and animation

[smartalecH]

First stab at creating a python interface for visualizing the simulation domain and generating animations. The goal is to create a simple python function that doesn't require any writing to disk or extra command line steps.

I've added two major functions to the simulation class: visualize_domain and animate_fields.

visualize_domain

This function is used to produce a pyplot of the simulation domain. It visualizes the geometry, simulation bounds, boundary layers (green), all of the sources (red) and all of the monitors (blue).

Several helper functions are used to pull the metadata from sources, monitors, PML layers, etc. I'm sure there are much more efficient methods to implement each of these, however.

The user can pass a numpy array of field data and the function will automatically superimpose the fields over the simulation domain.

Note, I've made several assumptions with each of these helper functions:
1.) The simulation domain lies along the XY plane (i.e. Z is coming out of the board)
2.) Everything is centered at Z=0 (in other words, if monitors/sources are above this plane, they won't be plotted).

Here are some examples grabbed from various tutorials:

holey-wvg-cavity.py

metal-cavity-ldos.py

metasurface_lens.py

coupler.py

wvg-src.py

animate_fields

animate_fields() uses the visualize_domain() function to create an animation video of the fields. matplotlib relies on the ffmpeg library to generate the mp4 videos. I'm sure there are more efficient ways to do this.

temp.mp4.zip

[HomerReid]

Nice job! I agree that a standard built-in module for visualizing geometries and fields would be a big advance, and it looks like you're off to a solid start in the direction of proposing and implementing one.

For your reference, I proposed and implemented one too, and in fact the code for mine has been in the master source repository since #795---but this shouldn't dissuade you from pursuing yours, as perhaps it will align better with the plans of the meep development team. Documentation and sample images for mine are here:

https://homerreid.github.io/meep-adjoint-documentation/AdjointSolver/Visualization/

and there are other sample images scattered throughout the adjoint-solver documentation. I'm traveling today and have to dash off right now, but I'll try to expand on this comment later tonight or tomorrow.

[smartalecH]

Thanks for your comments, @HomerReid! I had no idea you already developed a visualization interface. I'll go through your code and incorporate your ideas.

[stevengj]

Would be good to move most of this stuff out of the simulation.py file, which is getting a bit cluttered. e.g. using mixins

[ChristopherHogan]

I thought #865 should have made this unnecessary. Are you seeing buffered output somewhere?

[smartalecH]

Whoops, that slipped through. I'll fix that in the next commit. Thanks!

[smartalecH]

In an effort to make the visualization library as modular as possible, I separated it from the simulation class. My hope is that @HomerReid could use it for his adjoint visualizer and that we could easily integrate pympb as well.

The main 2D visualizer is now a function called plot2D. There is a convenience wrapper inside of simulation (also called plot2D). It can take arbitrary slices in the x, y, and z axes.

There is a run function (Animate2D) which can demonstrate live updates of the fields, or store an animation to be viewed after the fact. The animations can be manipulated inline a Jupyter notebook.

The easiest way to visualize the new features is with some notebooks I updated. You can view them in your browser here:

https://nbviewer.jupyter.org/github/smartalecH/meep/tree/animation/python/examples/

Specifically, I modified straight-waveguide.ipynb, bent-waveguide.ipynb, bend-flux.ipynb, and ring.ipynb. visualizer.ipynb demonstrates some of the slicing features, but unfortunately is not interactive using nbviewer. I also added a README to the examples folder that describes the tutorial notebooks. I'll add more of the tutorials to the README as I retrofit them.

[stevengj]

A few things:

1. At least mayavi, maybe matplotlib too, should be imported lazily (e.g. in the function where it is called). https://stackoverflow.com/questions/4177735/best-practice-for-lazy-loading-python-modules

2. make the animation class work with the other step functions like at_every and in_volume

3. would be nicer to save the animation frames rather than the field data, which is usually orders of magnitude larger. might require calling a lower-level interface in the matplotlib.animation module.

4. For the animation, would be good to take an optional user function(ax) as a parameter so that the user can plot additional things, add labels, or whatever.

5. Don't plot Ez by default — either plot nothing by default or give an error if there is nothing specified to plot and no user function.

6. At some point it would be nice to have more text labels in the plot, but this can be done later.

[stevengj]

By the way, if we ever want to give the opportunity get/plot the field on an arbitrary mesh, one option would be to call fields::get_field with the new parallel=false option (#790) on every mesh point, and then call sum_to_all on the array of field values at the end. This allows you to avoid doing an all-to-all communication for every mesh point.

[smartalecH]

This latest update includes the following new features:

1. Allows for inclusion in any arbitrary run function, including in_volume, at_every, etc. I've updated the above tutorials with this convention.
2. Rather than the user specifying 2D slices using the x=None y=None z=0 convention, the user now passes a 2D volume, just like the in_volume function requires. It can be any arbitrary volume in 3D space, so long as it is 2D (a plane) and not oblique (i.e. a cross product with one of the cartesian basis vectors).
3. I've dug into the low-level matplotlib animation routines and created a few functions so that the Animate2D run function now saves png images to memory (not disk) at each step. The user can then run to_mp4(), to_gif(), and to_jshtml to return mp4 videos, gifs, and interactive javascript animations respectively. The mp4 and gif functionality requires ffmpeg (just like matplotlib). As always, the user can opt to normalize the fields. In this case, the field slices are stored in memory at each timestep. This new implementation minimizes memory consumption and is much much faster.
4. lazy imports
5. require the user to specify a field component (rather than defaulting to Ez as before).

I still plan on integrating a few more features:

1. incorporate Homer's plot customization routines and labeling conventions, along with the ability to pass functions that modify the axis handles.
2. revamp 3D plotting with mayavi
3. implement a test to verify that the various corner cases don't break (would require adding ffmpeg to travis).
4. create a tutorial that explores all the 2D and 3D plotting features
5. ability to plot real, imag, abs, or angle, of the field components (defaults to real right now). This is only really useful for simulations with complex fields.
6. pympb integration
7. docs

[stevengj]

Would be nice to add a _repr_html method to the animation object so that it automatically displays as a video in IPython.

Tests would be good as well.

[smartalecH]

Updated with the following changes:

• Added _repr_html method to animation object
• Added ability to pass arbitrary functions to modify axis in animator
• Added customization features
• Added tutorial (visualizer.ipynb) that demonstrates just about every new feature (including customization.
• Added test (visualization.py). The test will fail until matplotlib, mayavi, and ffmpeg are added to Travis (recently added).
• Added docs

Once this passes, I think this is ready for merging.

One important note: The Jupyter notebooks that contain animations are rather large, for obvious reasons. Do you use Git Large File Storage for large files?

Future work (perhaps a separate PR) could include:

• MPB integration
• Enhanced 3D plotting that includes sources, monitors, PML layers, etc.
• Plotting components of complex fields

[smartalecH]

One of the builds fails (something to do with an MPB callback), while a few others simply time out while testing user_defined_material.py.

I'm not really sure why — any ideas?

[ChristopherHogan]

The jobs timing out are all MPI builds. Have you tried running python/tests/visualization.py locally with MPI? I'm guessing there is a deadlock somewhere. You probably only want to create the outputs (test.mp4, test.gif) on the master process.

The py 2.7 build fails when building without MPB because of #891. I'll push a fix shortly.

[ChristopherHogan]

#894 Fixes the mpb_printf_callback issue.

[smartalecH]

Nice catch! There are indeed deadlocks. Unfortunately, it's not just from writing the animations to disk. Even something simple, like calling plot2D() deadlocks. It doesn't seem to like to run sim.init_sim(). Specifically, it deadlocks when adding the sources (add_vol_src).

This is weird, since init_sim() obviously works outside of plot2D(). I'm wondering if something weird is going on when I pass the simulation object to plot2D. I'm going to keep looking.

[ChristopherHogan]

All processes must call init_sim. Moving this check below line 494 may fix the deadlock.

[smartalecH]

I switched it, but still see deadlocks. I'm trying to use your visualize_chunks() function as a template now. For whatever reason, that function does not deadlock, even though it is pretty similar.

[smartalecH]

Although after carefully debugging, it did get rid of some deadlocks...

[ChristopherHogan]

get_array must be called on all procs.

[smartalecH]

That's the issue. I'll go through and fix all of those. Thanks so much!

[smartalecH]

Is that requirement documented anywhere? We may want to add a page about using mp.am_master() with matplotlib and when am_master() shouldn't be used (i.e. with get_array()

[smartalecH]

It looks like all of the other tests are passing now. Once #894 is merged, that last test should pass too.

[ChristopherHogan]

Is visualization.ipynb supposed to be part of this PR? I don't see it anywhere.

[ChristopherHogan]

When I run test_animation_output, I don't get a test.mp4 file, and the test.gif file has a size of zero bytes.

For to_mp4, the err variable on line 826 contains

ffmpeg version 4.0 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 7.2.0 (crosstool-NG fa8859cb)
  configuration: --prefix=/home/chris/miniconda3/envs/mp3 --cc=/opt/conda/conda-bld/ffmpeg_1531088893642/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc --disable-doc --enable-shared --enable-static --enable-zlib --enable-pic --enable-gpl --enable-version3 --disable-nonfree --enable-hardcoded-tables --enable-avresample --enable-libfreetype --disable-openssl --disable-gnutls --enable-libvpx --enable-pthreads --enable-libopus --enable-postproc --disable-libx264
  libavutil      56. 14.100 / 56. 14.100
  libavcodec     58. 18.100 / 58. 18.100
  libavformat    58. 12.100 / 58. 12.100
  libavdevice    58.  3.100 / 58.  3.100
  libavfilter     7. 16.100 /  7. 16.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  1.100 /  5.  1.100
  libswresample   3.  1.100 /  3.  1.100
  libpostproc    55.  1.100 / 55.  1.100
[image2pipe @ 0x56370ab88680] Could not find codec parameters for stream 0 (Video: png, none(pc), 640x480): unspecified pixel format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
Input #0, image2pipe, from 'pipe:':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: png, none(pc), 640x480, 10 tbr, 10 tbn, 10 tbc
Unknown encoder 'h264'

The err for to_gif is

ffmpeg version 4.0 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 7.2.0 (crosstool-NG fa8859cb)
  configuration: --prefix=/home/chris/miniconda3/envs/mp3 --cc=/opt/conda/conda-bld/ffmpeg_1531088893642/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc --disable-doc --enable-shared --enable-static --enable-zlib --enable-pic --enable-gpl --enable-version3 --disable-nonfree --enable-hardcoded-tables --enable-avresample --enable-libfreetype --disable-openssl --disable-gnutls --enable-libvpx --enable-pthreads --enable-libopus --enable-postproc --disable-libx264
  libavutil      56. 14.100 / 56. 14.100
  libavcodec     58. 18.100 / 58. 18.100
  libavformat    58. 12.100 / 58. 12.100
  libavdevice    58.  3.100 / 58.  3.100
  libavfilter     7. 16.100 /  7. 16.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  1.100 /  5.  1.100
  libswresample   3.  1.100 /  3.  1.100
  libpostproc    55.  1.100 / 55.  1.100
[image2pipe @ 0x56115a6c1640] Could not find codec parameters for stream 0 (Video: png, none(pc), 640x480): unspecified pixel format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
Input #0, image2pipe, from 'pipe:':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: png, none(pc), 640x480, 10 tbr, 10 tbn, 10 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (png (native) -> gif (native))
Cannot determine format of input stream 0:0 after EOF
Error marking filters as finished
Conversion failed!

It would be nice to print this error info rather than having to inspect values in the debugger. I installed ffmpeg from conda-forge. How did you install ffmpeg? Maybe we need to require a specific version?