user-specified cell partition with boundary layers

[oskooi]

The user-specified cell partition feature added in #1528 currently requires that the boundary-layer regions (i.e., PMLs) be defined as separate chunks of the cell partition. This is inconsistent with the existing num_chunks API where the user specifies N chunks or MPI processes at runtime and then Meep forms additional chunks for the PML regions when creating the cell volume and assigns these new chunks the same process ID as the original chunk from which they were formed.

As a demonstration, suppose we want to divide a 10x5 cell into two equal-sized chunks by splitting at x=0 and also surround the cell on all sides with PML of thickness 1.0. This would involve specifying a BinaryPartition class object as the chunk_layout parameter of the Simulation constructor:

import meep as mp
import matplotlib.pyplot as plt

chunk_layout = mp.BinaryPartition(data=[ (mp.X,0.0), 0, 1 ])

cell_size = mp.Vector3(10.0,5.0,0)

pml_layers = [mp.PML(thickness=1)]

sim = mp.Simulation(cell_size=cell_size,
                    resolution=10,
                    boundary_layers=pml_layers,
                    chunk_layout=chunk_layout)

sim.visualize_chunks()
plt.savefig('chunk_layout.png',dpi=150,bbox_inches='tight')

Running this script aborts with the following error:

Using MPI version 3.1, 2 processes
-----------
Initializing structure...
Splitting into 2 chunks by voxels
meep: load_chunk_layout: wrong number of chunks.
meep: load_chunk_layout: wrong number of chunks.
time for choose_chunkdivision = 0.000227449 s
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1.

The problem here is that even though we have specified only two chunks, there are in fact 12 chunks in total once the PML regions are included. A schematic of the actual chunk layout is shown below. Meep is expecting to be supplied 12 chunks rather than 2 and thus quits with an error.

In this schematic, chunks 1, 3, 6, and 8 are 2d PMLs (i.e., PML conductivity in the x and y directions) whereas chunks 2, 4, 5, 7, 9, 10 are 1d PMLs. The original user-defined chunks are 11 and 12 which have been reduced in size by their intersecting PML regions that have been separated out.

It would be useful for the user-specified chunk_layout feature to replicate the functionality of the num_chunks API so that the user does not have to define a cell partition that also includes the PML regions (which is a little cumbersome).

[stevengj]

See #1528 (comment) … that is, we should always specify the chunk layout as a binary tree, whether it is passed by the user or constructed by choose_chunkdivision. Given that binary tree, we can then split off the PML regions etcetera as usual during structure construction.