merge_subgroup_data

[smartalecH]

Closes #1178

Adds a merge_subgroup_data method. Takes a numpy array (int, float, complex, etc) and returns a new array with an extra dimension corresponding to the concatenated input from each subgroup.

An example script test.py:

import meep as mp
import numpy as np

num_groups = 3
a = np.zeros((5,7),dtype=np.complex128)

n = mp.divide_parallel_processes(num_groups)
a[:] = (1+n) + 1j*(1+n)

b = mp.merge_subgroup_data(a,num_groups)
print(b.shape)
print(b)

Execute using:

mpirun -n 6 python3 test.py

Produces the expected output:

(5, 7, 3)
[[[1.+1.j 2.+2.j 3.+3.j]
  [1.+1.j 2.+2.j 3.+3.j]
  [1.+1.j 2.+2.j 3.+3.j]
  [1.+1.j 2.+2.j 3.+3.j]
  [1.+1.j 2.+2.j 3.+3.j]
  [1.+1.j 2.+2.j 3.+3.j]
  [1.+1.j 2.+2.j 3.+3.j]]

 [[1.+1.j 2.+2.j 3.+3.j]
  [1.+1.j 2.+2.j 3.+3.j]
  [1.+1.j 2.+2.j 3.+3.j]
  [1.+1.j 2.+2.j 3.+3.j]
  [1.+1.j 2.+2.j 3.+3.j]
  [1.+1.j 2.+2.j 3.+3.j]
  [1.+1.j 2.+2.j 3.+3.j]]

 [[1.+1.j 2.+2.j 3.+3.j]
  [1.+1.j 2.+2.j 3.+3.j]
  [1.+1.j 2.+2.j 3.+3.j]
  [1.+1.j 2.+2.j 3.+3.j]
  [1.+1.j 2.+2.j 3.+3.j]
  [1.+1.j 2.+2.j 3.+3.j]
  [1.+1.j 2.+2.j 3.+3.j]]

 [[1.+1.j 2.+2.j 3.+3.j]
  [1.+1.j 2.+2.j 3.+3.j]
  [1.+1.j 2.+2.j 3.+3.j]
  [1.+1.j 2.+2.j 3.+3.j]
  [1.+1.j 2.+2.j 3.+3.j]
  [1.+1.j 2.+2.j 3.+3.j]
  [1.+1.j 2.+2.j 3.+3.j]]

 [[1.+1.j 2.+2.j 3.+3.j]
  [1.+1.j 2.+2.j 3.+3.j]
  [1.+1.j 2.+2.j 3.+3.j]
  [1.+1.j 2.+2.j 3.+3.j]
  [1.+1.j 2.+2.j 3.+3.j]
  [1.+1.j 2.+2.j 3.+3.j]
  [1.+1.j 2.+2.j 3.+3.j]]]

[smartalecH]

New test script:

import meep as mp
import numpy as np

num_groups = 3
a = np.zeros((5,7),dtype=np.complex128)

n = mp.divide_parallel_processes(num_groups)
a[:] = (1+n) + 1j*(1+n)

b = mp.merge_subgroup_data(a)
print(b.shape)
print(b)

Produces the same output as above, but now more efficiently (no more overwriting redundant data).

I also added helper functions that return the number of groups (get_num_groups()) and an array of group master ranks (get_group_masters()).

[stevengj]

You can put a test in the documentation PR.

[zlin-opt]

@stevengj @smartalecH
If the num_groups does not divide the total number of processes, what happens?
do the extra processors get assigned to the first group?
for example, num_groups=3 and total procs is 13, then the first group has 5 procs, the second 4 and the third 4?

[oskooi]

Yes, those three groupings are correct with the ranks of each process divided into (0-4), (5-8), and (9-12).

The relevant line in the function divide_parallel_processes which assigns the group number to each process (based on its rank) is:

meep/src/mympi.cpp

Line 706 in 14fe4e0

int mygroup = (my_rank() * numgroups) / count_processors();