IndexMap::create_submap reshuffles ghost nodes: is this expected?

[francesco-ballarin]

Consider the following example in which I create a submap of a IndexMap which actually doesn't throw away any index.

import dolfinx.fem
import dolfinx.mesh
import mpi4py.MPI
import numpy as np

mesh = dolfinx.mesh.create_unit_square(mpi4py.MPI.COMM_WORLD, 4, 4)
V = dolfinx.fem.functionspace(mesh, ("Lagrange", "1"))
entities = np.arange(0, V.dofmap.index_map.size_local)
index_map_sub, ghost_map_sub = V.dofmap.index_map.create_submap(entities)
print(mpi4py.MPI.COMM_WORLD.rank, "Ghost map", ghost_map_sub)
print(mpi4py.MPI.COMM_WORLD.rank, "Original map ghosts", V.dofmap.index_map.ghosts)
print(mpi4py.MPI.COMM_WORLD.rank, "Sub map ghosts", index_map_sub.ghosts)

If I run it with 2 processes on my laptop I get

0 Ghost map [0 1 2 3 4 5 6]
0 Original map ghosts [19 21 22 24 17 18 23]
0 Sub map ghosts [19 21 22 24 17 18 23]
1 Ghost map [0 1 2 3]
1 Original map ghosts [7 0 2 5]
1 Sub map ghosts [7 0 2 5]

that is the ghosts in the original and new maps are ordered the same way.

If I run it with 3 processes on my laptop I get

0 Ghost map [ 0  1  2  4  7  9  3  5  6  8 10]
0 Original map ghosts [12 10  9 16  8 19 23 11 17  6 21]
0 Sub map ghosts [12 10  9  8 11  6 16 19 23 17 21]
1 Ghost map [0 1 3 9 2 4 5 6 7 8]
1 Original map ghosts [ 0  1 24  2 16 14 15 19 23  5]
1 Sub map ghosts [ 0  1  2  5 24 16 14 15 19 23]
2 Ghost map [0 1 6 8 2 3 4 5 7]
2 Original map ghosts [ 3  1  9 11  8 10  2  6  5]
2 Sub map ghosts [ 3  1  2  5  9 11  8 10  6]

i.e., some of the ghosts have been swapped.

Is this expected? I imagined that the submap in this case was supposed to be the identity.

If you run this on dolfinx/dolfinx:nigthly you may get slightly different outputs, probably because partitioning is different, but the overall question is still the same.

[jpdean]

In general, the relationship between the ghosts in the submap to the ghosts in the original map is given by the returned ghost map. It's probably better if this was the identity when all indices are retained, but as far as I'm aware, I don't think the current behaviour is particularly problematic. I'm happy to change it though if needed.

[francesco-ballarin]

Apart from the extreme case in my test, I would say that ghosts in the output being ordered in the same way as in the input seems reasonable to me considering that the order of the owned dofs gets preserved (right? because they were sorted when they were provided as input). One would of course still need the second output argument, in all the relevant cases in which at least one index is dropped.

[IgorBaratta]

The order of the new ghost nodes depends on many factors (there's some communication going on - from ghost to owners and back to ghosts). This ordering can be preserved to match the parent map, though it's unclear to me if the benefits outweigh the complexity.

However, I'm working on a branch to sort all ghosts which seems promising in simplifying computations and communication patterns, but it's still in the development phase and needs further work.

For your example I get:

$ mpirun -n 3 python3 test.py 

0 Ghost map [0 1 2 3 4 5 6]
0 Original map ghosts [10 13 14 15 16 18 20]
0 Sub map ghosts [10 13 14 15 16 18 20]

1 Ghost map [0 1 2 3 4 5 6 7]
1 Original map ghosts [ 4  5  6 18 20 22 23 24]
1 Sub map ghosts [ 4  5  6 18 20 22 23 24] 

2 Ghost map [0 1 2 3 4]
2 Original map ghosts [ 0  4  8 10 11]
2 Sub map ghosts [ 0  4  8 10 11]

[francesco-ballarin]

For your example I get:

With main, or with your branch?

[IgorBaratta]

With this branch: https://github.com/FEniCS/dolfinx/compare/igor/reorder

[jorgensd]

The order of the new ghost nodes depends on many factors (there's some communication going on - from ghost to owners and back to ghosts). This ordering can be preserved to match the parent map, though it's unclear to me if the benefits outweigh the complexity.

However, I'm working on a branch to sort all ghosts which seems promising in simplifying computations and communication patterns, but it's still in the development phase and needs further work.

For your example I get:

$ mpirun -n 3 python3 test.py 

0 Ghost map [0 1 2 3 4 5 6]
0 Original map ghosts [10 13 14 15 16 18 20]
0 Sub map ghosts [10 13 14 15 16 18 20]

1 Ghost map [0 1 2 3 4 5 6 7]
1 Original map ghosts [ 4  5  6 18 20 22 23 24]
1 Sub map ghosts [ 4  5  6 18 20 22 23 24] 

2 Ghost map [0 1 2 3 4]
2 Original map ghosts [ 0  4  8 10 11]
2 Sub map ghosts [ 0  4  8 10 11]

I agree with Igor that there isn't any clear benefit of adding a complex sorting pattern, that only would work the cases where a contiguous chunk of ghosts are retained (i.e. ghost0, .... ghostM) where there were N original ghosts, M<=M.