Add and calculate coordinate container for off-rank neighbor nodes.

[RyanWollaeger]

Background

• The off-rank Dual_Ghost_Layout container can now see node indices local to the off-rank meshes that share a face with the MPI rank boundary nodes (effectively one node-layer into the off-rank mesh).
• However, to expedite computation over node-based stencils, the coordinate values of the nodes can be obtained with a similar communication pattern used to get the node indices on the off-rank mesh.

Purpose of Pull Request

• Similar to how the off-rank node indices were obtained in PR Add neighboring node indices to dual layouts in Draco_Mesh #1078, add a allgatherv communication for the coordinates of the off-rank nodes sharing a face with the rank-boundary nodes.
  • This amounts to 4 doubles for each "corner element" (a cell adjacent to the current node and the other nodes of the cell's two faces adjacent to the current node).
• Fixes Redmine Issue #2457

Description of changes

• Create node-coordinate map that is 1-to-1 with map to node indexes in dual ghost layout.
• Add specializations on double of allgatherv and determinate_allgatherv.
• Use double determinate_allgatherv to get ghost neighbor node coordinates.
• Use consistent loop variable names in loops serializing data for allgatherv.
• Add tests of ghost coordinate values to tstDraco_Mesh_DD unit test.
• Add mesh decomposition ASCII art schematics to tstDraco_Mesh_DD tests.

Note: this performs an additional allgatherv of four doubles between ranks
(two doubles for each coordinate adjacent to the node and its neighbor cell).

Status

• Reference:
  • Pre-Merge Code Review
  • CDash Status
• Checks
  • Travis/Appveyor CI checks pass
  • Code coverage does not decrease
  • Valgrind test passes
  • Toss3 checks pass
  • Trinitite checks pass
  • Code reviewed/approved, sufficient DbC checks, testing, documentation

[RyanWollaeger]

Forgot about the double == double error; will replace these with soft_equiv.

[clevelam]

@brryan This is another good PR to track. Ryan is ensuring we have access to the full dual mesh which will be required for DDMC on unstructured mesh.

[codecov]

Codecov Report

Merging #1081 (c17e7c4) into develop (d376f66) will increase coverage by 0.0%.
The diff coverage is 100.0%.

@@           Coverage Diff           @@
##           develop   #1081   +/-   ##
=======================================
  Coverage     88.6%   88.7%           
=======================================
  Files          374     374           
  Lines        18579   18601   +22     
=======================================
+ Hits         16479   16505   +26     
+ Misses        2100    2096    -4     

[KineticTheory]

The Appveyor CI failure isn't related to this PR. Are you able to restart it? If not, I'll build/test this PR in my environment to confirm that it is clean. Let me know...

[KineticTheory]

LGTM. Just a couple of comments.

[RyanWollaeger]

@KineticTheory I can't seem to restart it from GitHub (I might be missing something however).
Tried creating an Appveyor account, but this did not seem to permit me to restart it (could not see how...).

[KineticTheory]

@KineticTheory I can't seem to restart it from GitHub (I might be missing something however).
Tried creating an Appveyor account, but this did not seem to permit me to restart it (could not see how...).

I was able to restart it. I forgot that I needed to log into the Appveyor system. Let's see what it reports.

It still fails with not an archive error. But I think I know the cause. I'll need to make some changes to re-activate Appveyor CI. For this PR, I'll test locally.

[KineticTheory]

@RyanWollaeger BTW - the MSVC builds/tests are clean.

I'll put in a PR to fix the broken Appveyor CI.

[KineticTheory]

@KineticTheory I can't seem to restart it from GitHub (I might be missing something however).
Tried creating an Appveyor account, but this did not seem to permit me to restart it (could not see how...).

I was able to restart it. I forgot that I needed to log into the Appveyor system. Let's see what it reports.

It still fails with not an archive error. But I think I know the cause. I'll need to make some changes to re-activate Appveyor CI. For this PR, I'll test locally.

I was able to fix the Appveyor issue. It seems to be running now.