Enable Adaptive Mesh Refinement

[nuclearkevin]

This PR enables adaptive mesh refinement for the OpenMC mesh mirror. This allows for the use of AMR (as implemented in MOOSE) within Cardinal.

The main changes include looping over active elements instead of all elements when generating the cell -> element map and creating a copy of the mesh containing only active elements when using a MeshTally. The latter is required for two reasons: to avoid errors which are thrown by MOOSE because of the equation system created in the openmc::LibMesh class, and to prevent erroneous tallying on non-active elements / tally bin scrambling in the refinement/coarsening hierarchy. Creating a copy of the mesh without the refinement hierarchy is a severe limitation of the current approach and should be remedied in the future.

As an added bonus, the changes made in this PR also made it trivial to add block restriction for MeshTally objects. When blocks is set by the user, a submesh is created which only contains the requested tally blocks which is used by OpenMC for tallying. This is useful for cases where multiphysics feedback is required for the full mesh mirror, but the user only wishes to tally on a subset of the OpenMC geometry. Previously MeshTally objects would use the full mesh mirror, resulting in many (potentially unnecessary) tally bins which degrades performance in active batches.

[moosebuild]

Job Documentation, step Sync to remote on 9d34242 wanted to post the following:

View the site here

This comment will be updated on new commits.

[nuclearkevin]

For some reason the AMR MeshTally test crashes in CIVET, but doesn't crash on Pinchot or my local machine. I've ran the test through gdb and Valgrind (thorough memtest) and it executes successfully, so I'm at a bit of a loss at the moment.

Since I use the MOOSE Conda environment on both my local machine and Pinchot (but CIVET builds libMesh from a specific commit hash), I'm starting to wonder if there is an incongruence between the version of libMesh in the Conda distribution and the one required for Cardinal's MOOSE submodule. I'm building from source to investigate that at the moment, hopefully it yields a simple fix so I can get the tests passing.

This wasn't the cause. Valgrind and GDB both report no errors when testing this branch compiled without the MOOSE Conda environment.

[moosebuild]

Job Coverage, step Generate coverage on 9d34242 wanted to post the following:

Coverage

	3b672d	#964 9d3424
	Total	Total	+/-	New
Rate	92.96%	92.92%	-0.04%	97.33%
Hits	7717	7748	+31	73
Misses	584	590	+6	2

Diff coverage report

Full coverage report

This comment will be updated on new commits.

[nuclearkevin]

@aprilnovak I managed to track down and patch the bug that's been plaguing this PR; it should be good for review now.

[aprilnovak]

Can you explain where this went?

[nuclearkevin]

This #else statement was the cause of the segmentation faults on CIVET, and more generally it resulted in memory leaks when Cardinal is built without DAGMC and using a moving mesh.

When the mesh is changing OpenMCCellAverageProblem needs to reset tallies on every timestep (after the first step), and so it calls resetTallies() at the beginning of the step. This is immediately followed by a call to setupProblem(). Afterwards, if the mesh is moving setupProblem() is called a second time in that #else statement resulting in an initialization of OpenMC tallies twice for each time they're reset. This causes a memory leak as we don't manage the extra OpenMC tally created on each step.

The reason why this caused a segmentation fault is a little more complicated and involves the copy of the mesh created in MeshTally containing only active elements. This mesh copy is deleted both when resetTally() and initializeTally() are called (it's a unique pointer). Since we've been creating two OpenMC tallies per timestep (by accident) the first of those tallies still thinks the mesh copy is valid, and so it tries to tally on datastructures that have had their memory freed. This behaviour is undefined and results in a segmentation fault.

This bug only occurs when Cardinal is compiled without DAGMC as resetTallies() is called again when skinning the geometry (the previous usecase for the #ifdef/#else statement), so nothing breaks.

[aprilnovak]

Would you please also add a news markdown file to announce this capability? I think it warrants it!

[nuclearkevin]

@aprilnovak I can definitely do that! Should I also add an entry for the filter system (since it's a pretty big addition to Cardinal as well)?

[moosebuild]

Job Precheck, step Clang format on 9683944 wanted to post the following:

Your code requires style changes.

A patch was auto generated and copied here
You can directly apply the patch by running, in the top level of your repository:

curl -s https://mooseframework.inl.gov/cardinal/docs/PRs/964/clang_format/style.patch | git apply -v

Alternatively, with your repository up to date and in the top level of your repository:

git clang-format b1f70500914c135b0ba116ba13b650858ed9acc8

[nuclearkevin]

@aprilnovak dropping the adaptivity tests to 1 pebble decreases the size of the adaptivity Exodiff tests to ~0.5 M in total. The block restriction test is now ~0.1 M.