Create gridpack::environment with existing MPI communicator #119

wperkins · 2022-12-14T16:32:13Z

This branch adds a facility to create a gridpack::environment with an existing MPI communicator. This allows an external application to call MPI_Initialize() and instantiate multiple GridPACK environments on an externally distribution of MPI ranks.
NOTE: This GridPACK branch depends on a specific branch of Global Arrays (feature/init-comm). This PR should not be accepted until the necessary changes have been merged into the GA master branch (and the GridPACK GA submodule is updated and actually works).

abhyshr · 2023-10-07T03:22:41Z

This branch needs to resolve the conflicts first.
Don't understand what is the additional functionality being added here. Is the ability to set an MPI_Comm with GridPACK's environment object & being able to set it via Python?

wperkins · 2023-10-11T13:35:28Z

2. Don't understand what is the additional functionality being added here. Is the ability to set an MPI_Comm with GridPACK's environment object & being able to set it via Python?

GridPACK expects to control MPI, that is GridPACK calls MPI_Init() and MPI_Finalize() to set up an environment. The changes in this PR simply allows a GridPACK environment to be created with an externally created MPI_Comm. And, not just in Python.

This was set up for the ExaLearn folks, who drive multiple GridPACK instances with an MPI-based driver.

wperkins · 2023-10-11T13:42:24Z

This branch needs to resolve the conflicts first.

Agreed. It should be noted, however, that this will require reconciling the HADREC application with current dynamic simulation API without alienating the ExaLearn application of HADREC. I do not know details.

wperkins · 2023-10-18T18:37:05Z

Note to self: The value of the CMake variable ENABLE_ENVIRONMENT_FROM_COMM needs to be in GridPACK.cmake and the Python interface adjusted to allow ENABLE_ENVIRONMENT_FROM_COMM=FALSE.

wperkins · 2023-11-01T16:20:12Z

src/environment/environment.cpp

 Environment::~Environment(void)
 {
-  // Finalize math libraries
-  gridpack::math::Finalize();
+  if (p_from_comm) {


Note the artifacts from rebase.
I don't understand the destructor. It seems to me that gridpack::math::Finalize(); and GA_Terminate() should be called no matter the state of p_from_comm.

wperkins · 2023-11-01T16:21:02Z

src/applications/dynamic_simulation_full_y/CMakeLists.txt

@@ -198,4 +215,11 @@ install(TARGETS dsf2.x DESTINATION bin)
 # run application as test
 # -------------------------------------------------------------
 gridpack_add_run_test("dynamic_simulation_full_y" dsf.x input_145.xml)
+<<<<<<< HEAD


Rebase artifacts.

bjpalmer · 2023-11-01T16:27:37Z

I'm in the process of cleaning up after the rebase. I'm not sure how I managed to overlook some of the artifacts since the code doesn't compile. I noticed the same thing about the destructor and fixed it.

wperkins · 2023-11-01T16:29:59Z

src/applications/dynamic_simulation_full_y/dsf_comm_main.cpp

+    //gridpack::utility::Configuration::CursorPtr cursor;
+    cursor = config->getCursor("Configuration.Dynamic_simulation");
+    std::vector<gridpack::dynamic_simulation::Event> faults;
+    faults = ds_app.getFaults(cursor);


Also, getFaults() is no longer a method. I think it's getEvents() now. I don't know if there is an internal difference.

abhyshr · 2023-11-17T15:05:49Z

@bjpalmer: any updates on when GA will be updated to have fewture/init-comm in develop?

bjpalmer · 2023-11-17T16:43:05Z

The stuff in feature/init-comm should already be in the develop branch of GA. I've been linking GridPACK against the develop branch and it seems to work okay.

wperkins · 2023-11-17T16:45:26Z

The stuff in feature/init-comm should already be in the develop branch of GA. I've been linking GridPACK against the develop branch and it seems to work okay.

FYI. In the GridPACK feature/init-comm branch, if you specify -D BUILD_GA=YES the GA develop branch will be built.

abhyshr · 2023-11-17T16:51:33Z

So then can we just use the develop branch of GA? And merge GridPack's feature/init-comm to develop? I can update the install script to use GA's develop branch. Thoughts?

abhyshr · 2023-11-17T16:52:38Z

First we need to resolve the issue with GridPACK python application hanging on multiple processors. @bjpalmer: do you have time to take a look at this?

bjpalmer · 2023-11-17T17:01:39Z

You should be able to switch the install script to use develop. Is the code hanging when using the progress ranks runtime in GA? You need to be careful with that runtime to get the code to exit cleanly.

abhyshr · 2023-11-17T17:17:33Z

I am not using progress ranks. You can try the updated 39 bus python example I've pushed to the branch

wperkins · 2023-12-12T16:19:30Z

You should be able to switch the install script to use develop. Is the code hanging when using the progress ranks runtime in GA? You need to be careful with that runtime to get the code to exit cleanly.

You can still use the official GA release to build this branch. But, if you do, you cannot specify -D ENABLE_ENVIRONMENT_FROM_COMM=TRUE. It would be nice to rework FindGA.cmake to test for the necessary GA routine and set that option automatically. That's a lot of work though.

Personally, I think we should merge this. The audience for -D ENABLE_ENVIRONMENT_FROM_COMM=TRUE is small and they are already navigating it.

…hese may confuse builds outside of gridpack.

…nts from some of the code and cleaned up the environment destructor so that it works properly for regular environments.

…ations that are initialized by a communicator and to prevent problems with tests using the GA progress ranks runtime.

… hangs/gets stuck

wperkins · 2023-12-13T19:29:28Z

@bjpalmer, I've added some very brief documentation. If it's enough, please approve this PR and I'll merge it. Thanks.

abhyshr · 2023-12-13T20:09:04Z

Good to merge after rebase.

wperkins self-assigned this Dec 14, 2022

wperkins added the enhancement label Dec 14, 2022

wperkins marked this pull request as draft December 14, 2022 16:33

wperkins requested a review from bjpalmer December 14, 2022 16:34

wperkins mentioned this pull request Feb 23, 2023

Error running Hadrrec application #136

Closed

wperkins added this to the GridPACK 3.5 Release milestone Oct 12, 2023

bjpalmer force-pushed the feature/init-comm branch from c4e4402 to b11cb47 Compare October 31, 2023 16:15

wperkins commented Nov 1, 2023

View reviewed changes

abhyshr force-pushed the feature/init-comm branch from b3348b4 to 117a1ec Compare November 16, 2023 21:20

wperkins mentioned this pull request Nov 17, 2023

building python wrapper for the develop branch #179

Closed

wperkins force-pushed the feature/init-comm branch from 117a1ec to ce60140 Compare November 29, 2023 14:59

wperkins changed the title ~~DRAFT: Create gridpack::environment with existing MPI communicator~~ Create gridpack::environment with existing MPI communicator Nov 29, 2023

wperkins marked this pull request as ready for review November 29, 2023 16:27

wperkins force-pushed the feature/init-comm branch from a64b54d to 424179e Compare December 12, 2023 15:22

wperkins mentioned this pull request Dec 13, 2023

GridPACK Release #168

Open

49 tasks

wperkins and others added 21 commits December 13, 2023 10:57

Add Python interface to new Environment constructor

d7de3b7

Fixed a bug when instantiating the environment with a communicator.

4d80221

Add test hadrec script that uses external communicator

c9a1da1

Change GA url and point to required feature/init-comm branch

133f673

Fix typo so option works correctly

dda5828

Run init-comm environment test if called for

11c243c

Always build with environment with comm

3cbf0c6

GridPACK needs to know GA headers are installed in .../include/ga

291e916

Change install directory for my system

fabaaf3

Make sure mpi4py is available before setup

f07dfdb

No, don't make sure mpi4py is available for setup, do it elsewhere

c14d125

Check Python and mpi4py availability separate from pybind11

2240dad

Switch GA submodule to develop

a772074

Modified environment so that no preprocessor symbols are in header. T…

8d07cd3

…hese may confuse builds outside of gridpack.

Fixed up a bunch of problems after rebase, removed some debugging pri…

dbce579

…nts from some of the code and cleaned up the environment destructor so that it works properly for regular environments.

Only warn about GA when using an external build

495a863

Boost and MPI flags are necessary, don't comment them out

01fb3b7

Tweaked a few things so that environment can exit cleanly from applic…

ce40ff6

…ations that are initialized by a communicator and to prevent problems with tests using the GA progress ranks runtime.

Updated the 39-bus python example to run in parallel. The application…

997f938

… hangs/gets stuck

Set installation directory from environment

d1085fc

Tweak Python installation

4965bb6

wperkins force-pushed the feature/init-comm branch from 8aa889a to 4965bb6 Compare December 13, 2023 18:58

Add some brief documentation for ENABLE_ENVIRONMENT_FROM_COMM

d24b4da

abhyshr self-requested a review December 13, 2023 20:08

bjpalmer approved these changes Dec 13, 2023

View reviewed changes

wperkins merged commit c8cf948 into develop Dec 13, 2023

wperkins deleted the feature/init-comm branch December 13, 2023 20:33

jacksavage mentioned this pull request Jan 20, 2024

Add Rocky build automation via PNNL GitLab #173

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create gridpack::environment with existing MPI communicator #119

Create gridpack::environment with existing MPI communicator #119

wperkins commented Dec 14, 2022

abhyshr commented Oct 7, 2023

wperkins commented Oct 11, 2023

wperkins commented Oct 11, 2023

wperkins commented Oct 18, 2023

wperkins Nov 1, 2023

wperkins Nov 1, 2023

bjpalmer commented Nov 1, 2023

wperkins Nov 1, 2023

bjpalmer Nov 1, 2023

abhyshr commented Nov 17, 2023

bjpalmer commented Nov 17, 2023

wperkins commented Nov 17, 2023

abhyshr commented Nov 17, 2023

abhyshr commented Nov 17, 2023

bjpalmer commented Nov 17, 2023

abhyshr commented Nov 17, 2023

wperkins commented Dec 12, 2023

wperkins commented Dec 13, 2023

abhyshr commented Dec 13, 2023

Create gridpack::environment with existing MPI communicator #119

Create gridpack::environment with existing MPI communicator #119

Conversation

wperkins commented Dec 14, 2022

abhyshr commented Oct 7, 2023

wperkins commented Oct 11, 2023

wperkins commented Oct 11, 2023

wperkins commented Oct 18, 2023

wperkins Nov 1, 2023

Choose a reason for hiding this comment

wperkins Nov 1, 2023

Choose a reason for hiding this comment

bjpalmer commented Nov 1, 2023

wperkins Nov 1, 2023

Choose a reason for hiding this comment

bjpalmer Nov 1, 2023

Choose a reason for hiding this comment

abhyshr commented Nov 17, 2023

bjpalmer commented Nov 17, 2023

wperkins commented Nov 17, 2023

abhyshr commented Nov 17, 2023

abhyshr commented Nov 17, 2023

bjpalmer commented Nov 17, 2023

abhyshr commented Nov 17, 2023

wperkins commented Dec 12, 2023

wperkins commented Dec 13, 2023

abhyshr commented Dec 13, 2023