Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Distributed Mesh Class #111

Open
kpwelsh opened this issue Oct 17, 2023 · 0 comments
Open

Distributed Mesh Class #111

kpwelsh opened this issue Oct 17, 2023 · 0 comments
Assignees

Comments

@kpwelsh
Copy link
Collaborator

kpwelsh commented Oct 17, 2023

For any kind of proper scalability, we rely on MPI. As it stands, we have separate code on top of our mesh classes that operate with MPI. We use Zoltan2 for node balancing and TPetra for MPI communications.

We should consolidate this into a single layer on top of a mesh class. The availability of out of the box distributed mesh scaling can be significantly valuable for ELEMENTS users.

The distributed mesh would handle:

  1. Node balancing
  2. Process mapping during read/write
  3. MPI Communications

During this process, we should consider separating from TPetra MultiVectors for MPI comms and implement our own for two reasons:

  1. The layout of MultiVectors is contrary to our own data layout for several arrays. This results in substantial over-communication when we have to transpose them. While only a mild slowdown now, there are future methods that we anticipate to take a much bigger hit.
  2. We would like to take advantage of MPI Direct GPU comms in the future, and TPetra is unlikely to support this.
@kpwelsh kpwelsh self-assigned this Nov 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant