Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MPI distributed parallelism #590

Merged
merged 100 commits into from
Mar 10, 2021
Merged

MPI distributed parallelism #590

merged 100 commits into from
Mar 10, 2021

Commits on Mar 4, 2021

  1. Sandbox implementation of fill_halo_regions_tiled for MPI 

    Former-commit-id: f9286e6
    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    4c22fcf View commit details
    Browse the repository at this point in the history

  2. Fix north/south/east/west indexing. 

    Former-commit-id: c78d5fa
    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    f6453eb View commit details
    Browse the repository at this point in the history

  3. Tests for correct halo comm. 

    Former-commit-id: b544d24
    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    b8a1f70 View commit details
    Browse the repository at this point in the history

  4. Starting an MPI version of filling halo regions. 

    Former-commit-id: 50599f8
    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    e8632d8 View commit details
    Browse the repository at this point in the history

  5. Fix typos and stuf 

    Former-commit-id: 7da2fba
    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    ff0f875 View commit details
    Browse the repository at this point in the history

  6. Sending and reciving halo data. 

    Former-commit-id: 541c389
    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    0defdc9 View commit details
    Browse the repository at this point in the history

  7. Receiving into contiguous buffers. 

    Former-commit-id: 8eab25e
    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    a75768b View commit details
    Browse the repository at this point in the history

  8. Sending buffers too... 

    Former-commit-id: 5435f78
    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    d4718c8 View commit details
    Browse the repository at this point in the history

  9. Woops, actually read from MPI buffer. 

    Former-commit-id: c1e3375
    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    e4083ee View commit details
    Browse the repository at this point in the history

  10. Halo comm working on 4 ranks 

    Former-commit-id: bd5cf58
    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    95b2edf View commit details
    Browse the repository at this point in the history

  11. Nice send and receive halo functions 

    Former-commit-id: 971a4f6
    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    800cb80 View commit details
    Browse the repository at this point in the history

  12. Fix nice functions 

    Former-commit-id: f7f4c22
    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    1e7a910 View commit details
    Browse the repository at this point in the history

  13. Less MPI issues, bigger problem 

    Former-commit-id: 971d12a
    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    d455bd9 View commit details
    Browse the repository at this point in the history

  14. Need CuArrays for broadcast 

    Former-commit-id: e38f3b8
    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    9c0748f View commit details
    Browse the repository at this point in the history

  15. note to self 

    Former-commit-id: 357c2c9
    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    14e6a84 View commit details
    Browse the repository at this point in the history

  16. Start prototyping a DistributedModel type.

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    d337ebf View commit details
    Browse the repository at this point in the history

  17. DistributedModel constructor that checks for consistent ranks

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    7ca306c View commit details
    Browse the repository at this point in the history

  18. Utility functions for converting between MPI rank and 3D index.

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    e16190e View commit details
    Browse the repository at this point in the history

  19. Make z the fast index and convert to 1-based indexing

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    aff034b View commit details
    Browse the repository at this point in the history

  20. Add size and length kwargs, and create proper grid for each rank

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    b789722 View commit details
    Browse the repository at this point in the history

  21. More general left/right endpoints for grid

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    2540596 View commit details
    Browse the repository at this point in the history

  22. Compute connectivity graph assuming box model and brick MPI topology 

    The connectivity graph needs to take into account periodic boundary 
conditions.
    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    d446d12 View commit details
    Browse the repository at this point in the history

  23. Communication boundary condition type for MPI

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    b56b13c View commit details
    Browse the repository at this point in the history

  24. Root process should gather connectivities from each rank

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    117853d View commit details
    Browse the repository at this point in the history

  25. We can MPI.Finalize in a finalizer for DistributedModel

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    89e8c5b View commit details
    Browse the repository at this point in the history

  26. Gotta be careful and run the rest of the script as rank 0

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    41346ce View commit details
    Browse the repository at this point in the history

  27. Finalizer could be a bad idea actually

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    724d2e0 View commit details
    Browse the repository at this point in the history

  28. Send and received connectivities as named tuples

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    c45cc4c View commit details
    Browse the repository at this point in the history

  29. No master/root model struct. Every rank stores their own model

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    e594c29 View commit details
    Browse the repository at this point in the history

  30. Cleanup

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    cd03e13 View commit details
    Browse the repository at this point in the history

  31. Isolate construction of connectivity graph

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    f37d607 View commit details
    Browse the repository at this point in the history

  32. Account for periodic boundary conditions when constructing connectivity

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    2d8cb40 View commit details
    Browse the repository at this point in the history

  33. Need to know index

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    ad29559 View commit details
    Browse the repository at this point in the history

  34. Split test

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    54c1860 View commit details
    Browse the repository at this point in the history

  35. Inject halo communication boundary conditions 

    But need to figure out what to do with pressure solver as it tries
to set up an NNN Poisson solver on some processes...
    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    ebbdc91 View commit details
    Browse the repository at this point in the history

  36. Move to an Oceananigans.Distributed submodule.

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    bd7fd4c View commit details
    Browse the repository at this point in the history

  37. Properly pass Model kwargs.

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    e99f78f View commit details
    Browse the repository at this point in the history

  38. No need to MPI.Finalize() anymore

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    db394c0 View commit details
    Browse the repository at this point in the history

  39. Multiple dispatch to fill west halo with MPI 😍

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    cabf31b View commit details
    Browse the repository at this point in the history

  40. Actually fill in the west halo. Also start to generalize.

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    35bb674 View commit details
    Browse the repository at this point in the history

  41. Oceananigans.Distributed: use macros to fill each halo.

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    4566636 View commit details
    Browse the repository at this point in the history

  42. Hmmm, halo communication seems to deadlock :(

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    227f2be View commit details
    Browse the repository at this point in the history

  43. Modernize distributed_model.jl

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    5cdeac3 View commit details
    Browse the repository at this point in the history

  44. Some basic tests for slab decomposition models

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    c84a958 View commit details
    Browse the repository at this point in the history

  45. Properly inject halo communication BCs

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    571eba5 View commit details
    Browse the repository at this point in the history

  46. New multi-architectures

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    ea72c5d View commit details
    Browse the repository at this point in the history

  47. Success communicating east/west halos!

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    c92dbb8 View commit details
    Browse the repository at this point in the history

  48. More responsibilities for MultiCPU

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    50fc387 View commit details
    Browse the repository at this point in the history

  49. Modular tests

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    a04c583 View commit details
    Browse the repository at this point in the history

  50. Need more files

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    bb9ab67 View commit details
    Browse the repository at this point in the history

  51. East/west halo communication passes tests!

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    493e84e View commit details
    Browse the repository at this point in the history

  52. More modular halo communication

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    e117b13 View commit details
    Browse the repository at this point in the history

  53. Send and receive views to avoid memory allocations

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    3cd719c View commit details
    Browse the repository at this point in the history

  54. Nuke super ancient sandbox

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    8fcfe01 View commit details
    Browse the repository at this point in the history

  55. Beautiful metaprogramming for halo communication

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    f816d54 View commit details
    Browse the repository at this point in the history

  56. Testing xy decompositions

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    d18f670 View commit details
    Browse the repository at this point in the history

  57. Add include_corners kwarg for halo functions

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    fff4de7 View commit details
    Browse the repository at this point in the history

  58. Test that halo communication doesn't leak

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    0dec958 View commit details
    Browse the repository at this point in the history

  59. Distributed FFT based Poisson solver

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    7dbe2bb View commit details
    Browse the repository at this point in the history

  60. Janky Poisson solve works out

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    da21e2f View commit details
    Browse the repository at this point in the history

  61. Gotta send boundary points, not halos!

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    20d5f07 View commit details
    Browse the repository at this point in the history

  62. Distributed Poisson solver solution is divergence-free!

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    5e14925 View commit details
    Browse the repository at this point in the history

  63. In-place distributed FFTs

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    42c36e4 View commit details
    Browse the repository at this point in the history

  64. Distributed Poisson solver needs work for rectangular grids

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    999d1f4 View commit details
    Browse the repository at this point in the history

  65. Pass distributed pressure solver to model

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    1b2bda2 View commit details
    Browse the repository at this point in the history

  66. MPI incompressible turbulence!

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    080db74 View commit details
    Browse the repository at this point in the history

  67. Gotta communicate pressure halos

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    631d7ef View commit details
    Browse the repository at this point in the history

  68. Beautiful MPI turbulence

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    c6b92eb View commit details
    Browse the repository at this point in the history

  69. Fix bug in grid used for distributed pressure BCs

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    eddbdb0 View commit details
    Browse the repository at this point in the history

  70. Make it easier to run MPI tests

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    72cb4e2 View commit details
    Browse the repository at this point in the history

  71. Add MPI.jl and PencilFFTs.jl as dependencies

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    865164e View commit details
    Browse the repository at this point in the history

  72. New Distributed sub-module

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    efe89da View commit details
    Browse the repository at this point in the history

  73. Move MPI tests into test directory

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    e75209a View commit details
    Browse the repository at this point in the history

  74. MPI tests passing locally

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    0356e0d View commit details
    Browse the repository at this point in the history

  75. Need a distributed solve_for_pressure

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    bc11899 View commit details
    Browse the repository at this point in the history

  76. Test time stepping and running simulations

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    bfcf223 View commit details
    Browse the repository at this point in the history

  77. DistributedModel -> DistributedIncompressibleModel

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    703f9e0 View commit details
    Browse the repository at this point in the history

  78. Add new Buildkite job for MPI

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    c1e4c2c View commit details
    Browse the repository at this point in the history

  79. Resolve packages for Julia 1.5

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    655bb44 View commit details
    Browse the repository at this point in the history

  80. Fix dispatch for fill_halo_regions!

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    8869b3d View commit details
    Browse the repository at this point in the history

  81. Buildkite should have access to mpiexec now

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    56c9727 View commit details
    Browse the repository at this point in the history

  82. Use system MPI on Buildkite/Tartarus

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    a89d248 View commit details
    Browse the repository at this point in the history

  83. Escape characters

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    4307d82 View commit details
    Browse the repository at this point in the history

  84. Fixing dispatch for fill_halo_regions!: part 2

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    fddca94 View commit details
    Browse the repository at this point in the history

  85. Everyone gets system MPI

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    e240070 View commit details
    Browse the repository at this point in the history

  86. Use MPI.jl's mpiexecjl

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    56695d7 View commit details
    Browse the repository at this point in the history

  87. Need julia binary in $PATH for mpiexecjl to work

    @ali-ramadhan
    ali-ramadhan committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    922f155 View commit details
    Browse the repository at this point in the history

Commits on Mar 5, 2021

  1. Better design for multi-architectures and distributed models

    @ali-ramadhan
    ali-ramadhan committed Mar 5, 2021
    Configuration menu
    Copy the full SHA
    0d59c69 View commit details
    Browse the repository at this point in the history

  2. Update tests and Buildkite zoo

    @ali-ramadhan
    ali-ramadhan committed Mar 5, 2021
    Configuration menu
    Copy the full SHA
    da22b16 View commit details
    Browse the repository at this point in the history

  3. More tests

    @ali-ramadhan
    ali-ramadhan committed Mar 5, 2021
    Configuration menu
    Copy the full SHA
    4baba22 View commit details
    Browse the repository at this point in the history

  4. Address PR comments

    @ali-ramadhan
    ali-ramadhan committed Mar 5, 2021
    Configuration menu
    Copy the full SHA
    f7eb664 View commit details
    Browse the repository at this point in the history

  5. Bump v0.53.0

    @ali-ramadhan
    ali-ramadhan committed Mar 5, 2021
    Configuration menu
    Copy the full SHA
    faa0729 View commit details
    Browse the repository at this point in the history

Commits on Mar 9, 2021

  1. Update src/Distributed/halo_communication.jl

    @ali-ramadhan
    ali-ramadhan authored Mar 9, 2021
    Configuration menu
    Copy the full SHA
    42d2eab View commit details
    Browse the repository at this point in the history

Commits on Mar 10, 2021

  1. Inject halo communication BCs in Field constructor

    @ali-ramadhan
    ali-ramadhan committed Mar 10, 2021
    Configuration menu
    Copy the full SHA
    3ae697f View commit details
    Browse the repository at this point in the history

  2. Nuke sandbox

    @ali-ramadhan
    ali-ramadhan committed Mar 10, 2021
    Configuration menu
    Copy the full SHA
    50c9b7a View commit details
    Browse the repository at this point in the history

  3. Some renaming and added a communicator to MultiCPU

    @ali-ramadhan
    ali-ramadhan committed Mar 10, 2021
    Configuration menu
    Copy the full SHA
    3a4f9ea View commit details
    Browse the repository at this point in the history

  4. Left to right

    @ali-ramadhan
    ali-ramadhan committed Mar 10, 2021
    Configuration menu
    Copy the full SHA
    447e121 View commit details
    Browse the repository at this point in the history

  5. Strong scaling benchmark for incompressible model

    @ali-ramadhan
    ali-ramadhan committed Mar 10, 2021
    Configuration menu
    Copy the full SHA
    3992e47 View commit details
    Browse the repository at this point in the history

  6. Slightly better strong scaling benchmark

    @ali-ramadhan
    ali-ramadhan committed Mar 10, 2021
    Configuration menu
    Copy the full SHA
    b3a75b8 View commit details
    Browse the repository at this point in the history

  7. Update [compat] entries

    @ali-ramadhan
    ali-ramadhan committed Mar 10, 2021
    Configuration menu
    Copy the full SHA
    ab3e539 View commit details
    Browse the repository at this point in the history