Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implementation of chunk balancer feature #1775

Merged
merged 10 commits into from
Oct 9, 2021

Commits on Sep 16, 2021

  1. Implementation of chunk balancer feature

    CONTEXT: This feature uses empirical data about the timings of each MPI process to adjust the chunk sizes to achieve optimal load balancing for repeated simulations, such as iterations in an optimization run. Processes which are consistently slow and overworked will have their chunk sizes reduced for the next iteration of an optimization run, and underworked processes will handle larger chunks. This approach has the advantage that it implicitly incorporates variable load between different machines running the MPI processes.
    
    SCOPE:
    - Added abstract class for chunk balancer
    - Added DefaultChunkBalancer implementation, which adjusts chunk sizes according to per-process working time (ignoring all-all comms) while maintaining previous split directions
    - Wrote unit tests for DefaultChunkBalancer to check for improvement in load-balancing, convergence to a load-balanced state, and that split_pos values are adjusted correctly for each iteration.
    - Wrote unit tests for the MockSimulation class used by the DefaultChunkBalancerTests
    - Added a binary_partition_utils.py library which includes lots of tree traversal algorithms which are useful to have for the chunk balancer
    - Wrote unit tests for binary_partition_utils.py
    bencbartlett committed Sep 16, 2021
    Configuration menu
    Copy the full SHA
    4e1b67c View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2021

  1. Configuration menu
    Copy the full SHA
    2543719 View commit details
    Browse the repository at this point in the history

Commits on Sep 29, 2021

  1. Configuration menu
    Copy the full SHA
    f816486 View commit details
    Browse the repository at this point in the history
  2. bugfix in tests

    bencbartlett committed Sep 29, 2021
    Configuration menu
    Copy the full SHA
    a99227d View commit details
    Browse the repository at this point in the history
  3. more test bugfixes

    bencbartlett committed Sep 29, 2021
    Configuration menu
    Copy the full SHA
    a4471a0 View commit details
    Browse the repository at this point in the history
  4. avoid merge conflict

    bencbartlett committed Sep 29, 2021
    Configuration menu
    Copy the full SHA
    168f147 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    cd7c1d8 View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2021

  1. refactored class names, added documentation for adjusting chunk layou…

    …ts between runs of a program
    bencbartlett committed Oct 7, 2021
    Configuration menu
    Copy the full SHA
    e3ed37b View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    371d1f3 View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2021

  1. test bugfix

    bencbartlett committed Oct 8, 2021
    Configuration menu
    Copy the full SHA
    d89104b View commit details
    Browse the repository at this point in the history