Grid SparsePartitioner

Overview

The Cabana::Grid SparsePartitioner is used to partition sparse grids during the simulation steps. One can optimize the grid partition (unit: tile) on multiple MPI ranks, to makes the workload distributed on multiple MPI ranks in a relatively balanced manner.

Implementation

Header File: Cabana_Grid_SparsePartitioner.hpp Usage: Store the partition information for sparse grids; Optimize partition on multiple MPI ranks according to current tile distribution.

Example

• Initialize sparse partitioner

    using namespace Cabana::Grid;

    // define the domain size
    constexpr int size_tile_per_dim = 32;
    constexpr int cell_per_tile_dim = 4;
    constexpr int size_per_dim = size_tile_per_dim * cell_per_tile_dim;
    
    // some settings for partitioner
    float max_workload_coeff = 1.5;
    int particle_num = total_size;
    int num_step_rebalance = 100;
    int max_optimize_iteration = 10;

    // cells per dimension in the whole simulation domain
    std::array<int, 3> global_cells_per_dim = { size_per_dim, size_per_dim,
                                                size_per_dim };

    // partitioner def
    SparseDimPartitioner<TEST_DEVICE, cell_per_tile_dim> partitioner(
        MPI_COMM_WORLD, max_workload_coeff, particle_num, num_step_rebalance,
        global_cells_per_dim, max_optimize_iteration );

    // initialize partitions (average partition)
    std::array<std::vector<int>, 3> rec_partitions;
    for ( int d = 0; d < 3; ++d )
    {
        int ele = size_tile_per_dim / ranks_per_dim[d];
        int part = 0;
        for ( int i = 0; i < ranks_per_dim[d]; ++i )
        {
            rec_partitions[d].push_back( part );
            part += ele;
        }
        rec_partitions[d].push_back( size_tile_per_dim );
    }

    partitioner.initializeRecPartition( rec_partitions[0], rec_partitions[1],
                                        rec_partitions[2] );

• Optimize partitioner

    auto sis = createSparseMap<TEST_EXECSPACE>( global_mesh, pre_alloc_size );
    // do sth ... (register active tiles in the sparse map)
    partitioner.optimizePartition( sis, MPI_COMM_WORLD );

• check the current partitioner and check the imbalance factor of the current MPI rank

    std::array<std::vector<int>, 3> part = partitioner.getCurrentPartition();
    float imbalance_factor = partitioner.computeImbalanceFactor( cart_comm );

This is part of the Programming Guide series