Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#410: Dependent Epochs rewritten #2204

Open
wants to merge 20 commits into
base: develop
Choose a base branch
from
Open

Conversation

lifflander
Copy link
Collaborator

Fixes #410

  • After discussions about what is needed to support the resilience work, I've reopened this issue and semi-rebased/rewrote the old PR (that was closed) in a much cleaner way.
  • I believe the current implementation adds very little overhead to any of the current code paths.
  • This generalizes epochs allowing more complex dependency structures.

@github-actions
Copy link

github-actions bot commented Oct 18, 2023

Pipelines results

PR tests (gcc-8, ubuntu, mpich, address sanitizer)

Build for 48f7a8c (2023-10-31 19:56:21 UTC)

Compilation - successful

Testing - passed

Build log


PR tests (clang-9, ubuntu, mpich)

Build for 48f7a8c (2023-10-31 19:56:21 UTC)

Compilation - successful

Testing - passed

Build log


PR tests (clang-11, ubuntu, mpich)

Build for 48f7a8c (2023-10-31 19:56:21 UTC)

Compilation - successful

Testing - passed

Build log


PR tests (gcc-11, ubuntu, mpich, trace runtime, coverage)

Build for 48f7a8c (2023-10-31 19:56:21 UTC)

Compilation - successful

Testing - passed

Build log


PR tests (gcc-10, ubuntu, openmpi, no LB)

Build for 48f7a8c (2023-10-31 19:56:21 UTC)

Compilation - successful

Testing - passed

Build log


PR tests (clang-13, ubuntu, mpich)

Build for 48f7a8c (2023-10-31 19:56:21 UTC)

Compilation - successful

Testing - passed

Build log


PR tests (gcc-9, ubuntu, mpich, zoltan, json schema test)

Build for 48f7a8c (2023-10-31 19:56:21 UTC)

Compilation - successful

Testing - passed

Build log


PR tests (clang-12, ubuntu, mpich)

Build for 48f7a8c (2023-10-31 19:56:21 UTC)

Compilation - successful

Testing - passed

Build log


PR tests (clang-14, ubuntu, mpich)

Build for 48f7a8c (2023-10-31 19:56:21 UTC)

Compilation - successful

Testing - passed

Build log


PR tests (clang-10, ubuntu, mpich)

Build for 48f7a8c (2023-10-31 19:56:21 UTC)

Compilation - successful

Testing - passed

Build log


PR tests (gcc-12, ubuntu, mpich)

Build for 48f7a8c (2023-10-31 19:56:21 UTC)

Compilation - successful

Testing - passed

Build log


PR tests (intel icpc, ubuntu, mpich)

Build for 48f7a8c (2023-10-31 19:56:21 UTC)

Compilation - successful

Testing - passed

Build log


PR tests (nvidia cuda 11.2, gcc-9, ubuntu, mpich)

Build for 48f7a8c (2023-10-31 19:56:21 UTC)

/vt/src/vt/pipe/pipe_manager.impl.h(133): warning: missing return statement at end of non-void function "vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&vt::vrt::collection::lb::GreedyLB::collectHandler, Target=vt::objgroup::proxy::ProxyElm<vt::vrt::collection::lb::GreedyLB>]"
          detected during:
            instantiation of "auto vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&vt::vrt::collection::lb::GreedyLB::collectHandler, Target=vt::objgroup::proxy::ProxyElm<vt::vrt::collection::lb::GreedyLB>]" 
/vt/src/vt/objgroup/proxy/proxy_objgroup.impl.h(154): here
            instantiation of "vt::objgroup::proxy::Proxy<ObjT>::PendingSendType vt::objgroup::proxy::Proxy<ObjT>::reduce<f,Op,Target,Args...>(Target, Args &&...) const [with ObjT=vt::vrt::collection::lb::GreedyLB, f=&vt::vrt::collection::lb::GreedyLB::collectHandler, Op=vt::collective::PlusOp, Target=vt::objgroup::proxy::ProxyElm<vt::vrt::collection::lb::GreedyLB>, Args=<vt::vrt::collection::lb::GreedyPayload>]" 
/vt/src/vt/vrt/collection/balance/greedylb/greedylb.cc(222): here

/vt/src/vt/pipe/pipe_manager.impl.h(133): warning: missing return statement at end of non-void function "vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&MyObj::handler, Target=vt::objgroup::proxy::ProxyElm<MyObj>]"
          detected during instantiation of "auto vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&MyObj::handler, Target=vt::objgroup::proxy::ProxyElm<MyObj>]" 
/vt/examples/callback/callback.cc(147): here

/vt/src/vt/pipe/pipe_manager.impl.h(133): warning: missing return statement at end of non-void function "vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&colHan, Target=vt::vrt::collection::VrtElmProxy<MyCol, vt::Index1D>]"
          detected during instantiation of "auto vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&colHan, Target=vt::vrt::collection::VrtElmProxy<MyCol, vt::Index1D>]" 
/vt/examples/callback/callback.cc(153): here

/vt/src/vt/pipe/pipe_manager.impl.h(133): warning: missing return statement at end of non-void function "vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&MyObj::handler, Target=vt::objgroup::proxy::ProxyElm<MyObj>]"
          detected during instantiation of "auto vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&MyObj::handler, Target=vt::objgroup::proxy::ProxyElm<MyObj>]" 
/vt/examples/callback/callback.cc(147): here

/vt/src/vt/pipe/pipe_manager.impl.h(133): warning: missing return statement at end of non-void function "vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&colHan, Target=vt::vrt::collection::VrtElmProxy<MyCol, vt::Index1D>]"
          detected during instantiation of "auto vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&colHan, Target=vt::vrt::collection::VrtElmProxy<MyCol, vt::Index1D>]" 
/vt/examples/callback/callback.cc(153%0D%0A%0D%0A%0D%0A ==> And there is more. Read log. <==

Build log


*
* \return the new epoch
*/
EpochType makeEpochRooted(
UseDS use_ds = UseDS{true},
ParentEpochCapture parent = ParentEpochCapture{}
ParentEpochCapture parent = ParentEpochCapture{},
bool is_dep = false
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Boolean parameters (especially defaulted ones) are a bit of an anti-pattern. Preferable to use named enum class members instead

NodeType this_node_ = uninitialized_destination;
EpochStackType epoch_stack_;
// released epoch list for dependent epochs
std::unordered_set<EpochType> epoch_released_ = {};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this be a range container as is used for other stuff like this, so that it won't tend to grow without bound?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I was planning this improvement. The bounds are lower than one might think because they are always removed once the epoch terminates.

vt::theTerm()->addAction([=]{
EXPECT_EQ(TestDep::num_non_dep, 1);
EXPECT_EQ(TestDep::num_dep, (num_nodes - 1)*k);
});
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not a fan of the style choice of having the scheduler run and the test assertions get checked as a result of running to quiescence as the process is finishing up. If nothing else, it makes for really funky looking stack traces if/when an assertion fails.

@@ -384,6 +384,17 @@ void Scheduler::resume(ThreadIDType tid) {
suspended_.resumeRunnable(tid);
}

void Scheduler::releaseEpoch(EpochType ep) {
auto iter = pending_work_.find(ep);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another possible use of pending_work_.extract(ep)

Comment on lines 97 to 98
ep != no_epoch and ep != term::any_epoch_sentinel and
not theTerm()->epochReleased(ep))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't epochReleased() already check for the no_epoch and any_epoch_sentinel values?

if (node_ == theContext()->getNode()) {
theSched()->releaseEpochObjgroup(epoch, proxy_);
} else {
theMsg()->send<releaseRemoteObjGroup<ObjT>>(vt::Node(node_), *this, epoch);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it going to be valid to release an epoch for a single node?

if (auto iter2 = iter->second.find(untyped); iter2 != iter->second.end()) {
auto& container = iter2->second;
while (container.size() > 0) {
work_queue_.emplace(container.pop());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe better to move the values from container, rather than actually removing them, so that no work needs to be done on the structure of container itself, which is going away right after this loop.

I.e.

for (auto &unit : container) {
  work_queue_.emplace(std::move(unit));
}

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem with this is that container is not iterable. The container is a hybrid queue implemented differently depending on whether priorities are enabled. If they are not, it is a 64 entry circular buffer with an overflow std::queue container. We could make it iterable, but it would need a custom iterator.

}
}
} else if (not theTerm()->epochReleased(ep)) {
pending_work_[ep].push(unit);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The nesting is getting really deep here. It might be nice to peel off some of the cases as early returns, if/as possible.

@PhilMiller
Copy link
Member

PhilMiller commented Oct 20, 2023

So, the one big question I have in reviewing this is: why the move to releasing epochs per-collection/objgroup/object?

Everything else I've marked in comments is pretty minor, and can/should be addressed after writing up an answer for the big question.

Also, squashing fixup commits into their progenitors would be good. Or, even rewriting the history entirely to provide a clearer to understand narrative sequence of commits.

@Matthew-Whitlock
Copy link
Contributor

It looks like creating an epoch within a dependent epoch circumvents waiting for release on all of the messages within the nested epoch. As is, I don't think we could make any assertions about the behavior of general code like this:

auto epoch = vt::theTerm()->makeEpochCollective(term::ParentEpochCapture{}, true);
vt::theMsg()->pushEpoch(epoch);

some::library::function();

vt::theMsg()->popEpoch(epoch);
vt::theTerm()->finishedEpoch(epoch);

vt::theCollective()->barrier();

auto epoch = vt::theTerm()->makeEpochCollective(term::ParentEpochCapture{}, true);
vt::theMsg()->pushEpoch(epoch);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since dependent epochs make this pattern necessary, a returning equivalent of vt::runInEpoch<Collective/Rooted> would be a nice helper for users.

theSched()->releaseEpoch(epoch);
}

void TerminationDetector::onReleaseEpoch(EpochType epoch, ActionType action) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should use the same approach as term actions after #2196

Comment on lines +1172 to +1178
// Terminated epochs are always released
bool const is_term = theEpoch()->getTerminatedWindow(epoch)->isTerminated(
epoch
);
if (is_term) {
return true;
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this should be true, or an empty dependent epoch could terminate before being released, which seems counter-intuitive.

auto epoch = vt::theTerm()->makeEpochCollective(term::ParentEpochCapture{}, true);
vt::theTerm()->finishedEpoch(epoch);
vt::theTerm()->addAction(epoch, {...});

Your added action could run before releasing in the above, or in cases where a set of functions may or may not actually send messages. We should produce once on dependent epoch creation and consume on release, to align the no-messages behavior with typical behavior.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lifflander What are your thoughts on Matthew's concern?

@lifflander
Copy link
Collaborator Author

So, the one big question I have in reviewing this is: why the move to releasing epochs per-collection/objgroup/object?

The short answer to this question is that it will allow us to express dependencies per-object. An object can release work when it's ready, which is a per-object state.

Everything else I've marked in comments is pretty minor, and can/should be addressed after writing up an answer for the big question.

Also, squashing fixup commits into their progenitors would be good. Or, even rewriting the history entirely to provide a clearer to understand narrative sequence of commits.

Yes, I am planning on re-writing the history.

@lifflander
Copy link
Collaborator Author

lifflander commented Oct 23, 2023

@PhilMiller @Matthew-Whitlock Please take a look at the follow-on PR #2206. I nearly have the new abstraction working on top of dependent epochs called taskCollective. Here is the example Jacobi program. This might clarify why the dependencies are per-object.

void runToConvergence() {
while (not converged_ and cur_iter_ < max_iter_) {
vt_print(gen, "runToConvergence: cur_iter_={}, max_iter_={}\n", cur_iter_, max_iter_);
auto xl = chains_->taskCollective("exchange left", [&](auto idx, auto t) {
if (prev_kernel) {
t->dependsOn(idx, *prev_kernel);
}
return proxy_[idx].template send<&LinearPb1DJacobi::sendLeft>();
});
auto xr = chains_->taskCollective("exchange right", [&](auto idx, auto t) {
if (prev_kernel) {
t->dependsOn(idx, *prev_kernel);
}
return proxy_[idx].template send<&LinearPb1DJacobi::sendRight>();
});
auto kernel = chains_->taskCollective("kernel", [&](auto idx, auto t) {
if (idx.x() != 0) {
auto left = vt::Index1D(idx.x() - 1);
t->dependsOn(left, *xr);
}
if (static_cast<std::size_t>(idx.x()) != num_objs_ - 1) {
auto right = vt::Index1D(idx.x() + 1);
t->dependsOn(right, *xl);
}
return proxy_[idx].template send<&LinearPb1DJacobi::kernel>();
});
prev_kernel = kernel;
if (cur_iter_++ % check_conv_freq == 0) {
chains_->taskCollective("checkConv", [&](auto idx, auto t) {
t->dependsOn(idx, *kernel);
auto cb = vt::theCB()->makeBcast<&NodeObj::reducedNorm>(this_proxy_);
return proxy_[idx].template send<&LinearPb1DJacobi::reduceMaxNorm>(cb);
});
chains_->waitForTasks();
chains_->startTasks();
}
}
if (not converged_) {
chains_->waitForTasks();
fmt::print("Maximum Number of Iterations Reached without convergence.\n");
} else {
fmt::print("Convergence is reached at iteration {}.\n", cur_iter_);
}
chains_->phaseDone();
}

@PhilMiller
Copy link
Member

Ok, I'll take a look. That maybe answers my follow-up question of what it means to release an epoch for some object/collection, but not for others.

@lifflander
Copy link
Collaborator Author

Ok, I'll take a look. That maybe answers my follow-up question of what it means to release an epoch for some object/collection, but not for others.

@Matthew-Whitlock I have the 2-D Jacobi correctly running now with asynchronous iterations. It runs faster (even with the dependency resolution) than the synchronized version.

auto iteration = chains_->createTaskRegion([&]{
std::array<Tuple, 4> dirs = {
Tuple{-1, 0}, Tuple{0,-1}, Tuple{1,0}, Tuple{0,1}
};
std::map<std::tuple<int, int>, vt::task::TaskCollective<vt::Index2D>*> dep_dir;
for (auto const& [x, y] : dirs) {
auto dep = chains_->taskCollective("exchange", [&](auto idx, auto t) {
if (prev_kernel) {
t->dependsOn(idx, prev_kernel);
vt::Index2D idx_dir{idx.x() + x, idx.y() + y};
if (idx_dir.x() >= 0 and idx_dir.x() < int(num_objs_x_) and
idx_dir.y() >= 0 and idx_dir.y() < int(num_objs_y_)) {
t->dependsOn(idx_dir, prev_kernel);
}
}
return proxy_[idx].template send<&LinearPb2DJacobi::sendNeighbor>(Tuple{x,y});
});
dep_dir[Tuple{x,y}] = dep;
}
prev_kernel = chains_->taskCollective("kernel", [&](auto idx, auto t) {
if (idx.x() != 0) {
t->dependsOn(vt::Index2D(idx.x() - 1, idx.y()), dep_dir[Tuple{1,0}]);
}
if (idx.y() != 0) {
t->dependsOn(vt::Index2D(idx.x(), idx.y() - 1), dep_dir[Tuple{0,1}]);
}
if (static_cast<std::size_t>(idx.x()) != num_objs_x_ - 1) {
t->dependsOn(vt::Index2D(idx.x() + 1, idx.y()), dep_dir[Tuple{-1,0}]);
}
if (static_cast<std::size_t>(idx.y()) != num_objs_y_ - 1) {
t->dependsOn(vt::Index2D(idx.x(), idx.y() + 1), dep_dir[Tuple{0,-1}]);
}
return proxy_[idx].template send<&LinearPb2DJacobi::kernel>();
});
});
while (not converged_ and cur_iter_ < max_iter_) {
iteration->enqueueTasks();
if (cur_iter_++ % check_conv_freq == 0) {
chains_->taskCollective("checkConv", [&](auto idx, auto t) {
t->dependsOn(idx, prev_kernel);
auto cb = vt::theCB()->makeBcast<&NodeObj::reducedNorm>(this_proxy_);
return proxy_[idx].template send<&LinearPb2DJacobi::reduceMaxNorm>(cb);
});
iteration->waitCollective();
vt::thePhase()->nextPhaseCollective();
}
}

@@ -89,6 +89,24 @@ struct EpochManip : runtime::component::Component<EpochManip> {
*/
static bool isRooted(EpochType const& epoch);

/**
* \brief Gets whether an epoch is DS or onot
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo.

static bool isDS(EpochType epoch);

/**
* \brief Gets whether an epoch is dependent or onot
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo.

Comment on lines +375 to +394
/**
* \brief Release an epoch to run
*
* \param[in] ep the epoch to release
*/
void releaseEpoch(EpochType ep);

/**
* \brief Release an epoch to run
*
* \param[in] ep the epoch to release
*/
void releaseEpochObjgroup(EpochType ep, ObjGroupProxyType proxy);

/**
* \brief Release an epoch to run
*
* \param[in] ep the epoch to release
*/
void releaseEpochCollection(EpochType ep, UntypedCollection* untyped);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Doxygen here needs to be improved.

Comment on lines +130 to +131
auto obj = r->getObj();
if (ep != no_epoch and ep != term::any_epoch_sentinel) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with Phil's comment about peeling off cases. If you're not going to do that, you might at least reverse the order of these two lines.

*
* \return whether it is released
*/
bool isReleasedEpoch(EpochType epoch) const {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might want to call this isEpochReleased() to have naming consistent with theTerm()->isEpochReleased().


if (is_dep) {
// Put the epoch in the released set. The epoch any_epoch_sentinel does not
// count as a succcessor.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo.

@@ -814,16 +863,20 @@ struct TerminationDetector :
// hang detector termination state
TermStateType hang_;
private:
using ActionListType = std::list<ActionType>;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why was ActionListType defined here?

Comment on lines +1172 to +1178
// Terminated epochs are always released
bool const is_term = theEpoch()->getTerminatedWindow(epoch)->isTerminated(
epoch
);
if (is_term) {
return true;
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lifflander What are your thoughts on Matthew's concern?

@ppebay ppebay changed the title 410 Dependent Epochs rewritten #410: Dependent Epochs rewritten May 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Semantics of collection chain set wrt nextStepCollective
4 participants