Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce more flexible storeChunk() syntax, use to add ADIOS2 memory selection #1620

Open
wants to merge 39 commits into
base: dev
Choose a base branch
from

Conversation

franzpoeschel
Copy link
Contributor

@franzpoeschel franzpoeschel commented May 17, 2024

We currently don't support memory selections yet, e.g. when you have a 2D memory buffer but want to write only an arbitrary block from it. That block is then non-contiguous, but ADIOS2 has so-called memory selections to support this.

We currently don't have an API that would be able to expose this (except in Python where the native Python buffer protocol is powerful enough to express this; we currently throw an error when detecting this strides in selection are inefficient, not implemented!). Since the loadChunk()/storeChunk() API is somewhat convoluted by now anyway, I didn't want to add even further overloads.

Instead, this PR proposes a new syntax for setting up load/store calls:

RecordComponent E_y;

E_y.prepareLoadStore()
    .offset({0, 5}) // optional
    .extent({1,5}) // optional
    .enqueueLoad<int>(); // -> std::shared_ptr<int>

E_y.prepareLoadStore()
    .offset({0, 5}) // optional
    .extent({1,5}) // optional
    .enqueueLoadVariant(); // ->std::variant<std::shared_ptr<char>, ...>

E_y.prepareLoadStore()
    .offset({0, 5}) // optional
    .extent({1,5}) // optional
    .enqueueStore<int>(); // -> DynamicMemoryView<int>, i.e. Span API

std::shared_ptr<int> data;
E_y.prepareLoadStore()
    .offset({0, 5}) // optional
    .extent({1,5}) // optional
    .withSharedPtr(data) // or withUniquePtr(), withRawPtr(), withContiguousContainer()
    .memorySelection({{1,1},{5,5}}) // call only available after having called withSharedPtr()
    .enqueueStore(); // -> void, no template parameter needed since withSharedPtr(data) already gives the type

E_y.prepareLoadStore()
    .withSharedPtr(data) // or withUniquePtr(), withRawPtr(), withContiguousContainer()
    .enqueueLoad(); // load the whole dataset into the buffer

This API makes it easier to add functionality without going through every single overload and adapting it. Further, it makes the API more consistent, e.g. offset and extent are now always optional and not just for those calls that implement a default logic.

With this PR, the old API calls are now fully implemented via the new API, giving it a good test coverage.

TODO:

  • What about Python? Maybe stick with the buffer protocol there for now.
  • Experiment: Return futures instead of bare return types: Use the chance to introduce a safe async API. Introduce option .noop_future() to disable future return types.

src/LoadStoreChunk.cpp Fixed Show fixed Hide fixed
src/LoadStoreChunk.cpp Fixed Show fixed Hide fixed
@franzpoeschel franzpoeschel force-pushed the adios2-memory-selection branch 2 times, most recently from 332932f to efb0876 Compare June 25, 2024 13:56
src/LoadStoreChunk.cpp Fixed Show fixed Hide fixed
@franzpoeschel franzpoeschel force-pushed the adios2-memory-selection branch 4 times, most recently from 47d497b to b2bc355 Compare July 22, 2024 10:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants