Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Campaign manager #3932

Merged
merged 63 commits into from
Dec 15, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
60976c3
Added CampaignManager to be created by ADIOS object. It does nothing …
pnorbert May 15, 2023
45b0892
store campaign records in memory so far
pnorbert May 16, 2023
6825d94
Added example to write many different files from different communicat…
pnorbert May 17, 2023
b085e37
convert info to json at Close() and write out.
pnorbert May 19, 2023
08a0715
IO.cpp to create Campaign Reader
pnorbert May 22, 2023
fdfa1d6
add missing file
pnorbert May 22, 2023
f45262d
add explicit map for which engine supports ReadRandomAccess mode
pnorbert May 22, 2023
957ed2f
save work: campaing reader copies variables from subengines, bpls is …
pnorbert May 22, 2023
010beb7
Maintain a map for each variable (original variable, engine and IO) s…
pnorbert May 23, 2023
bb4fed9
Support Read(sync) and block reading.
pnorbert May 24, 2023
af3916e
Save campaign data in adios-campaign/
pnorbert Jun 5, 2023
68c65d9
added (beginning of) script to create campaing archive files
pnorbert Jun 6, 2023
5885c48
Pack more information in campaign archive.
pnorbert Jun 7, 2023
d89557b
don't use BP3
pnorbert Jun 20, 2023
c48d22b
Use SQLite3 database for campaign archive (instead of a zipfile)
pnorbert Jun 20, 2023
6ad3345
missing cmake file for campaign manager python script
pnorbert Jun 20, 2023
192543e
Add SQLite3 to build options, make CampaignReader dependent on it
pnorbert Jun 20, 2023
c43bbf8
campaign reader opens/closes sqlite db but it doesn't do anything yet
pnorbert Jun 20, 2023
d0c52e9
functions to get the Fully Qualified Domain Name, and a cluster name …
pnorbert Jun 20, 2023
875a7ff
CampaignReader starts using SQLite3
pnorbert Jun 20, 2023
0e35446
CampaignReader processes sqlite3 database. Able to open local files (…
pnorbert Jun 26, 2023
2f482ae
Save compressed metadata to disk and read from there.
pnorbert Jun 30, 2023
bb16f6a
check ctime of individual files and save from DB only if it is older
pnorbert Jun 30, 2023
a1732a7
add missing file
pnorbert Jul 12, 2023
14225cd
add another missing file
pnorbert Jul 12, 2023
f5877bf
enable using multiple directories in same host
pnorbert Jul 12, 2023
026f915
fix print
pnorbert Jul 12, 2023
6f53bb2
unfix print
pnorbert Jul 12, 2023
a3eba6d
add info command
pnorbert Jul 12, 2023
0730209
don't check campaign files when running info
pnorbert Jul 12, 2023
334eeca
compile some functions unconditionally on all platforms
pnorbert Jul 31, 2023
6049c7a
flake the campaign-manager python script
pnorbert Aug 8, 2023
8d2e617
do not convert all parameter values to lower case. Full paths may con…
pnorbert Aug 8, 2023
ffcb09b
Use RemoteDataPath parameter for BP5 to open remote data files. clang…
pnorbert Aug 8, 2023
86bc688
remove debug prints
pnorbert Aug 8, 2023
ad9f23c
remote unused variables and functions
pnorbert Aug 8, 2023
3af1909
reformat
pnorbert Aug 8, 2023
c01ace4
\\. for \.
pnorbert Aug 8, 2023
8eba0dc
don't include sqlite3.h in CampaignRecord.h yet
pnorbert Aug 8, 2023
539ba34
Merge branch 'master' into campaign-manager
pnorbert Sep 1, 2023
1208433
fixes for compiling
pnorbert Sep 1, 2023
3af9b55
StringTo<long> does not exist, use <int64_t> instead
pnorbert Sep 1, 2023
0a3e0bb
fix comm name
pnorbert Sep 1, 2023
934ae46
don't include netdb on windows
pnorbert Sep 1, 2023
c6ae903
when campaign reader uses sqlite3 it also needs zlib
pnorbert Sep 1, 2023
cc2f83a
Merge branch 'master' into campaign-manager
pnorbert Nov 17, 2023
e26b736
remove merge conflict line
pnorbert Nov 17, 2023
a6a13c1
Replace cmake option SQLite3 for Campaign, which depends on SQLite3 a…
pnorbert Nov 20, 2023
986eecd
Rename campaign extension to .aca, Adios Campaign Archive
pnorbert Nov 20, 2023
0532696
Fix building source without campaign support
pnorbert Nov 20, 2023
e650a74
Merge branch 'master' into campaign-manager
pnorbert Nov 20, 2023
ab9205c
Added CampaignManager to be created by ADIOS object. Added CampaignRe…
pnorbert May 15, 2023
e8b561b
add back deleted line
pnorbert Nov 21, 2023
434f7a7
Merge branch 'master' into campaign-manager
pnorbert Nov 21, 2023
3d50e73
do not include unistd on win32
pnorbert Nov 21, 2023
79f4463
format
pnorbert Nov 21, 2023
17f4398
attempt to fix windows build
pnorbert Nov 21, 2023
92f5bd6
attempt to fix linux build
pnorbert Nov 21, 2023
4d33f87
Merge branch 'master' into campaign-manager
pnorbert Dec 12, 2023
b14d43e
Compile CampaignManager.cpp only if ADIOS2_HAVE_CAMPAIGN, otherwise u…
pnorbert Dec 12, 2023
358b121
add the Campaign writer as example. This produces several files that …
pnorbert Dec 12, 2023
2652035
fix formatting
pnorbert Dec 12, 2023
4f7367a
remove unused variable to avoid warning in nvhpc build
pnorbert Dec 12, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# accompanying file Copyright.txt for details.
#------------------------------------------------------------------------------#

cmake_minimum_required(VERSION 3.12)
cmake_minimum_required(VERSION 3.14)

# Fail immediately if not using an out-of-source build
if(CMAKE_CURRENT_SOURCE_DIR STREQUAL CMAKE_CURRENT_BINARY_DIR)
Expand Down Expand Up @@ -170,6 +170,7 @@ adios_option(Profiling "Enable support for profiling" AUTO)
adios_option(Endian_Reverse "Enable support for Little/Big Endian Interoperability" AUTO)
adios_option(Sodium "Enable support for Sodium for encryption" AUTO)
adios_option(Catalyst "Enable support for in situ visualization plugin using ParaView Catalyst" AUTO)
adios_option(Campaign "Enable support for Campaigns (requires SQLite3 and ZLIB)" OFF)
adios_option(AWSSDK "Enable support for S3 compatible storage using AWS SDK's S3 module" OFF)
adios_option(Derived_Variable "Enable support for derived variables" OFF)
include(${PROJECT_SOURCE_DIR}/cmake/DetectOptions.cmake)
Expand Down Expand Up @@ -245,7 +246,7 @@ set(ADIOS2_CONFIG_OPTS
DataMan DataSpaces HDF5 HDF5_VOL MHS SST Fortran MPI Python Blosc2 BZip2
LIBPRESSIO MGARD MGARD_MDR PNG SZ ZFP DAOS IME O_DIRECT Sodium Catalyst SysVShMem UCX
ZeroMQ Profiling Endian_Reverse Derived_Variable AWSSDK GPU_Support CUDA Kokkos
Kokkos_CUDA Kokkos_HIP Kokkos_SYCL
Kokkos_CUDA Kokkos_HIP Kokkos_SYCL Campaign
)

GenerateADIOSHeaderConfig(${ADIOS2_CONFIG_OPTS})
Expand Down Expand Up @@ -411,7 +412,7 @@ message(" Examples: ${ADIOS2_BUILD_EXAMPLES}")
message(" Build Options:")

foreach(opt IN LISTS ADIOS2_CONFIG_OPTS)
message_pad(" ${opt}" 16 label)
message_pad(" ${opt}" 25 label)
if(${ADIOS2_HAVE_${opt}})
message("${label}: ON")
else()
Expand All @@ -424,3 +425,4 @@ message(" RDMA Transport for Staging: Available")
else()
message(" RDMA Transport for Staging: Unconfigured")
endif()

24 changes: 24 additions & 0 deletions cmake/DetectOptions.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -573,6 +573,30 @@ if(AWSSDK_FOUND)
set(ADIOS2_HAVE_AWSSDK TRUE)
endif()

# Campaign Management
if(ADIOS2_USE_Campaign STREQUAL AUTO)
find_package(SQLite3)
if (SQLite3_FOUND)
find_package(ZLIB)
endif()
if (SQLite3_FOUND AND ZLIB_FOUND)
message(STATUS "Sqlite3 and ZLIB found. Turn on Campaign Management")
set(ADIOS2_HAVE_Campaign TRUE)
endif()
elseif(ADIOS2_USE_Campaign)
find_package(SQLite3 REQUIRED)
if (SQLite3_FOUND)
find_package(ZLIB REQUIRED)
endif()
if (SQLite3_FOUND AND ZLIB_FOUND)
message(STATUS "Sqlite3 and ZLIB found. Turn on Campaign Management")
set(ADIOS2_HAVE_Campaign TRUE)
endif()
endif()

# Multithreading
find_package(Threads REQUIRED)

# Floating point detection
include(CheckTypeRepresentation)

Expand Down
1 change: 1 addition & 0 deletions examples/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,4 @@ add_subdirectory(hello)
add_subdirectory(plugins)
add_subdirectory(simulations)
add_subdirectory(useCases)
add_subdirectory(campaign)
9 changes: 9 additions & 0 deletions examples/campaign/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
#------------------------------------------------------------------------------#
# Distributed under the OSI-approved Apache License, Version 2.0. See
# accompanying file Copyright.txt for details.
#------------------------------------------------------------------------------#

if(ADIOS2_HAVE_MPI)
add_executable(adios2_campaign_write_mpi campaign_write.cpp)
target_link_libraries(adios2_campaign_write_mpi adios2::cxx11_mpi adios2_core MPI::MPI_C)
endif()
Loading
Loading