-
Notifications
You must be signed in to change notification settings - Fork 126
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Campaign manager #3932
Campaign manager #3932
Commits on Aug 8, 2023
-
Added CampaignManager to be created by ADIOS object. It does nothing …
…at this point. Added CampaignReader engine as a skeleton reader.
Configuration menu - View commit details
-
Copy full SHA for 60976c3 - Browse repository at this point
Copy the full SHA 60976c3View commit details -
Configuration menu - View commit details
-
Copy full SHA for 45b0892 - Browse repository at this point
Copy the full SHA 45b0892View commit details -
Added example to write many different files from different communicat…
…ors and different ADIOS objects.
Configuration menu - View commit details
-
Copy full SHA for 6825d94 - Browse repository at this point
Copy the full SHA 6825d94View commit details -
Configuration menu - View commit details
-
Copy full SHA for b085e37 - Browse repository at this point
Copy the full SHA b085e37View commit details -
Configuration menu - View commit details
-
Copy full SHA for 08a0715 - Browse repository at this point
Copy the full SHA 08a0715View commit details -
Configuration menu - View commit details
-
Copy full SHA for fdfa1d6 - Browse repository at this point
Copy the full SHA fdfa1d6View commit details -
Configuration menu - View commit details
-
Copy full SHA for f45262d - Browse repository at this point
Copy the full SHA f45262dView commit details -
save work: campaing reader copies variables from subengines, bpls is …
…stuck on empty m_AvailableStepBlockIndexOffsets
Configuration menu - View commit details
-
Copy full SHA for 957ed2f - Browse repository at this point
Copy the full SHA 957ed2fView commit details -
Maintain a map for each variable (original variable, engine and IO) s…
…o that we can use the actual engine and IO functions. bpls works properly showing dimensions now. Min/max is missing. Reading is not supported. BeginStep/EndStep not supported.
Configuration menu - View commit details
-
Copy full SHA for 010beb7 - Browse repository at this point
Copy the full SHA 010beb7View commit details -
Configuration menu - View commit details
-
Copy full SHA for bb4fed9 - Browse repository at this point
Copy the full SHA bb4fed9View commit details -
Configuration menu - View commit details
-
Copy full SHA for af3916e - Browse repository at this point
Copy the full SHA af3916eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 68c65d9 - Browse repository at this point
Copy the full SHA 68c65d9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5885c48 - Browse repository at this point
Copy the full SHA 5885c48View commit details -
Configuration menu - View commit details
-
Copy full SHA for d89557b - Browse repository at this point
Copy the full SHA d89557bView commit details -
Configuration menu - View commit details
-
Copy full SHA for c48d22b - Browse repository at this point
Copy the full SHA c48d22bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 6ad3345 - Browse repository at this point
Copy the full SHA 6ad3345View commit details -
Configuration menu - View commit details
-
Copy full SHA for 192543e - Browse repository at this point
Copy the full SHA 192543eView commit details -
Configuration menu - View commit details
-
Copy full SHA for c43bbf8 - Browse repository at this point
Copy the full SHA c43bbf8View commit details -
functions to get the Fully Qualified Domain Name, and a cluster name …
…parsed from FQDN
Configuration menu - View commit details
-
Copy full SHA for d0c52e9 - Browse repository at this point
Copy the full SHA d0c52e9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 875a7ff - Browse repository at this point
Copy the full SHA 875a7ffView commit details -
CampaignReader processes sqlite3 database. Able to open local files (…
…based on hostname match)
Configuration menu - View commit details
-
Copy full SHA for 0e35446 - Browse repository at this point
Copy the full SHA 0e35446View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2f482ae - Browse repository at this point
Copy the full SHA 2f482aeView commit details -
Configuration menu - View commit details
-
Copy full SHA for bb16f6a - Browse repository at this point
Copy the full SHA bb16f6aView commit details -
Configuration menu - View commit details
-
Copy full SHA for a1732a7 - Browse repository at this point
Copy the full SHA a1732a7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 14225cd - Browse repository at this point
Copy the full SHA 14225cdView commit details -
Configuration menu - View commit details
-
Copy full SHA for f5877bf - Browse repository at this point
Copy the full SHA f5877bfView commit details -
Configuration menu - View commit details
-
Copy full SHA for 026f915 - Browse repository at this point
Copy the full SHA 026f915View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6f53bb2 - Browse repository at this point
Copy the full SHA 6f53bb2View commit details -
Configuration menu - View commit details
-
Copy full SHA for a3eba6d - Browse repository at this point
Copy the full SHA a3eba6dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0730209 - Browse repository at this point
Copy the full SHA 0730209View commit details -
Configuration menu - View commit details
-
Copy full SHA for 334eeca - Browse repository at this point
Copy the full SHA 334eecaView commit details -
Configuration menu - View commit details
-
Copy full SHA for 6049c7a - Browse repository at this point
Copy the full SHA 6049c7aView commit details -
do not convert all parameter values to lower case. Full paths may con…
…tain upper case for a reason.
Configuration menu - View commit details
-
Copy full SHA for 8d2e617 - Browse repository at this point
Copy the full SHA 8d2e617View commit details -
Use RemoteDataPath parameter for BP5 to open remote data files. clang…
…-format campaign source files.
Configuration menu - View commit details
-
Copy full SHA for ffcb09b - Browse repository at this point
Copy the full SHA ffcb09bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 86bc688 - Browse repository at this point
Copy the full SHA 86bc688View commit details -
Configuration menu - View commit details
-
Copy full SHA for ad9f23c - Browse repository at this point
Copy the full SHA ad9f23cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 3af1909 - Browse repository at this point
Copy the full SHA 3af1909View commit details -
Configuration menu - View commit details
-
Copy full SHA for c01ace4 - Browse repository at this point
Copy the full SHA c01ace4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8eba0dc - Browse repository at this point
Copy the full SHA 8eba0dcView commit details
Commits on Sep 1, 2023
-
Merge branch 'master' into campaign-manager
* master: Remove unused SelectionType values Update ADIOS2 HDF5 VOL with basic set of capability flags removed a comment ci: disable MGARD static build cmake: fix ffs dependency removed comments and cleaned up more code bp5: remove ADIOS2_USE_BP5 option operators: fix module library ci: Create static minimal build cmake: correct prefer_shared_blosc behavior removing warning from auto clang-format match type of timestep for h5 engine to size_t (same as adios VariableBase class) when storing to hdf5, check the limit of size_t and match to the right type (uint, ulong, default ulonglong) BP5Deserialize: modify changes to keep abi compt Rename test. fix windows compile errors. add codeql workflow Fix ChunkV maintaining CurOffset when downsizing current chunk in Allocate() and creating the next chunk. ci: fix docker images names removed kwsys gitattributes KWSys 2023-05-25 (c9f0da47) fixup! ci: add mgard dependency to spack builds ci: remove unused script part cmake: remove redundant pugixml dependency ci: add mgard dependency to spack builds cmake: Remove enet config warning cmake: resolve cmake python deprecation enet 2023-08-15 (7587eb15) EVPath 2023-08-15 (657c7fa4) dill 2023-08-15 (235dadf0) atl 2023-08-15 (7286dd11) remove data.py since BP5 data file has no format format bp5dbg parse records and check sizes during it for mmd.0 and md.0. No decoding FFS records. cmake: correct info.h installation path ffs 2023-08-14 (2f3fe98d) CI: add two static builds CI: use full image tag Organize the processes into groups so that the two steps of metadata aggregation has more or less the same number or participants. This replaces in-node aggregation in first step. The new strategy balances the size of metadata gathered in the two steps. CI: added blosc2 dependency to Spack build cmake: Add Blosc2 2.10.1 compatibility. readthedocs: build current ADIOS2 cmake,readthedocs: update python enable instructions Update ReadMe.md
Configuration menu - View commit details
-
Copy full SHA for 539ba34 - Browse repository at this point
Copy the full SHA 539ba34View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1208433 - Browse repository at this point
Copy the full SHA 1208433View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3af9b55 - Browse repository at this point
Copy the full SHA 3af9b55View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0a3e0bb - Browse repository at this point
Copy the full SHA 0a3e0bbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 934ae46 - Browse repository at this point
Copy the full SHA 934ae46View commit details -
Configuration menu - View commit details
-
Copy full SHA for c6ae903 - Browse repository at this point
Copy the full SHA c6ae903View commit details
Commits on Nov 17, 2023
-
Merge branch 'master' into campaign-manager
* master: (126 commits) ReadMe.md: Mention 2.9.2 release Cleanup server output a bit (ornladios#3914) ci: set openmpi and openmp params Example using Kokkos buffers with SST Changes to MallocV to take into consideration the memory space of a variable Change install directory of Gray scott files again ci,crusher: increase supported num branches ci: add shellcheck coverage to source and testing Change install directory of Gray scott files Only rank 0 should print the initialization message in perfstub Defining and computing derived variables (ornladios#3816) Add Remote "-status" command to see if a server is running and where (ornladios#3911) examples,hip: use find_package(hip) once in proj Add Steps Tutorial Add Operators Tutorial Add Attributes Tutorial Add Variables Tutorial Add Hello World Tutorial Add Tutorials' Download and Build section Add Tutorials' Overview section Improve bpStepsWriteRead* examples Rename bpSZ to bpOperatorSZWriter Convert bpAttributeWriter to bpAttributeWriteRead Improve bpWriter/bpReader examples Close file after reading for hello-world.py Fix names of functions in engine Fix formatting warnings Add dataspaces.rst in the list of engines Add query.rst cmake: find threads package first docs: update new_release.md Bump version to v2.9.2 ci: update number of task for mpich build clang-format: Correct format to old style Merge pull request ornladios#3878 from anagainaru/test-null-blocks Merge pull request ornladios#3588 from vicentebolea/fix-mpi-dp Adding tests for writing null blocks with and without compression bp5: make RecMap an static anon namespaced var Replace LookupWriterRec's linear search on RecList with an unordered_map. For 250k variables, time goes from 21sec to ~1sec in WSL. The order of entries in RecList was not necessary for the serializer to work correctly. Replace LookupWriterRec's linear search on RecList with an unordered_map. For 250k variables, time goes from 21sec to ~1sec in WSL. The order of entries in RecList was not necessary for the serializer to work correctly. (ornladios#3877) Fix data length calculation for hash (ornladios#3875) Merge pull request ornladios#3823 from eisenhauer/SstMemSel Merge pull request ornladios#3805 from pnorbert/fix-bpls-string-scalar Merge pull request ornladios#3804 from pnorbert/fix-aws-version Merge pull request ornladios#3759 from pnorbert/bp5dbg-metadata new attempt to commit query support of local array. (ornladios#3868) MPI::MPI_Fortran should be INTERFACE not PUBLIC Fix hip example compilation error (ornladios#3865) Server Improvements (ornladios#3862) ascent,ci: remove unshallow flag Remove Slack as a contact mechanism (ornladios#3866) bug fix: syntax error in json output (ornladios#3857) Update the bpWriterReadHip example's cmake to run on crusher Examples: Use BPFile instead of BP3/4/5 for future-proof inlineMWE example: Close files at the end Examples: Add BeginStep/EndStep wherever it was missing BP5Serializer: handle local variables that use operators (ornladios#3859) gha,ci: update checkout to v4 Blosc2 USE ON: Fix Module Fallback cmake: correct prefer_shared_blosc behavior cmake: correct info.h installation path ci: disable MGARD static build operators: fix module library ci: add downloads readthedocs cmake: Add Blosc2 2.10.1 compatibility. Blosc2 USE ON: Fix Module Fallback (ornladios#3774) Fix destdir install test (ornladios#3850) cmake: update minimum cmake to 3.12 (ornladios#3849) MPI: add timeout for conf test for MPI_DP (ornladios#3848) MPI_DP: do not call MPI_Init (ornladios#3847) install: export adios2 device variables (ornladios#3819) Merge pull request ornladios#3799 from vicentebolea/support-new-yaml-cpp Merge pull request ornladios#3737 from vicentebolea/fix-evpath-plugins-path SST,MPI,DP: soft handle peer error SST,MPI,DP: improve uniq identifier Fix destdir install test (ornladios#3850) cmake: include ctest before detectoptions ci: enable tau check Add/Improve the ReadMe.md files in examples directory Disable BUILD_TESTING and ADIOS2_BUILD_EXAMPLES by default Remove testing based on ADIOS2-examples Fix formatting issue in DetectOptions.cmake Add examples from ADIOS2-Examples Improve existing examples MPI_DP: do not call MPI_Init (ornladios#3847) cmake: update minimum cmake to 3.12 (ornladios#3849) MPI: add timeout for conf test for MPI_DP (ornladios#3848) Tweak Remote class and test multi-threaded file remote access (ornladios#3834) Add prototype testing of remote functionality (ornladios#3830) Try always using the MPI version Try always using the MPI version Import tests from bp to staging common, implement memory selection in SST ci: fix codeql ignore path (ornladios#3772) install: export adios2 device variables (ornladios#3819) added support to query BP5 files (ornladios#3809) Partial FFS Upstream, only changes to type_id ffs 2023-09-19 (67e411c0) Fix abs/rel step in BP5 DoCount fix dummy Win build Pass Array Order of reader to remote server for proper Get() operation ...
Configuration menu - View commit details
-
Copy full SHA for cc2f83a - Browse repository at this point
Copy the full SHA cc2f83aView commit details -
Configuration menu - View commit details
-
Copy full SHA for e26b736 - Browse repository at this point
Copy the full SHA e26b736View commit details
Commits on Nov 20, 2023
-
Configuration menu - View commit details
-
Copy full SHA for a6a13c1 - Browse repository at this point
Copy the full SHA a6a13c1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 986eecd - Browse repository at this point
Copy the full SHA 986eecdView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0532696 - Browse repository at this point
Copy the full SHA 0532696View commit details -
Merge branch 'master' into campaign-manager
* master: Example using DataMan with Kokkos buffers Propagating the GPU logic inside the DataMan engine ci: Use mpich built with ch3:sock:tp for faster tests
Configuration menu - View commit details
-
Copy full SHA for e650a74 - Browse repository at this point
Copy the full SHA e650a74View commit details
Commits on Nov 21, 2023
-
Added CampaignManager to be created by ADIOS object. Added CampaignRe…
…ader engine. - convert info to json at Close() and write out. - add example to write many different files from different communicators and different ADIOS objects. - add explicit map for which engine supports ReadRandomAccess mode - save campaign data (in json) in adios-campaign/ - use SQLite3 database for campaign archive, compresses metadata - CampaignReader uses SQLite3 - check ctime of individual files and save from DB only if it is older - enable using multiple directories in same host - new parameter RemoteDataPath for BP5 engine, and Campaign Engine controls this parameter when opening a remote file (with metadata files locally present in another (local) path)
Configuration menu - View commit details
-
Copy full SHA for ab9205c - Browse repository at this point
Copy the full SHA ab9205cView commit details -
Configuration menu - View commit details
-
Copy full SHA for e8b561b - Browse repository at this point
Copy the full SHA e8b561bView commit details -
Merge branch 'master' into campaign-manager
* master: CI Contract: Build examples with external ADIOS
Configuration menu - View commit details
-
Copy full SHA for 434f7a7 - Browse repository at this point
Copy the full SHA 434f7a7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3d50e73 - Browse repository at this point
Copy the full SHA 3d50e73View commit details -
Configuration menu - View commit details
-
Copy full SHA for 79f4463 - Browse repository at this point
Copy the full SHA 79f4463View commit details -
Configuration menu - View commit details
-
Copy full SHA for 17f4398 - Browse repository at this point
Copy the full SHA 17f4398View commit details -
Configuration menu - View commit details
-
Copy full SHA for 92f5bd6 - Browse repository at this point
Copy the full SHA 92f5bd6View commit details
Commits on Dec 12, 2023
-
Merge branch 'master' into campaign-manager
* master: Have HDF5 write raise error if operator(s) requested (ornladios#3951) fix for ASAN issue related to JoinedDimArray handling in BP5 deserializer (ornladios#3963) New operator MDR, for refactoring floating point arrays using MGARD's new MDR extension. (ornladios#3826) restricted http transport from windows builds. XMLConfigTest: Add RemoveIO test adios2::core::ADIOS: Initialize new IO objects with config file removed unsused variable Update readme for heat transfer example with new location and build instructions Ignore tests with defects for now Adapt libfabric dataplane of SST to Cray CXI provider (ornladios#3672) ci: fix path to lsan suppressions, fix broken gh status post Use adios2_mode_readRandomAccess in matlab open to make it work for BP5 (ornladios#3956) Add Global Array Capabilities and Limitations Add Section for Anatomy of an ADIOS Program Enable Shell-Check for gh-actions scripts Enable Shell-Check for circle CI scripts Enable Shell-Check for tau contract scripts Enable Shell-Check for scorpio contract scripts Enable Shell-Check for lammps contract scripts Delete VTK code in examples Fix MATLAB bindings for MacOS (ornladios#3950) Set the compiler for the Kokkos DataMan example to what is used to build Kokkos Fix the HIP architecture CMAKE variable (ornladios#3931) perfstubs 2023-11-27 (845d0702) (ornladios#3944) Revert "Only rank 0 should print the initialization message in perfstub" Formatting Formatting Revision Added buffered data receive in the client side. A socket version of HTTP connector. Proxy server host is hardwired to "localhost" and port to 9999 Remote bpls: bpls -E bp4 -T "Library=HTTP" /remote_path/myVector_cpp.bp -d bpInts
Configuration menu - View commit details
-
Copy full SHA for 4d33f87 - Browse repository at this point
Copy the full SHA 4d33f87View commit details -
Compile CampaignManager.cpp only if ADIOS2_HAVE_CAMPAIGN, otherwise u…
…se empty functions for campaign recorder.
Configuration menu - View commit details
-
Copy full SHA for b14d43e - Browse repository at this point
Copy the full SHA b14d43eView commit details -
add the Campaign writer as example. This produces several files that …
…can be added to a single campaign. It also supports restarting the application and adding more steps, so that a campaign archive can be updated later.
Configuration menu - View commit details
-
Copy full SHA for 358b121 - Browse repository at this point
Copy the full SHA 358b121View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2652035 - Browse repository at this point
Copy the full SHA 2652035View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4f7367a - Browse repository at this point
Copy the full SHA 4f7367aView commit details