Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Campaign manager #3932

Merged
merged 63 commits into from
Dec 15, 2023
Merged

Campaign manager #3932

merged 63 commits into from
Dec 15, 2023

Commits on Aug 8, 2023

  1. Added CampaignManager to be created by ADIOS object. It does nothing …

    …at this point. Added CampaignReader engine as a skeleton reader.
    pnorbert committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    60976c3 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    45b0892 View commit details
    Browse the repository at this point in the history
  3. Added example to write many different files from different communicat…

    …ors and different ADIOS objects.
    pnorbert committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    6825d94 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    b085e37 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    08a0715 View commit details
    Browse the repository at this point in the history
  6. add missing file

    pnorbert committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    fdfa1d6 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    f45262d View commit details
    Browse the repository at this point in the history
  8. save work: campaing reader copies variables from subengines, bpls is …

    …stuck on empty m_AvailableStepBlockIndexOffsets
    pnorbert committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    957ed2f View commit details
    Browse the repository at this point in the history
  9. Maintain a map for each variable (original variable, engine and IO) s…

    …o that we can use the actual engine and IO functions. bpls works properly showing dimensions now. Min/max is missing. Reading is not supported. BeginStep/EndStep not supported.
    pnorbert committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    010beb7 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    bb4fed9 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    af3916e View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    68c65d9 View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    5885c48 View commit details
    Browse the repository at this point in the history
  14. don't use BP3

    pnorbert committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    d89557b View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    c48d22b View commit details
    Browse the repository at this point in the history
  16. Configuration menu
    Copy the full SHA
    6ad3345 View commit details
    Browse the repository at this point in the history
  17. Configuration menu
    Copy the full SHA
    192543e View commit details
    Browse the repository at this point in the history
  18. Configuration menu
    Copy the full SHA
    c43bbf8 View commit details
    Browse the repository at this point in the history
  19. Configuration menu
    Copy the full SHA
    d0c52e9 View commit details
    Browse the repository at this point in the history
  20. Configuration menu
    Copy the full SHA
    875a7ff View commit details
    Browse the repository at this point in the history
  21. CampaignReader processes sqlite3 database. Able to open local files (…

    …based on hostname match)
    pnorbert committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    0e35446 View commit details
    Browse the repository at this point in the history
  22. Configuration menu
    Copy the full SHA
    2f482ae View commit details
    Browse the repository at this point in the history
  23. Configuration menu
    Copy the full SHA
    bb16f6a View commit details
    Browse the repository at this point in the history
  24. add missing file

    pnorbert committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    a1732a7 View commit details
    Browse the repository at this point in the history
  25. add another missing file

    pnorbert committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    14225cd View commit details
    Browse the repository at this point in the history
  26. Configuration menu
    Copy the full SHA
    f5877bf View commit details
    Browse the repository at this point in the history
  27. fix print

    pnorbert committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    026f915 View commit details
    Browse the repository at this point in the history
  28. unfix print

    pnorbert committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    6f53bb2 View commit details
    Browse the repository at this point in the history
  29. add info command

    pnorbert committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    a3eba6d View commit details
    Browse the repository at this point in the history
  30. Configuration menu
    Copy the full SHA
    0730209 View commit details
    Browse the repository at this point in the history
  31. Configuration menu
    Copy the full SHA
    334eeca View commit details
    Browse the repository at this point in the history
  32. Configuration menu
    Copy the full SHA
    6049c7a View commit details
    Browse the repository at this point in the history
  33. do not convert all parameter values to lower case. Full paths may con…

    …tain upper case for a reason.
    pnorbert committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    8d2e617 View commit details
    Browse the repository at this point in the history
  34. Use RemoteDataPath parameter for BP5 to open remote data files. clang…

    …-format campaign source files.
    pnorbert committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    ffcb09b View commit details
    Browse the repository at this point in the history
  35. remove debug prints

    pnorbert committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    86bc688 View commit details
    Browse the repository at this point in the history
  36. Configuration menu
    Copy the full SHA
    ad9f23c View commit details
    Browse the repository at this point in the history
  37. reformat

    pnorbert committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    3af1909 View commit details
    Browse the repository at this point in the history
  38. \\. for \.

    pnorbert committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    c01ace4 View commit details
    Browse the repository at this point in the history
  39. Configuration menu
    Copy the full SHA
    8eba0dc View commit details
    Browse the repository at this point in the history

Commits on Sep 1, 2023

  1. Merge branch 'master' into campaign-manager

    * master:
      Remove unused SelectionType values
      Update ADIOS2 HDF5 VOL with basic set of capability flags
      removed  a comment
      ci: disable MGARD static build
      cmake: fix ffs dependency
      removed comments and cleaned up more code
      bp5: remove ADIOS2_USE_BP5 option
      operators: fix module library
      ci: Create static minimal build
      cmake: correct prefer_shared_blosc behavior
      removing warning from auto
      clang-format
      match type of timestep for h5 engine to size_t (same as adios VariableBase class) when storing to hdf5, check the limit of size_t and match to the right type (uint, ulong, default ulonglong)
      BP5Deserialize: modify changes to keep abi compt
      Rename test. fix windows compile errors.
      add codeql workflow
      Fix ChunkV maintaining CurOffset when downsizing current chunk in Allocate() and creating the next chunk.
      ci: fix docker images names
      removed kwsys gitattributes
      KWSys 2023-05-25 (c9f0da47)
      fixup! ci: add mgard dependency to spack builds
      ci: remove unused script part
      cmake: remove redundant pugixml dependency
      ci: add mgard dependency to spack builds
      cmake: Remove enet config warning
      cmake: resolve cmake python deprecation
      enet 2023-08-15 (7587eb15)
      EVPath 2023-08-15 (657c7fa4)
      dill 2023-08-15 (235dadf0)
      atl 2023-08-15 (7286dd11)
      remove data.py since BP5 data file has no format
      format
      bp5dbg parse records and check sizes during it for mmd.0 and md.0. No decoding FFS records.
      cmake: correct info.h installation path
      ffs 2023-08-14 (2f3fe98d)
      CI: add two static builds
      CI: use full image tag
      Organize the processes into groups so that the two steps of metadata aggregation has more or less the same number or participants. This replaces in-node aggregation in first step. The new strategy balances the size of metadata gathered in the two steps.
      CI: added blosc2 dependency to Spack build
      cmake: Add Blosc2 2.10.1 compatibility.
      readthedocs: build current ADIOS2
      cmake,readthedocs: update python enable instructions
      Update ReadMe.md
    pnorbert committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    539ba34 View commit details
    Browse the repository at this point in the history
  2. fixes for compiling

    pnorbert committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    1208433 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    3af9b55 View commit details
    Browse the repository at this point in the history
  4. fix comm name

    pnorbert committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    0a3e0bb View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    934ae46 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    c6ae903 View commit details
    Browse the repository at this point in the history

Commits on Nov 17, 2023

  1. Merge branch 'master' into campaign-manager

    * master: (126 commits)
      ReadMe.md: Mention 2.9.2 release
      Cleanup server output a bit (ornladios#3914)
      ci: set openmpi and openmp params
      Example using Kokkos buffers with SST
      Changes to MallocV to take into consideration the memory space of a variable
      Change install directory of Gray scott files again
      ci,crusher: increase supported num branches
      ci: add shellcheck coverage to source and testing
      Change install directory of Gray scott files
      Only rank 0 should print the initialization message in perfstub
      Defining and computing derived variables (ornladios#3816)
      Add Remote "-status" command to see if a server is running and where (ornladios#3911)
      examples,hip: use find_package(hip) once in proj
      Add Steps Tutorial
      Add Operators Tutorial
      Add Attributes Tutorial
      Add Variables Tutorial
      Add Hello World Tutorial
      Add Tutorials' Download and Build section
      Add Tutorials' Overview section
      Improve bpStepsWriteRead* examples
      Rename bpSZ to bpOperatorSZWriter
      Convert bpAttributeWriter to bpAttributeWriteRead
      Improve bpWriter/bpReader examples
      Close file after reading for hello-world.py
      Fix names of functions in engine
      Fix formatting warnings
      Add dataspaces.rst in the list of engines
      Add query.rst
      cmake: find threads package first
      docs: update new_release.md
      Bump version to v2.9.2
      ci: update number of task for mpich build
      clang-format: Correct format to old style
      Merge pull request ornladios#3878 from anagainaru/test-null-blocks
      Merge pull request ornladios#3588 from vicentebolea/fix-mpi-dp
      Adding tests for writing null blocks with and without compression
      bp5: make RecMap an static anon namespaced var
      Replace LookupWriterRec's linear search on RecList with an unordered_map. For 250k variables, time goes from 21sec to ~1sec in WSL. The order of entries in RecList was not necessary for the serializer to work correctly.
      Replace LookupWriterRec's linear search on RecList with an unordered_map. For 250k variables, time goes from 21sec to ~1sec in WSL. The order of entries in RecList was not necessary for the serializer to work correctly. (ornladios#3877)
      Fix data length calculation for hash (ornladios#3875)
      Merge pull request ornladios#3823 from eisenhauer/SstMemSel
      Merge pull request ornladios#3805 from pnorbert/fix-bpls-string-scalar
      Merge pull request ornladios#3804 from pnorbert/fix-aws-version
      Merge pull request ornladios#3759 from pnorbert/bp5dbg-metadata
      new attempt to commit query support of local array. (ornladios#3868)
      MPI::MPI_Fortran should be INTERFACE not PUBLIC
      Fix hip example compilation error (ornladios#3865)
      Server Improvements (ornladios#3862)
      ascent,ci: remove unshallow flag
      Remove Slack as a contact mechanism (ornladios#3866)
      bug fix:  syntax error in json  output (ornladios#3857)
      Update the bpWriterReadHip example's cmake to run on crusher
      Examples: Use BPFile instead of BP3/4/5 for future-proof
      inlineMWE example: Close files at the end
      Examples: Add BeginStep/EndStep wherever it was missing
      BP5Serializer: handle local variables that use operators (ornladios#3859)
      gha,ci: update checkout to v4
      Blosc2 USE ON: Fix Module Fallback
      cmake: correct prefer_shared_blosc behavior
      cmake: correct info.h installation path
      ci: disable MGARD static build
      operators: fix module library
      ci: add downloads readthedocs
      cmake: Add Blosc2 2.10.1 compatibility.
      Blosc2 USE ON: Fix Module Fallback (ornladios#3774)
      Fix destdir install test (ornladios#3850)
      cmake: update minimum cmake to 3.12 (ornladios#3849)
      MPI: add timeout for conf test for MPI_DP (ornladios#3848)
      MPI_DP: do not call MPI_Init (ornladios#3847)
      install: export adios2 device variables (ornladios#3819)
      Merge pull request ornladios#3799 from vicentebolea/support-new-yaml-cpp
      Merge pull request ornladios#3737 from vicentebolea/fix-evpath-plugins-path
      SST,MPI,DP: soft handle peer error
      SST,MPI,DP: improve uniq identifier
      Fix destdir install test (ornladios#3850)
      cmake: include ctest before detectoptions
      ci: enable tau check
      Add/Improve the ReadMe.md files in examples directory
      Disable BUILD_TESTING and ADIOS2_BUILD_EXAMPLES by default
      Remove testing based on ADIOS2-examples
      Fix formatting issue in DetectOptions.cmake
      Add examples from ADIOS2-Examples
      Improve existing examples
      MPI_DP: do not call MPI_Init (ornladios#3847)
      cmake: update minimum cmake to 3.12 (ornladios#3849)
      MPI: add timeout for conf test for MPI_DP (ornladios#3848)
      Tweak Remote class and test multi-threaded file remote access (ornladios#3834)
      Add prototype testing of remote functionality (ornladios#3830)
      Try always using the MPI version
      Try always using the MPI version
      Import tests from bp to staging common, implement memory selection in SST
      ci: fix codeql ignore path (ornladios#3772)
      install: export adios2 device variables (ornladios#3819)
      added support to query BP5 files (ornladios#3809)
      Partial FFS Upstream, only changes to type_id
      ffs 2023-09-19 (67e411c0)
      Fix abs/rel step in BP5 DoCount
      fix dummy Win build
      Pass Array Order of reader to remote server for proper Get() operation
      ...
    pnorbert committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    cc2f83a View commit details
    Browse the repository at this point in the history
  2. remove merge conflict line

    pnorbert committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    e26b736 View commit details
    Browse the repository at this point in the history

Commits on Nov 20, 2023

  1. Configuration menu
    Copy the full SHA
    a6a13c1 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    986eecd View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    0532696 View commit details
    Browse the repository at this point in the history
  4. Merge branch 'master' into campaign-manager

    * master:
      Example using DataMan with Kokkos buffers
      Propagating the GPU logic inside the DataMan engine
      ci: Use mpich built with ch3:sock:tp for faster tests
    pnorbert committed Nov 20, 2023
    Configuration menu
    Copy the full SHA
    e650a74 View commit details
    Browse the repository at this point in the history

Commits on Nov 21, 2023

  1. Added CampaignManager to be created by ADIOS object. Added CampaignRe…

    …ader engine.
    
    - convert info to json at Close() and write out.
    - add example to write many different files from different communicators and different ADIOS objects.
    - add explicit map for which engine supports ReadRandomAccess mode
    - save campaign data (in json) in adios-campaign/
    - use SQLite3 database for campaign archive, compresses metadata
    - CampaignReader uses SQLite3
    - check ctime of individual files and save from DB only if it is older
    - enable using multiple directories in same host
    - new parameter RemoteDataPath for BP5 engine, and Campaign Engine controls this parameter when opening a remote file (with metadata files locally present in another (local) path)
    pnorbert committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    ab9205c View commit details
    Browse the repository at this point in the history
  2. add back deleted line

    pnorbert committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    e8b561b View commit details
    Browse the repository at this point in the history
  3. Merge branch 'master' into campaign-manager

    * master:
      CI Contract: Build examples with external ADIOS
    pnorbert committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    434f7a7 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    3d50e73 View commit details
    Browse the repository at this point in the history
  5. format

    pnorbert committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    79f4463 View commit details
    Browse the repository at this point in the history
  6. attempt to fix windows build

    pnorbert committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    17f4398 View commit details
    Browse the repository at this point in the history
  7. attempt to fix linux build

    pnorbert committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    92f5bd6 View commit details
    Browse the repository at this point in the history

Commits on Dec 12, 2023

  1. Merge branch 'master' into campaign-manager

    * master:
      Have HDF5 write raise error if operator(s) requested (ornladios#3951)
      fix for ASAN issue related to JoinedDimArray handling in BP5 deserializer (ornladios#3963)
      New operator MDR, for refactoring floating point arrays using MGARD's new MDR extension. (ornladios#3826)
      restricted http transport from windows builds.
      XMLConfigTest: Add RemoveIO test
      adios2::core::ADIOS: Initialize new IO objects with config file
      removed unsused variable
      Update readme for heat transfer example with new location and build instructions
      Ignore tests with defects for now
      Adapt libfabric dataplane of SST to Cray CXI provider (ornladios#3672)
      ci: fix path to lsan suppressions, fix broken gh status post
      Use adios2_mode_readRandomAccess in matlab open to make it work for BP5 (ornladios#3956)
      Add Global Array Capabilities and Limitations
      Add Section for Anatomy of an ADIOS Program
      Enable Shell-Check for gh-actions scripts
      Enable Shell-Check for circle CI scripts
      Enable Shell-Check for tau contract scripts
      Enable Shell-Check for scorpio contract scripts
      Enable Shell-Check for lammps contract scripts
      Delete VTK code in examples
      Fix MATLAB bindings for MacOS (ornladios#3950)
      Set the compiler for the Kokkos DataMan example to what is used to build Kokkos
      Fix the HIP architecture CMAKE variable (ornladios#3931)
      perfstubs 2023-11-27 (845d0702) (ornladios#3944)
      Revert "Only rank 0 should print the initialization message in perfstub"
      Formatting
      Formatting
      Revision
      Added buffered data receive in the client side.
      A socket version of HTTP connector. Proxy server host is hardwired to "localhost" and port to 9999 Remote bpls: bpls -E bp4 -T "Library=HTTP" /remote_path/myVector_cpp.bp -d bpInts
    pnorbert committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    4d33f87 View commit details
    Browse the repository at this point in the history
  2. Compile CampaignManager.cpp only if ADIOS2_HAVE_CAMPAIGN, otherwise u…

    …se empty functions for campaign recorder.
    pnorbert committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    b14d43e View commit details
    Browse the repository at this point in the history
  3. add the Campaign writer as example. This produces several files that …

    …can be added to a single campaign. It also supports restarting the application and adding more steps, so that a campaign archive can be updated later.
    pnorbert committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    358b121 View commit details
    Browse the repository at this point in the history
  4. fix formatting

    pnorbert committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    2652035 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    4f7367a View commit details
    Browse the repository at this point in the history