Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom archive mode #627

Merged
merged 76 commits into from
Jun 5, 2021
Merged
Changes from 1 commit
Commits
Show all changes
76 commits
Select commit Hold shift + click to select a range
fbca19a
local/remote switch for rsync
jkbecker Apr 2, 2021
a39f39a
fix local rsync dest
jkbecker Apr 2, 2021
fc7ea41
add notes to config.yaml
jkbecker Apr 2, 2021
f3281d6
Merge branch 'development' of github.com:ericaltendorf/plotman into a…
jkbecker Apr 2, 2021
ad86f13
Merge branch 'setup' into archivelocal
jkbecker Apr 4, 2021
715267b
Merge branch 'setup' of github.com:jkbecker/plotman into archivelocal
jkbecker Apr 5, 2021
6882e10
Merge branch 'development' of github.com:jkbecker/plotman into archiv…
jkbecker Apr 7, 2021
829661c
Update config.yaml
jkbecker Apr 7, 2021
aae3654
Update src/plotman/archive.py
jkbecker Apr 7, 2021
9f205df
Merge branch 'development' of github.com:ericaltendorf/plotman into a…
jkbecker Apr 28, 2021
c570f7c
Merge branch 'archivelocal' of github.com:jkbecker/plotman into archi…
jkbecker Apr 28, 2021
c7d8ce2
adapt config template
jkbecker Apr 28, 2021
68b683f
adapt error message
jkbecker Apr 28, 2021
9d5ae8a
doc fix
jkbecker Apr 28, 2021
2d917fa
whitespace, defaults
jkbecker Apr 28, 2021
43ead3d
Update src/plotman/configuration.py
jkbecker Apr 28, 2021
4aee539
Merge branch 'development' into archivelocal
altendky Apr 28, 2021
194dc6d
Update src/plotman/configuration.py
jkbecker Apr 29, 2021
c5d5ad6
Update src/plotman/archive.py
jkbecker Apr 29, 2021
59de0c6
Update src/plotman/archive.py
altendky Apr 29, 2021
7f15c43
Merge branch 'development' of github.com:ericaltendorf/plotman into a…
jkbecker May 8, 2021
11d06a7
new config file format supporting local archive mode
jkbecker May 8, 2021
6df447e
prepare for legacy vs extendable new configs
jkbecker May 8, 2021
b97c385
merging
jkbecker May 8, 2021
4c92cb1
better example value for local mode
jkbecker May 8, 2021
57ebfe5
some archive tests
jkbecker May 8, 2021
160b3f9
rsync_dest -> arch_dest refactoring
jkbecker May 8, 2021
fff7c22
Merge branch 'development' into archivelocal
altendky May 17, 2021
12a8214
custom archive
altendky May 19, 2021
c3737ee
less debuggy
altendky May 19, 2021
8978b4d
locally detects free space and transfers
altendky May 19, 2021
82b3684
rsync argument tweaks
altendky May 19, 2021
33e2760
skip malformed df-alike output
altendky May 21, 2021
239892a
Merge branch 'development' into custom_archive
altendky May 21, 2021
73fe689
switch to scripts
altendky May 22, 2021
7480d2d
correct default scripts
altendky May 22, 2021
a35959c
add a little config example
altendky May 22, 2021
28d434f
Merge branch 'development' into custom_archive
altendky May 22, 2021
5866683
just break archiving and make it all new
altendky May 22, 2021
141ba38
less path
altendky May 22, 2021
6e291bb
fix archive prefix identification
altendky May 23, 2021
3c8ead9
add preliminary explanation and examples in config
altendky May 23, 2021
0a1e96e
misc
altendky May 23, 2021
f24a372
fixup for archive subcommand as well
altendky May 23, 2021
68db1f2
links to wiki for configuration versions
altendky May 23, 2021
cc25e05
minor cleanup
altendky May 24, 2021
de73ed4
cleanup temp script files
altendky May 24, 2021
2519534
--preallocate --whole-file
altendky May 27, 2021
41d9033
correct indentation in example config
altendky May 27, 2021
7ad3405
explain disk space script output format
altendky May 27, 2021
d512f6b
lots of logging from the disk script
altendky May 27, 2021
8fa5f37
fixup
altendky May 27, 2021
816f430
slight refactor for reporting
altendky May 27, 2021
87cbdb8
less recalculation in get_running_archive_jobs()
altendky May 27, 2021
cf12e4c
hold multiple archival target definitions
altendky May 28, 2021
efbe6d0
add env with defaults and mandatories for target definitions
altendky May 28, 2021
22ce23f
add preset archiving target definitions
altendky May 28, 2021
4a4a4f3
log archive transfer activities
altendky May 30, 2021
d12ffd4
Merge branch 'development' into custom_archive
altendky May 30, 2021
50fcceb
Apply suggestions from code review
altendky May 30, 2021
c87f6e8
add set -evx to archival presets
altendky May 30, 2021
1132260
archive -> disk space in log message
altendky May 30, 2021
7ab1523
be agnostic to trailing /s on site_root (again)
altendky May 30, 2021
2d87e20
add logging (for disk space script output)
altendky May 30, 2021
2461304
allow integers as env values to avoid user confusion specifying ports
altendky May 31, 2021
4001446
Merge branch 'development' into custom_archive
altendky May 31, 2021
d7aca71
top level logging: and plots/transfers/application log path configura…
altendky May 31, 2021
33b3b97
tidy
altendky May 31, 2021
c070605
shift some IO up a layer
altendky May 31, 2021
77c5ec1
a couple more cfg.logging.plots
altendky May 31, 2021
ab0614b
Merge branch 'interactive-noauto-start' into custom_archive
altendky Jun 1, 2021
af155b2
Merge branch 'interactive-noauto-start' into custom_archive
altendky Jun 1, 2021
3abc9d2
Merge branch 'interactive-noauto-start' into custom_archive
altendky Jun 1, 2021
175aebc
Merge branch 'development' into custom_archive
altendky Jun 1, 2021
2e27644
handle some None outputs
altendky Jun 2, 2021
d680322
printf for mawk to avoid scientific notation etc
altendky Jun 3, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
--preallocate --whole-file
  • Loading branch information
altendky committed May 27, 2021
commit 25195340c4c22aca72f2449e52f289acec7d06a4
4 changes: 2 additions & 2 deletions src/plotman/resources/plotman.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ directories:
df -BK | grep " ${site_root}/" | awk '{ gsub(/K$/,"",$4); print $6 ":" $4*1024 }'
altendky marked this conversation as resolved.
Show resolved Hide resolved
transfer_script: |
#!/bin/bash
"${command}" --skip-compress plot --remove-source-files --inplace "${source}" "${destination}"
"${command}" --inplace --preallocate --remove-source-files --skip-compress plot --whole-file "${source}" "${destination}"
transfer_process_name: "{command}"
transfer_process_argument_prefix: "{site_root}"

Expand All @@ -113,7 +113,7 @@ directories:
# transfer_script: |
# #!/bin/bash
# relative_path=$(realpath --relative-to="${site_root}" "${destination}")
# "${command}" --bwlimit=80000 --skip-compress plot --remove-source-files --inplace "${source}" "${url_root}/${relative_path}"
# "${command}" --bwlimit=80000 --inplace --preallocate --remove-source-files --skip-compress plot --whole-file "${source}" "${url_root}/${relative_path}"
# transfer_process_name: "{command}"
# transfer_process_argument_prefix: "{url_root}"

Expand Down