More efficient rsync parameters for Archive

[Handri-Kosada]

It seems this is way more efficient for Rsync large files parameters for Archive

rsync --bwlimit=80000 -av --partial --inplace --append --progress --remove-source-files /path/source/plots rsync://host:12000/sites/003/

Tested in my local.
source : https://fedoramagazine.org/copying-large-files-with-rsync-and-some-misconceptions/

[ericaltendorf]

We should definitely make archiving so it just calls out to a command specified in the config, so people can use whatever commands and options they want.

…

On Mon, Apr 12, 2021, 21:31 Handri Kosada ***@***.***> wrote: It seems this is way more efficient for Rsync large files parameters for Archive rsync --bwlimit=80000 -av --partial --inplace --append --progress --remove-source-files /path/source/plots rsync://host:12000/sites/003/ — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#110>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ARPZIFRU72GN4RNABRZRCK3TIPCK7ANCNFSM422RNZAA> .

[13thProgression]

If this ticket evolves to allowing the user to specify their archive settings, please make sure that the interface allows for requesting multiple archive events at the same time in parallel to different targets. (Should be similar to how dst scheduling works, but with multiple remote targets or multiple drives on a single remote target.)

[graemes]

These settings work well for resyncing large files (such as VM images) but provide little/no benefit to our use case in terms of transfer speed over the current settings imo.
They also have the downside of making it appear to the harvester that the plot is being copied but is not available (the current setting rsync to a .plot file which is simply renamed once the transfer is complete and becomes visible to the harvester as it is now complete).
I do agree that abstracting out the archiving mechanism is a good thing though :-)

[altendky]

Archiving has been significantly reworked and that has been merged to the development branch. https://github.com/ericaltendorf/plotman/wiki/Archiving links out to an updated version in a gist until it is released.