-
Notifications
You must be signed in to change notification settings - Fork 279
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
rsync --skip-compress plot #565
Conversation
@jameswood, any opinions? I noticed that I was getting low throughput and 100% CPU usage with |
Mm... maybe this doesn't cut it. It's sometimes fast and sometimes still 100% load and slow. I'll poke some more. |
Hey Kyle, I don't really have any input on this one, other than to say that the Mac says I haven't seen the issues you mentioned, but I have had one Mac that seems to crash during-or-after phase 4 and I'm not sure why. Possibly related to the rsync task?? It was a pretty old Mac Mini that was taking over a day to do a plot anyway, so it has been retired and replaced with a Ryzen. |
Maybe |
Ubuntu server:
macOS 11.2.3: |
I did some testing... Slow transfers only happened if rsync grabs an existing file and tries to resume that. Inside of |
@jameswood, could you share the full output of But yeah, I do think at this point that the partial stuff isn't useful. That's great when both sides can read a pile of data off the disk faster than they can transfer it, but that's not significantly the case here. I think it's a bit more robust and simple to just restart the transfer if it fails. Anyways, elsewhere we are going to get arbitrarily configurable options so everyone can pick themselves. |
@altendky I'm pretty sure it's the standard system-supplied rsync, it's at |
|
Ahhhh that is OLDE |
Alrighty, thanks for that. Though, as mentioned, it will soon only matter for the sake of providing reference configurations for people as we'll have complete control on each system. For completeness, here's mine.
|
Would it be a good idea for the Mac installation instructions could mention that the factory-default rsync is old, do you think? |
Like for the upcoming custom configuration? Yeah, that would be a valuable note. I'm not sure it is important for the present setup though? |
I have experienced if I interrupt last archive and leave an old plot file in archive folder, next time run archive will go in stuck. |
--inplace doesn't use a temp file so if your destination is a farming directory chia will mark it bad and wont re-read until you restart the farmer. Unless you have an unreliable connection, don't add the -P or --partial flags. If you try to resume that transfer you'll have to hash the partial file before starting transfers anyway so you may not actually save any time but you will hit the CPU on the destination. |
I think thats only partially correct. The file gets detected as too small to be a plot and wont be added until the next cycle of directory scan. Shouldnt be a problem, but with the tempfile you can find broken uploads more easily, and it changes nothing performancy-wise. |
Go ahead and test it for yourself, I did. Chia adds the file as corrupt and doesn't re-read. Whatever test that might be done on filesystem doesn't work for me. |
I'm going to throw this out since #627 is nearing completion. You can look in |
No description provided.