Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1.49.4 is slow (uploading) vs 1.48 ? #3578

Closed
Rootax opened this issue Sep 30, 2019 · 41 comments
Closed

1.49.4 is slow (uploading) vs 1.48 ? #3578

Rootax opened this issue Sep 30, 2019 · 41 comments
Labels
Milestone

Comments

@Rootax
Copy link

Rootax commented Sep 30, 2019

Hi,

So, i'm under w10 x64, and using rclone => google drive for backups.

I'm not updating rclone often, but today I updated from 1.48 to 1.49.4. And to my surprise, I was limited around 300Mbits/sec uploading to gdrive... Ok, maybe my isp was bottlenecking somewhere (I can upload at 500Mbits/sec).
But, It was strange to me, so, I deleted the same files I just uploaded with 1.49.4, put 1.48 again, and bam, 500mbits. The tests were a few minutes appart, so I don't believe it was a network problem, and it was big files around 10-15gb so speed has time to "settle".

I didn't notice anything weird with 1.49.4, no strange cpu usage, etc... just the "slow" upload speed.

I'm using this command :
C:\rclonewin64\rclone.exe sync --delete-before --verbose --transfers 8 --checkers 8 --contimeout 45s --timeout 75s --retries 3 --low-level-retries 10 --multi-thread-cutoff=100 --multi-thread-streams=8 --drive-chunk-size 256M --fast-list --exclude "Films 4K HDR/" --exclude "Films 4K/" --exclude "cheg/" --exclude "seed/" --stats 2s R:\ gdriveencryptedownapi:backupnas

(I know the mul-thread stuff is not working when uploading)

Sorry for my english :/

@ncw
Copy link
Member

ncw commented Sep 30, 2019

Hmm, can you try 1.49.3 - I had a report that 1.49.4 has a strange interaction with Google drive.

@Rootax
Copy link
Author

Rootax commented Oct 1, 2019

I just tried 1.49.3, and yes, full speed again.

@ncw
Copy link
Member

ncw commented Oct 1, 2019

Any chance you could try these 3 tests?

1.49.4 as released but disable http2

export GODEBUG=http2client=0 # disable HTTP/2 client support

1.49.4 compiled with go1.12

https://pub.rclone.org/rclone-v1.49.4-go1.12.6.zip

1.49.3 compiled with go1.13

https://pub.rclone.org/rclone-v1.49.3-go1.13.zip

See also a possibly relevant thread in the forum: https://forum.rclone.org/t/1-49-4-plex-internal-errors-on-google-drive/12108/

Thank you :-)

@ncw
Copy link
Member

ncw commented Oct 1, 2019

Ah, just realised you are on Windows so those binaries won't work for you... Can you try the first test anyway?

@Rootax
Copy link
Author

Rootax commented Oct 1, 2019

So i've to add -export GODEBUG=http2client=0 to my command line ?

@ncw
Copy link
Member

ncw commented Oct 1, 2019

So i've to add -export GODEBUG=http2client=0 to my command line ?

Sorry that was unix speak... You need to set the environment variable GODEBUG to http2client=0.

You can do this in cmd with this I think

set GODEBUG http2client=0
rclone ...your command...

@Rootax
Copy link
Author

Rootax commented Oct 1, 2019

Ok I did that.

Still not full speed, still limited around 300Mbits/s :/

@ncw
Copy link
Member

ncw commented Oct 1, 2019

Can you try this binary which is rclone 1.49.4 but compiled with go1.12 and see if that helps?

rclone-v1.49.4-go1.12.zip

And also this one which is rclone 1.49.3 but compiled with go1.13

rclone-v1.49.3-go1.13-windows.zip

Thank you :-)

@Rootax
Copy link
Author

Rootax commented Oct 1, 2019

It seems you have found the problem : )
With rclone 1.49.4 go 1.12 i'm at full speed again
With 1.49.3 but go 1.13, same "slower" speed, around 300Mb/sec

Hope it can help !

@ncw
Copy link
Member

ncw commented Oct 1, 2019

It seems you have found the problem : )
With rclone 1.49.4 go 1.12 i'm at full speed again
With 1.49.3 but go 1.13, same "slower" speed, around 300Mb/sec

Thank you very much - very useful testing :-)

@ncw
Copy link
Member

ncw commented Oct 1, 2019

Could you, with the original as released v1.49.4 try this and see if it makes any difference?

set GODEBUG tls13=0

(note that has to be done in the same CMD window as you run the rclone command)

That TLS change of all the changes in the go1.13 release log looks like the most likely to cause a regression.

@Rootax
Copy link
Author

Rootax commented Oct 1, 2019

I'll in a few hours, I'll let you know :)

@Rootax
Copy link
Author

Rootax commented Oct 2, 2019

Aaaah, I don't know if that make any sense. With the 1.49.4 (original) + set GODEBUG tls13=0 , I'm not a full speed, but a little better than 300mb/sec, like around 300/400.

I tried with a "full speed" version to see if it wasn't a isp problem, and no, I'm at full speed with thoses versions...

@ncw
Copy link
Member

ncw commented Oct 2, 2019

OK so that probably isn't the problem....

I really need to reproduce this myself so I can bisect the go source, but when I try it I get identical (but much slower) results...

I think what I'll do is build a 1.49.5 using go1.12 and hope that the go team figure out the problem before I release 1.50!

ncw added a commit that referenced this issue Oct 2, 2019
The go1.13 build has had various problems reported so revert back to
go1.12 for the stable branch.

See: https://forum.rclone.org/t/1-49-4-plex-internal-errors-on-google-drive/12108
Fixes #3578
@ncw
Copy link
Member

ncw commented Oct 2, 2019

OK I've decided to revert the 1.49.x builds back to using go1.12

Here is a proposed build - testing appreciated! This is code identical to 1.49.4 but

  • build with go1.12
  • the rpm generation is fixed

If it tests well I'll turn it into 1.49.5

https://beta.rclone.org/branch/v1.49.4-003-g2c093214-v1.49-fixes-beta/ (uploaded in 15-30 mins)

ncw added a commit that referenced this issue Oct 2, 2019
The go1.13 build has had various problems reported so revert back to
go1.12 for the stable branch.

See: https://forum.rclone.org/t/1-49-4-plex-internal-errors-on-google-drive/12108
Fixes #3578
@ncw
Copy link
Member

ncw commented Oct 2, 2019

Broke that build... try this one

https://beta.rclone.org/branch/v1.49.4-003-gb5ea6af6-v1.49-fixes-beta/ (uploaded in 15-30 mins)

@Rootax
Copy link
Author

Rootax commented Oct 2, 2019

I'll test as soon as I can, thx !

@Rootax
Copy link
Author

Rootax commented Oct 2, 2019

All right, back to full speed with this beta :) I do nothing fancy with rclone, just backuping data to gdrive, so I can't really tell you if it's working without hiccups for other stuff...

@ncw
Copy link
Member

ncw commented Oct 2, 2019

Thank you very much for testing :-)

I'll release that as 1.49.5 once I get a report from my other testers.

@ncw ncw added the bug label Oct 2, 2019
@ncw ncw added this to the v1.50 milestone Oct 2, 2019
@ivandeex
Copy link
Member

ivandeex commented Oct 3, 2019

@Rootax Please try
$env:GODEBUG="http2client=0"; rclone cmd...

@Rootax
Copy link
Author

Rootax commented Oct 3, 2019

@Rootax Please try
$env:GODEBUG="http2client=0"; rclone cmd...

With which version ?

@ivandeex
Copy link
Member

ivandeex commented Oct 3, 2019

Could you, with the original as released v1.49.4 try this and see if it makes any difference?
set GODEBUG tls13=0
(note that has to be done in the same CMD window as you run the rclone command)

I recall windows has two native shells and mean, in case your windows shell is powershell not cmd, the SET syntax would have no effect, use $env:GODEBUG="tls13=0"; rclone cmd... instead.

@Rootax
Copy link
Author

Rootax commented Oct 3, 2019

Well I tried with cmd, so the set syntax was working, right ?

@ncw
Copy link
Member

ncw commented Oct 3, 2019

Well I tried with cmd, so the set syntax was working, right ?

Actually I think I got the syntax wrong, it needs to be this

set GODEBUG=tls13=0
rclone. ..

@Rootax
Copy link
Author

Rootax commented Oct 3, 2019

set GODEBUG=tls13=0

Just did the test : with original 1.49.4, still slower than the "patched" version or older rclone version. A little more unstable (talking about speed) too, like yoyo-ing between 320 and 420 Mb/s. With working version, it's 500 all the way.

I always do the tests with the same number of files, same files, etc.

@ncw
Copy link
Member

ncw commented Oct 3, 2019

Thanks for testing :-)

@ncw
Copy link
Member

ncw commented Oct 5, 2019

I've released a v1.49.5 which you can find at in the normal place which is compiled with go1.12.x

Release announcement: https://forum.rclone.org/t/rclone-1-49-5-release/12166

I need to get to the bottom of this though so would you be willing to do more testing?

The first thing would be to try the latest beta which is compiled with go1.13.x so likely will have the same problem.

If it does then I'll make a version compiled with the unreleased patches for go.

If that doesn't fix it then we'll need to go deeper!

@Rootax
Copy link
Author

Rootax commented Oct 5, 2019

Yeah latest beta has irregular speed too (and is slower) for me.

@ivandeex
Copy link
Member

ivandeex commented Oct 5, 2019

FYI
TLS 1.3 was landed as optional feature in go 1.12 and activates by default in go 1.13.
This affects how golang client and google drive servers do the connection initialization dance, negotiate ciphers and new tls features like 0-RTT, interact with DPI in provider's middle boxes and downgrade if needed. TLS 1.3 also involves renegotiations which happen middle-stream and depending on local DPI quirks in your country may trigger traffic shaping. This will improve with time as providers upgrade equipment.

I believe that crypto/tls in go 1.13 provides some means to disable tls 1.3. The opt-out will be removed in Go 1.14 according to docs.

https://golang.org/doc/go1.12#tls_1_3

https://golang.org/doc/go1.13#tls_1_3

golang/go#30055

golang/go#9671 (comment)

openssl/openssl#7482

@dfunkt
Copy link

dfunkt commented Oct 5, 2019

Unless I misread the message from this commit (golang/go@a8c2e5c) you can't opt-out from TLS 1.3 anymore.

@ncw
Copy link
Member

ncw commented Oct 5, 2019

Yeah latest beta has irregular speed too (and is slower) for me.

Can you try this one.. this is v1.49.5 built with go1.13.1 + the latest unreleased patches that will go into go1.13.2 - cross fingers this will work!

rclone-v1.49.5-windows-amd64-go1.13.1plus.zip

@Rootax
Copy link
Author

Rootax commented Oct 5, 2019

Yeah latest beta has irregular speed too (and is slower) for me.

Can you try this one.. this is v1.49.5 built with go1.13.1 + the latest unreleased patches that will go into go1.13.2 - cross fingers this will work!

rclone-v1.49.5-windows-amd64-go1.13.1plus.zip

Nop , still unsteady speed and slower :/ For each test I retry with a "stable/fast" release just after, to eliminate a isp or gdrive problem, and, yeah, this build is still problematic for me.

Am I the only one with the problem ??

@ncw
Copy link
Member

ncw commented Oct 5, 2019

Nop , still unsteady speed and slower :/ For each test I retry with a "stable/fast" release just after, to eliminate a isp or gdrive problem, and, yeah, this build is still problematic for me.

OK thanks for testing and being methodical :-)

Am I the only one with the problem ??

See this thread on the forum which was also fixed by reverting to go1.12 but I don't know if that is the same problem as yours!

@ncw
Copy link
Member

ncw commented Oct 6, 2019

Would it be possible for you to try this test on a Linux VM? So we can see whether this is a Windows specific problem or not?

@Rootax
Copy link
Author

Rootax commented Oct 6, 2019

On paper I can, but I don't have the time right now. It will took, what, 1-2 hours to set this up, so it's not a lot but right now for personal reasons, 2h is a lot. I'll have time next week end or the one after, but before that, it's challenging, sorry : /

@ncw
Copy link
Member

ncw commented Oct 7, 2019

On paper I can, but I don't have the time right now. It will took, what, 1-2 hours to set this up, so it's not a lot but right now for personal reasons, 2h is a lot. I'll have time next week end or the one after, but before that, it's challenging, sorry : /

If you have time that would be great, if not, then we can find another way I'm sure - thank you :-)

@ncw
Copy link
Member

ncw commented Oct 26, 2019

Can you test with v1.50 now it has been released?

@ncw ncw modified the milestones: v1.50, v1.51 Oct 26, 2019
@Rootax
Copy link
Author

Rootax commented Nov 17, 2019

Hello guys. I haven't forgotten this problem. But my fiber is down since mid october, so I can't reproduce the speed problem with my (very) slow backup link.
I'll give an update when my fiber is up again.

@Rootax
Copy link
Author

Rootax commented Nov 22, 2019

Well I tried tonight, but it seems I've a problem on my side, since with the "working" version or 1.50.2, I'm at 0k, nothing is uploading tonight... The checks are working... I'm using my own api, and I've not maxed out 750gb in the last 24h... will try tomorrow...

EDIT : a lot of ppl are seeing the same thing on the gdrive twitter account, so I guess it's them, not my isp, having a problem at the moment for some users.

@Rootax
Copy link
Author

Rootax commented Nov 22, 2019

All right, so 1.50.2 is working full speed, no problem with it.

@ncw
Copy link
Member

ncw commented Nov 23, 2019

Thanks for testing :-)

I'll close this now then since I think it the same issue as one I've already fixed.

@ncw ncw closed this as completed Nov 23, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants