Download Speed Slow

[Dailison]

With WinScp plugin i have 1,6MB/s download speed, with SSH.NET my download is 70 KBPs or lesss, how i can fix this?

I'm using the last versions of Both and VS2015 Nugget Plugins

Code SSH.nET:
using (var client = new SftpClient("XXXXX", "XXXX", "XXXX"))
{
client.Connect();
if (client.IsConnected)
{
Program.setLog("Demorou " + TimeSpan.FromSeconds((DateTime.Now - startTime).TotalSeconds) + " ms para conectar!");
Program.setLog("Baixando: " + client.WorkingDirectory + strRmPath);
using (Stream stream = File.OpenWrite(strLcGz))
{
client.DownloadFile(strRmPath, stream);
}
var timespan = TimeSpan.FromSeconds((DateTime.Now - startTime).TotalSeconds);
var size = (new FileInfo(strLcGz).Length / 1024) / 1024;
var speed = size / timespan.TotalSeconds;
Program.setLog("Speed: " + speed + " MB/s");
Program.setLog("Demorou " + timespan + " ms para baixar!");
Program.setLog("Download concluido!");
client.Disconnect();
}
}
Code WinScp:

SessionOptions soptSettings = new SessionOptions();
soptSettings.Protocol = Protocol.Sftp;
soptSettings.HostName = "XXXX";
soptSettings.UserName = "XXXX";
soptSettings.Password = "XXXX";
//soptSettings.SshHostKeyFingerprint = "XXXXXXXX";
Program.setLog("Demorou " + TimeSpan.FromSeconds((DateTime.Now - startTime).TotalSeconds) + " ms para conectar!");
Program.setLog("Baixando: " + strRmPath);
using (WinSCP.Session sesConnection = new WinSCP.Session())
{
sesConnection.Open(soptSettings);
TransferOptions toptStreamSettings = new TransferOptions();
toptStreamSettings.TransferMode = TransferMode.Binary;
sesConnection.GetFiles(strRmPath, strLcGz, false, toptStreamSettings);
}
var timespan = TimeSpan.FromSeconds((DateTime.Now - startTime).TotalSeconds);
var size = (new FileInfo(strLcGz).Length / 1024) / 1024;
var speed = size / timespan.TotalSeconds;
Program.setLog("Speed: {0} MB/s" + speed);
Program.setLog("Demorou " + timespan + " ms para baixar!");
Program.setLog("Download concluido para pasta: " + strLcGz);

[drieseng]

What SSH server are you using ?
Would it possible to set up a VM that hosts the SSH server, and share it with me if you can reproduce this issue with that VM?

[Dailison]

I can create a sftp user into the machine and you can use, it's a OVH Host VPN.

user: easykits
pass: SENDED FOR YOU IN EMAIL [email protected]

Upload a file and try'it....

I stop to SFTP beacause that problem and start to use FTP.

I try to use WinScp but is a bad idea install that program in my users...

I'm here for help you resolve that problem....

[drieseng]

I'm making good progress on this as part of issue #100.
I'll update this issue once I have new performance numbers for downloading from the OVH host.

[imapangolin]

I'm experiencing the same slow down. I don't know the SFTP host software but with FileZilla my 28 meg download takes 8 seconds. Over a minute with SSH.NET. Here's my code. I've dinked around with buffer sizes but that appears to be the only thing I can set that will impact it. 4K buffer gives me a 2 minute+ transfer. This 63K buffer gives me the minute. I'm using the beta1 pre-release package.

    public string GetFileSFTP(string ClientFileName, string ServerFileName, string ServerName, string Port, string UserID, string Password)
    {
    System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
    string Result = "Success";
    try
    {
        using (var client = new SftpClient(ServerName, Convert.ToInt32(Port), UserID, Password))
        {
            client.Connect();

            client.BufferSize = 63 * 1024;
            using (Stream file1 = File.OpenWrite(ClientFileName))
            {
                client.DownloadFile(ServerFileName, file1);
            }
        }
    }
    catch (Exception e)
    {
        Result = "Fail: " + e.ToString();
    }
    return (Result);

    }

[imapangolin]

Could this simply be that the download is single threaded?

[drieseng]

I modified DownloadFile to read-ahead 10 chunks asynchronously.
This greatly improves performance, even up to a point where we're fasten than WS_FTP.
I went from around 30 MB to 56 MB/s with my local SSH server.
Do note that these changes are not yet available (not in beta1, and not even in the develop branch).

In case of the original issue reported by @Dailison, I got around 250 KB/s against the server he used before my changes (while he claimed 70 KB/s).
After my changes - and based on preliminary tests - I got around 850 KB/s.
The "read-ahead" has more effect on slower servers and/or connections.

But even with these changes, I doubt throughput will increase in such a way that downloading a 28 MB file will go from 2+ minutes to 8 seconds.
I think there may be another issue here.
If you can debug or instrument the code, it would be great to learn if much time is spent in KeyExchangeDiffieHellman.PopulateClientExchangeValue().

[imapangolin]

Not sure how to do the instrument but I’ll look to see if theres a lot of time there. Your change I would love to get and test - let me know when it is available on the developer download and I’ll grab it.

…

On Jan 19, 2017, at 3:16 PM, Gert Driesen ***@***.***> wrote: I modified DownloadFile to read-ahead 10 chunks asynchronously. This greatly improves performance, even up to a point where we're fasten than WS_FTP. I went from around 30 MB to 56 MB/s with my local SSH server. Do note that these changes are not yet available (not in beta1, and not even in the develop branch). In case of the original issue reported by @Dailison <https://github.com/Dailison>, I got around 250 KB/s against the server he used before my changes (while he claimed 70 KB/s). After my changes - and based on preliminary tests - I got around 850 KB/s. The "read-ahead" has more effect on slower servers and/or connections. But even with these changes, I doubt throughput will increase in such a way that downloading a 28 MB file will go from 2+ minutes to 8 seconds. I think there may be another issue here. If you can debug or instrument the code, it would be great to learn if much time is spent in KeyExchangeDiffieHellman.PopulateClientExchangeValue(). — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#145 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADQXsXut0lz8AB74_mF7zc5NvqaGGJfJks5rT8SfgaJpZM4LX-0o>.

[Dailison]

I'm looking forward to a new version with this bug fixed.
And again @drieseng , tank's for support us.

[drieseng]

I committed a preliminary version of these changes in the develop branch.
Please hammer away at them, and let me know if it reduces download speed for you.

[imapangolin]

I tried the newly committed code. I wish I had good news - but for me with the following code to call it, it never return from the download. I am calling it from a .net com server so I have to create a console application to debug further but as it stands right now it just doesn't work at all!

It there something I need to do to activate it - tried various buffer sizes as well.

    string Result = "Success";
    try
    {         
        using (var client = new SftpClient(ServerName, Convert.ToInt32(Port), UserID, Password))
        {
            client.Connect();

            client.BufferSize = 4 * 1024;
            using (Stream file1 = File.OpenWrite(ClientFileName))
            {
                client.DownloadFile(ServerFileName, file1);
            }
        }
    }
    catch (Exception e)
    {
        Result = "Fail: " + e.ToString();
    }
    return (Result);

[imapangolin]

The committed developer build now functions! I will give you some performance comparisons tomorrow.
I really really appreciate what you are doing here. Thank you.

[imapangolin]

I know you say it's a WIP but I'd call it a rousing success. WinSCP which was the fastest transfer I could find has been doing 29-31 seconds on my sample file. Consistently getting 15-16 using the changes you put into SSH.NET. I say go for it and put it in the next beta! THANK YOU!

[drieseng]

@Dailison Can you also build SSH.NET from the develop branch, and validate my changes?

[Dailison]

I have changed the password authentication method for private key.

Gert i have tested the developer branch and the download speed was increased to 600 ~~ 700 kbps ....

But it's not my max download speed, currently is 1500 kbps

[drieseng]

For now, I don't see much room for improvement.
It would be great if someone would scan the code for any possible performance improvements.
Since you've reported a throughput increase of about 1000%, I think we can safely consider this specific issue closed.
Agree?

[imapangolin]

It was faster before the stability changes. That's the only place I would look. What did you do to make it recoverable?

…

On Feb 17, 2017, at 2:04 AM, Gert Driesen ***@***.***> wrote: For now, I don't see much room for improvement. It would be great if someone would scan the code for any possible performance improvements. Since you've reported a throughput increase of about 1000%, I think we can safely consider this specific issue closed. Agree? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[drieseng]

We cannot sacrifice stability for speed.
Since you confirmed that - with these changes - throughput went from roughly 70 KB/s to 700 KB/s, we can surely consider this issue fixed.

[drieseng]

Here's some more info on the read-ahead implementation

Maximum shunk size

The maximum size of a chunk that we read ahead is the configured buffer size (default is 32 KB) limited to the maximum packet size (which currently is 64 KB) minus 13 bytes (which is the protocol overhead of a SSH_FXP_DATA message):

maximum chunk size = MAX(buffer size, maximum packet size) - 13

This a technique we generally apply because most SSH servers limit the size of the payload of a SSH_MSG_CHANNEL_DATA message to 16 KB.

If we requested 16 KB of data, then the SSH_FXP_DATA payload of the SSH_MSG_CHANNEL_DATA message would be too big (16 KB + 13 bytes) to fit a single SSH_MSG_CHANNEL_DATA response.

As a result, the SSH server would split the SSH_FXP_DATA over two SSH_MSG_CHANNEL_DATA responses: one containing 16384 bytes (13 bytes header, and 16371 bytes file data), and one with the remaining 13 bytes of file data.

Maximum pending reads

The definition for a pending read is a read-ahead that either the server has not yet responsed to, or for which the response has not been consumed.

The maximum number of pending reads is the total size of the file to download divided by the maximum chunk size, plus 1 limited to 10:

maximum pending reads = MIN(10, ROUNDUP(file size / maximum chunk size) + 1)

If the file size cannot be determined, the maximum pending reads is set to 3.

Once this limit is reached, no further chunks a read-ahead until a pending read is completed.

[laurentchougrani]

as of 2020 I can state that the Sftp process is very slow....
if i use SSH directly from powershell for transfering, say a 700Mb file, it takes less than a minute, now using this transfer method it takes about 5 cigarettes and three coffees (which makes about 30min).... I do like [Coffee + cigarettes] but still... ^^

[jjxtra]

Upload speed is definitely broken. I can't get more than 500K / second out of it regardless of how many clients I create per file. Even one file has the same speed limit. Must be a global bottleneck or lock somewhere... Using mobaxterm I get 50mb / sec upload.

[zybexXL]

I've added two PRs today that improve SFTP performance (by a LOT, in my case), and simultaneously reduce CPU usage: #865 and #866. I'm not sure if the SCP file transfers also benefit, I didn't test/check that.

According to my benchmarks, both FileUpload and FileDownload have massive speed gains and are now comparable to Filezilla.

I'd be very interested to know if these changes also help you guys :)

[imapangolin]

It’ll take me a couple days to get and build this. I’ll let you know then.

…

On Aug 30, 2021, at 5:02 PM, zybexXL ***@***.***> wrote:  I've added two PRs today that improve SFTP performance (by a LOT, in my case), and simultaneously reduce CPU usage: #865 and #866. I'm not sure if the SCP file transfers also benefit, I didn't test/check that. According to my benchmarks, both Upload and Download have massive speed gains and are now comparable to Filezilla. I'd be very interested to know if these changes also help you guys :) — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[zybexXL]

Thanks!