Noop

Certain FTP servers require NOOP commands to be sent periodically to keep the connection alive. FluentFTP supports sending a NOOP command at the specified interval.

Please note that if you turn on the NOOP feature, SSL Buffering on the control connection will be forcibly turned off. The data connection will still have SSL Buffering without issues.

All credits for the development and testing of our excellent NOOP support go to Michael Stiemke and Carl Krig.

Settings

• Config.Noop - Install the NOOP Daemon whenever an FTP connection is established, which ensures that NOOPs are sent at regular intervals. This is the master switch for all NOOP functionality.

• Config.NoopInterval - Time to wait (in milliseconds) between sending NOOP commands to keep the control socket alive during long file transfers and command idle times. Setting this interval too low will have a (slightly) negative impact on performance. Use this setting if you are getting dropped control connections during file transfers. Default: 3 minutes, which catches the typical 5 minute timeout by FTP servers. Setting this to zero will cause NOOPs not to be issued. You can change this setting on the fly repeatedly during your FTP session. Warning: Do not set this to values under 15 seconds.

• Config.NoopInactiveCommands - These commands are to be used when the dataconnection is not active (no transfer is taking place).

• Config.NoopActiveCommands - These commands are to be used when the dataconnection is active (a transfer is taking place). It is strongly recommended to only use "NOOP" in this situation.

• Config.NoopTestConnectivity - Cause each and every command issued by FluentFTP to be preceded by a discrete "NOOP" command to test the connectivity first. This will allow your application to react better in cases of a sudden connection failure. Set this to true if you want to make sure beforehand that commands are going to fail due to a connection that has suddenly dropped. This can incur some control connection overhead and does not alleviate inactivity timeouts, it just helps to identify connectivity issues early on. Using this in conjunction with Config.NoopInterval further improves recovery from connection problems.

What is the current status of NOOP support?

2022/09/29 with FluentFTP release v41:

proftpd

Framework	NOOP plain text	NOOP encrypted
.NET 6.0+	works	works
.NET Framework 4.7+	works	works

vsftpd

Framework	NOOP plain text	NOOP encrypted
.NET 6.0+	works	works
.NET Framework 4.7+	works	works

bftpd

Framework	NOOP plain text	NOOP encrypted
.NET 6.0+	works	unknown
.NET Framework 4.7+	works	unknown

IBM z/OS CS FTP Server

Framework	NOOP plain text	NOOP encrypted
.NET 6.0+	works	works
.NET Framework 4.7+	works	works

Other servers: Currently Unknown.

See the full issue for more info.

If NOOP handling fails, what does it result in?

The incorrect handling of 200... responses to the NOOP command will typically cause the next command after the file transfer command to fail. The RETR or STOR command that was using the NOOP commands to keep it alive will typically end successfully.

As a workaround, you can disconnect the session after the file transfer and reconnect to continue.

Why do we need to send NOOP commands?

NOOP commands are a way to keep-alive the FTP control connection. Failure to send these periodically results in Timed out trying to read data from the socket stream! errors.

Being able to solve problems where your control connection is being dropped because of a too long idle state really depends on many factors. There are a lot of places in between the client and server where this can happen, it can happen on the server too.

The servers configuration might discourage keep alives, depending on its configuration.

The way a server reacts to keep alive tests (such as NOOP) will differ not only by its configuration - different servers are programmed differently.

Last but not least, we see different behaviour on the client side, depending on encryption, ssl protocol and the underlying .NET frameworks.

References:

1. NOOP handling of different servers
2. Keep Alive Techniques in general (proftpd) (here, read the section on FTP Keep Alive in detail).

How do I set the NoopInterval property?

It seems that the AzureVM router or firewall does not respect TCP Keep-Alive on the control socket (port 21). When large file transfers runs on data socket (port 20 or passive) there is no traffic on the control socket and it breaks for after ~280s (perhaps 240s). Before mentioned workaround setting EnableThreadSafeDataConnections = true simply caused the exception from the broken connection to be discarded, pretending all was fine and opening a new control socket.

The NOOP command responds something like '200 NOOP command successful.' or '200 NOOP: data transfer in progress' and is typically used by a FTP UI client to keep the control socket alive in-between commands, so it is a good candidate for using during long-running command executions (i.e. RETR/STOR). However the https://tools.ietf.org/html/rfc959 neither suggests that nor against it.

Indeed, googling shows that it has been proven in battle, with some odd complaints (see link to smartftp.com some comments back) that it could potentially cause file transfers to abort (remains to be proven) or make FTP servers behave strangely like not responding 200 (that is what I can see from IIS) or respond at the end of the file transfer (have no such evidence, but pull request attempted to cope with that in a rudimentary manner).

Suggested usage for NoopInterval (in milliseconds) from past experience;

• 0 (default) if you can get away with it do not fiddle with it, keeping in mind that sending commands during command execution might not be by the book for all FTP server implementations out there
• firewall timeout divided by 2, if you can determine the timeout by transferring files 10MB, 50MB, 100MB, 500MB, 1GB, 5GB
• 30000 or 45000 or 90000 (corresponding to 30s, 45s or 90s) would make sense
• not something like 1000 or 5000, that would be nonsensical because any socket must survive 5s silence, problem should be sought elsewhere like in the firewall

References:

1. Original author comments from the PR that implemented Noop

When are NOOP commands internally sent?

NOOP commands are sent on the FTP control connection when:

1. The data connection is idle and the last command to be issued to the control connection is more than NoopInterval milliseconds ago. The Noop command to be issued is taken randomly from the list defined in NoopInactiveCommands. A reply is awaited and checked for success. Failures (such as due to a lost connection / timeout) will cause a reconnect on the next real command - but not in all cases. Use NoopTestConnectivity for additional safety.

2. The data connection is active and transferring data and the last command to be issued to the control connection is more than NoopInterval milliseconds ago. The Noop command to be issued is taken randomly from the list defined in NoopActiveCommands (preferably only "NOOP"). A reply is not awaited and all Noop commands issued during the transfer period will be collected at the end of the transfer and investigated.

Why do I see many NOOP commands in sequence?

If you see a log like the following, and you don't want it, you can filter out Verbose messages.

Command:  NOOP
Command:  NOOP
Command:  NOOP
Command:  NOOP
Command:  NOOP
Command:  NOOP
Command:  NOOP
Command:  NOOP
Command:  NOOP
Status:   Disposing FtpSocketStream...
Status:   Waiting for a response
Skipped:  200 Noop ok.
Skipped:  200 Noop ok.
Skipped:  200 Noop ok.
Skipped:  200 Noop ok.
Skipped:  200 Noop ok.
Skipped:  200 Noop ok.
Skipped:  200 Noop ok.
Skipped:  200 Noop ok.
Skipped:  200 Noop ok.
Skipped:  200 Noop ok.
Status:   Waiting - 8 seconds left
Status:   Waiting - 7 seconds left
Status:   Waiting - 6 seconds left
Status:   Waiting - 5 seconds left
Status:   Waiting - 4 seconds left
Status:   Waiting - 3 seconds left
Status:   Waiting - 2 seconds left
Status:   Waiting - 1 seconds left
Status:   Waiting - 0 seconds left
Status:   GetReply(...) sequence: 226,200,200,200,200,200,200,200,200,200,200

This means a data transfer was running, then finished. While it was running, NOOP commands were being issued. At the end of the transfer, we are then Waiting for a response and dispose the socket stream.

Because NOOPs had benn issued we then go into the "pick up the pending responses loop". There is a 226 (which need not be the first one in the sequence, mind you), there are any number of responses to the NOOP, whose spelling may also be different from server to server. The 10 second wait is from the apache recommendation in this discussion , it only happens if someone used NOOPs.

The whole thing actually only happens if someone caused one or more NOOPs to be issued. If not we would be in a blocking wait for the 226 or any fail messages and not in a 10 second loop.

If you request a NOOP every 30 seconds, but the transfer is over after 15 seconds, no NOOP would have been issued, therefore we would be in the non-NOOP blocking mode.

If you disable NOOPs entirely, the same is true, non-NOOP blocking mode.

If any number of NOOPs had been issued, you would go into the "10 second timer loop mode" (non-blocking) which picks up any number of errant 200 NOOP OK messages and also picks up the one important success/failure message that tells us about the status of the finished transfer.

The recommendation, mentioned above:

I think that what we need to do is to collect (#NOOPs + 1) responses at the end with a read timeout 10s, filter out 500 and 200 and the remaining status will be 226 or an error status.

How do servers respond to NOOP?

Race conditions

When retrieving a file, FTP servers send the transfer status right after they send the EOF on the data socket, but before the client closes the socket. This may lead to a race condition: If we've sent a NOOP not longer than a second ago and want to consume the NOOP reply, we may receive the transfer status instead. Subsequent call to completePendingCommand() will consume the NOOP reply.

Maybe it's not a big problem, because we will have either 226-File successfully transferred or 200 NOOP command successful. as our last message and a high-level FTPClient method will return true.

Pure-FTPd

During a transmission Pure-FTPd replies to NOOPs immediately with: 500 Unknown command (perhaps because it only expects ABOR in this state)

At the end of transmission it sends the transmission status (usually 226)

BulletProof FTP Server

During a transmission BulletProof-FTP replies to NOOPs immediately with: 200 NOOP command successful.

If there was at least one NOOP and the command is STOR, the transmission status is not sent. If there were no NOOPs or the command is RETR, then the status is returned (usually 226)

Most other FTP servers

NOOP replies are delayed. After a transmission completes, servers send 226 first and the NOOP replies after that.

Responses

Here are possible response sequences:

Pure-FTPd success

500,500,500,500,226
500,500,500,226,200 (in case of race condition)

Pure-FTPd error (RETR data socket closed by client)

500,500,500,500,150
500,500,500,150,200 (in case of race condition)

BulletProof FTP Server STOR, success/error (STOR data socket closed by client)

200,200,200,200,---
200,200,200,---,200 (in case of race condition)
226 (when no NOOPs sent)

BulletProof FTP Server RETR, success

226 (when no NOOPs sent)
200,200,200,200,---
200,200,200,---,200 (in case of race condition)

BulletProof FTP Server RETR, error (RETR data socket closed by client)

426 (when no NOOPs sent)
200,200,200,200,426
200,200,200,426,200 (in case of race condition)

Most other FTP servers success

226,200,200,200,200 (no race condition possible)

Most other FTP servers error (RETR data socket closed by client)

426,200,200,200,200 (no race condition possible)