Properly handle unavailable external storages

[PVince81]

An unavailable external storage is whenever a storage has already been mounted once, the mount point exists, but one of the following occurred:

1. Remote credentials/password changed
2. Temporary network failure, remote not reachable

So far, not all the backends properly support this and some can have weird behaviors.
The only one that seems to work correctly is the "ownCloud" backend (and server to server sharing).

Here are the backends to verify and/or raise tickets for:

• Local !
• SFTP: SFTP downloads empty files if remote password was changed #11785 (possible fix at [WIP] Use exceptions in SFTP backend #13390 )
• FTP
• Dropbox
• Google Drive
• Amazon S3
• Swift
• WebDAV
• SMB: SMB storage must throw StorageNotAvailable in specific cases #21219

We need to make sure that every backend will throw a \OCP\Files\StorageNotAvailableException whenever one of the causes mentioned above appear.

@MTRichards @craigpg @karlitschek @icewind1991 @jvillafanez @jmaciasportela

[PVince81]

The sync client also needs to be able to react to 503, this was raised here: owncloud/client#1923

[PVince81]

Note: it is the responsibility of the Storage class to decide whenever the remote error needs to lead to StorageNotAvailableException or another exception.

[PVince81]

@icewind1991 can you help with this ?

[PVince81]

Goal is that when an ext storage isn't available WebDAV and Web UI still work instead of throwing 500.
Making the ext storages all raise \OCP\Files\StorageNotAvailableException when appropriate should make it work.
Note that #11992 #11992 needs to be merged first for it to work with WebDAV.

[icewind1991]

WebDAV should throw the correct exceptions already
FTP I dont think is doable with the current backend (based on the builtin stream wrapper)
SMB might be doable with the currently backend but probably a lot of work, the new SMB backend will make it easy.
SFT, Amazon, Swift, Google Drive and Dropbox should be doable as long as their libraries expose the correct exceptions

[PVince81]

Thanks for the info.
If needed we can split the task in separate tickets and do this progressively.

From what I saw with Dropbox, if I simulate a missing "api.dropbox.com" it will throw a "NotFound" exception which might or might not be the same as a file not found exception. This means potentially extra logic to properly detect such cases.

[PVince81]

Ideally we should always show the mount points in the file tree, even if the config is invalid/temporarily available. This will make it possible to display a warning in the UI: #20427

@icewind1991 @Xenopathic I believe we might need to change the code that mounts the storages and make it still create the mount point even if the storage threw an exception ?

CC @jmaciasportela

[PVince81]

@icewind1991 can you help with this ?
For now assume that the storages will throw StorageNotAvailable (ex: Webdav) and make it work for those.

[RobinMcCorkell]

@PVince81 The mount point is always created, even when failed, so we just need a way to catch the exceptions coming back and create an appropriate file entry.

[PVince81]

I added the "Local". Someone reported on IRC that they had a NFS or something mounted to a data folder, then mounted inside OC as a "Local" external storage.

Maybe to detect such unavailable cases, OC could write a file ".ocdata" there at init time. Then later if all files disappeared, check if ".ocdata" is still there. If yes: files were really deleted. If not: mount point is missing. This assumed that the NFS is mounted already when configuring it in OC. Tricky...

[RobinMcCorkell]

@PVince81 That will only work for the situation where the mount gets unmounted for whatever reason, which is clearly a serious system issue (and so should be handled at the system level). Usually though, NFS will block read/writes if the server goes down, so ownCloud will just hang instead of continuing. Eventually an error will be reported back to ownCloud, which should trigger the unavailability logic.

[PVince81]

Ongoing => 9.2

[ownclouders]

Hey, this issue has been closed because the label status/STALE is set and there were no updates for 7 days. Feel free to reopen this issue if you deem it appropriate.

(This is an automated comment from GitMate.io.)

[ownclouders]

Hey, this issue has been closed because the label status/STALE is set and there were no updates for 7 days. Feel free to reopen this issue if you deem it appropriate.

(This is an automated comment from GitMate.io.)