[16.0] fs_storage + EDI: connection closed after some time

[JordiMForgeFlow]

Module

fs_storage

Describe the bug

I am using fs_storage to connect to SFTP with EDI (the PR is still not merged OCA/edi-framework#26). The connection works fine, but after some time the EDI jobs start to fail due to the following issue:

File "/opt/odoo/auto/addons/fs_storage/models/fs_storage.py", line 376, in add
    self.fs.makedirs(self.fs.sep.join(path))
  File "/usr/local/lib/python3.10/site-packages/fsspec/implementations/sftp.py", line 89, in makedirs
    self.ftp.mkdir(new_path, mode)
  File "/usr/local/lib/python3.10/site-packages/paramiko/sftp_client.py", line 460, in mkdir
    self._request(CMD_MKDIR, path, attr)
  File "/usr/local/lib/python3.10/site-packages/paramiko/sftp_client.py", line 856, in _request
    num = self._async_request(type(None), t, *args)
  File "/usr/local/lib/python3.10/site-packages/paramiko/sftp_client.py", line 881, in _async_request
    self._send_packet(t, msg)
  File "/usr/local/lib/python3.10/site-packages/paramiko/sftp.py", line 209, in _send_packet
    self._write_all(out)
  File "/usr/local/lib/python3.10/site-packages/paramiko/sftp.py", line 173, in _write_all
    n = self.sock.send(out)
  File "/usr/local/lib/python3.10/site-packages/paramiko/channel.py", line 799, in send
    return self._send(s, m)
  File "/usr/local/lib/python3.10/site-packages/paramiko/channel.py", line 1196, in _send
    raise socket.error("Socket is closed")
OSError: Socket is closed

If I press the button 'TEST CONNECTION' in the fs storage, I also get the same error. However, if I restart Odoo, it starts working fine again.

My hint is that the problem comes from fs_storage storing the connection in a variable (

storage/fs_storage/models/fs_storage.py

Line 274 in 3083166

self.__fs = self._get_filesystem()

). The connection closes at some point (probably due to timeout) and it does not work anymore.

@lmignon @sebastienbeau have you faced something like this with fs_storage?

[lmignon]

@JordiMForgeFlow I never encountered this kind of error. You could patch the code to not store the filesystem objet on a variable to see if it solve the problem...

[JordiMForgeFlow]

@lmignon thanks for the feedback. I will try and let you know the outcome

[nilshamerlinck]

Hello @JordiMForgeFlow I faced a similar issue where the underlying ssh connection seems to be prematurely closed. Adding a timeout helped as a workaround. Can you try adding { "timeout": 2 }in your fs.storage options?

[JordiMForgeFlow]

Hello @JordiMForgeFlow I faced a similar issue where the underlying ssh connection seems to be prematurely closed. Adding a timeout helped as a workaround. Can you try adding { "timeout": 2 }in your fs.storage options?

Thank you for the hint @nilshamerlinck, I will give it a try and see if that solves it

[JordiMForgeFlow]

@nilshamerlinck @lmignon After trying different things I have managed to get it working with the following changes: #320

I have suggested them as settings in the FS Storage.

Apart from storing the connection, I also saw that fsspec instances also cache information. Invalidating the cache when the connection gives an error allowed to only force the reconnection when needed.