[IMP] fs_storage: add setting to store connection and invalidating fi…

…lesystem cache

fs_storage/README.rst

@@ -7,7 +7,7 @@ Filesystem Storage Backend
    !! This file is generated by oca-gen-addon-readme !!
    !! changes will be overwritten.                   !!
    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
-   !! source digest: sha256:f7f1c906045d3f8e370a9b1a9d952f9df4dc14ed1a3003b4370f1807993a64cd
+   !! source digest: sha256:e299098b480d86ad11d08cde42775014cfe9c766aa57457637c504705d4eac8e
    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 
 .. |badge1| image:: https://img.shields.io/badge/maturity-Beta-yellow.png
@@ -117,8 +117,12 @@ When you create a new backend, you must specify the following:
 * The protocol options. These are the options that will be passed to the
   fsspec python package when creating the filesystem. These options depend
   on the protocol used and are described in the fsspec documentation.
-* Resolve env vars. This options resolves the protocol options values starting 
+* Resolve env vars. This options resolves the protocol options values starting
   with $ from environment variables
+* Store connection. This option stores the filesystem object in memory for quick
+  access.
+* Invalidate Cache on Error. This option invalidates the filesystem cache when
+  the connection fails, in order to trigger the reconnection.
 
 Some protocols defined in the fsspec package are wrappers around other
 protocols. For example, the SimpleCacheFileSystem protocol is a wrapper

fs_storage/models/fs_storage.py

@@ -130,6 +130,17 @@ def __init__(self, env, ids=(), prefetch_ids=()):
         help="Relative path to the directory to store the file"
     )
 
+    store_connection = fields.Boolean(
+        default=True,
+        help="If marked, the connection to the filesystem is stored in memory.",
+    )
+    # https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.clear_instance_cache  # noqa: B950
+    invalidate_cache_on_error = fields.Boolean(
+        default=True,
+        help="If marked, the filesystem cache will be invalidated when there is an "
+        "error in the connection. Invalidating the cache will force a reconnection.",
+    )
+
     # the next fields are used to display documentation to help the user
     # to configure the backend
     options_protocol = fields.Selection(
@@ -270,6 +281,8 @@ def _compute_options_properties(self) -> None:
     def fs(self) -> fsspec.AbstractFileSystem:
         """Get the fsspec filesystem for this backend."""
         self.ensure_one()
+        if not self.store_connection:
+            return self._get_filesystem()
         if not self.__fs:
             self.__fs = self._get_filesystem()
         return self.__fs
@@ -363,6 +376,12 @@ def _get_filesystem(self) -> fsspec.AbstractFileSystem:
         directory_path = self.directory_path
         if directory_path:
             fs = fsspec.filesystem("rooted_dir", path=directory_path, fs=fs)
+        if self.invalidate_cache_on_error:
+            # Check whether we need to invalidate cache or not
+            try:
+                fs.exists("/")
+            except Exception:
+                fs.clear_instance_cache()
         return fs
 
     # Deprecated methods used to ease the migration from the storage_backend addons

fs_storage/readme/USAGE.rst

@@ -15,8 +15,12 @@ When you create a new backend, you must specify the following:
 * The protocol options. These are the options that will be passed to the
   fsspec python package when creating the filesystem. These options depend
   on the protocol used and are described in the fsspec documentation.
-* Resolve env vars. This options resolves the protocol options values starting 
+* Resolve env vars. This options resolves the protocol options values starting
   with $ from environment variables
+* Store connection. This option stores the filesystem object in memory for quick
+  access.
+* Invalidate Cache on Error. This option invalidates the filesystem cache when
+  the connection fails, in order to trigger the reconnection.
 
 Some protocols defined in the fsspec package are wrappers around other
 protocols. For example, the SimpleCacheFileSystem protocol is a wrapper

fs_storage/readme/newsfragments/320.feature

@@ -0,0 +1,2 @@
+Add storage setting to decide whether the filesystem object is stored in memory or not.
+Add storage setting to force the invalidation of the filesystem cache when the connection fails, forcing a reconnection.

fs_storage/static/description/index.html

@@ -1,3 +1,4 @@
+<?xml version="1.0" encoding="utf-8"?>
 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
@@ -366,7 +367,7 @@ <h1 class="title">Filesystem Storage Backend</h1>
 !! This file is generated by oca-gen-addon-readme !!
 !! changes will be overwritten.                   !!
 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
-!! source digest: sha256:f7f1c906045d3f8e370a9b1a9d952f9df4dc14ed1a3003b4370f1807993a64cd
+!! source digest: sha256:e299098b480d86ad11d08cde42775014cfe9c766aa57457637c504705d4eac8e
 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! -->
 <p><a class="reference external image-reference" href="https://odoo-community.org/page/development-status"><img alt="Beta" src="https://img.shields.io/badge/maturity-Beta-yellow.png" /></a> <a class="reference external image-reference" href="http://www.gnu.org/licenses/lgpl-3.0-standalone.html"><img alt="License: LGPL-3" src="https://img.shields.io/badge/licence-LGPL--3-blue.png" /></a> <a class="reference external image-reference" href="https://github.com/OCA/storage/tree/16.0/fs_storage"><img alt="OCA/storage" src="https://img.shields.io/badge/github-OCA%2Fstorage-lightgray.png?logo=github" /></a> <a class="reference external image-reference" href="https://translation.odoo-community.org/projects/storage-16-0/storage-16-0-fs_storage"><img alt="Translate me on Weblate" src="https://img.shields.io/badge/weblate-Translate%20me-F47D42.png" /></a> <a class="reference external image-reference" href="https://runboat.odoo-community.org/builds?repo=OCA/storage&amp;target_branch=16.0"><img alt="Try me on Runboat" src="https://img.shields.io/badge/runboat-Try%20me-875A7B.png" /></a></p>
 <p>This addon is a technical addon that allows you to define filesystem like
@@ -472,6 +473,10 @@ <h2><a class="toc-backref" href="#toc-entry-2">Configuration</a></h2>
 on the protocol used and are described in the fsspec documentation.</li>
 <li>Resolve env vars. This options resolves the protocol options values starting
 with $ from environment variables</li>
+<li>Store connection. This option stores the filesystem object in memory for quick
+access.</li>
+<li>Invalidate Cache on Error. This option invalidates the filesystem cache when
+the connection fails, in order to trigger the reconnection.</li>
 </ul>
 <p>Some protocols defined in the fsspec package are wrappers around other
 protocols. For example, the SimpleCacheFileSystem protocol is a wrapper

fs_storage/views/fs_storage_view.xml

@@ -36,6 +36,8 @@
                             <field name="protocol" />
                             <field name="directory_path" />
                             <field name="eval_options_from_env" />
+                            <field name="store_connection" />
+                            <field name="invalidate_cache_on_error" />
                             <field
                                 name="options"
                                 widget="ace"