[MIG] fs_storage: Migration to 18.0

fs_storage/README.rst

@@ -0,0 +1,301 @@
+==========================
+Filesystem Storage Backend
+==========================
+
+.. 
+   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+   !! This file is generated by oca-gen-addon-readme !!
+   !! changes will be overwritten.                   !!
+   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+   !! source digest: sha256:6e7abac468ff2942178ac2df68774f75820b618173acf8c7b76f51fbdbe3b7bc
+   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+
+.. |badge1| image:: https://img.shields.io/badge/maturity-Beta-yellow.png
+    :target: https://odoo-community.org/page/development-status
+    :alt: Beta
+.. |badge2| image:: https://img.shields.io/badge/licence-LGPL--3-blue.png
+    :target: http://www.gnu.org/licenses/lgpl-3.0-standalone.html
+    :alt: License: LGPL-3
+.. |badge3| image:: https://img.shields.io/badge/github-OCA%2Fstorage-lightgray.png?logo=github
+    :target: https://github.com/OCA/storage/tree/18.0/fs_storage
+    :alt: OCA/storage
+.. |badge4| image:: https://img.shields.io/badge/weblate-Translate%20me-F47D42.png
+    :target: https://translation.odoo-community.org/projects/storage-18-0/storage-18-0-fs_storage
+    :alt: Translate me on Weblate
+.. |badge5| image:: https://img.shields.io/badge/runboat-Try%20me-875A7B.png
+    :target: https://runboat.odoo-community.org/builds?repo=OCA/storage&target_branch=18.0
+    :alt: Try me on Runboat
+
+|badge1| |badge2| |badge3| |badge4| |badge5|
+
+This addon is a technical addon that allows you to define filesystem
+like storage for your data. It's used by other addons to store their
+data in a transparent way into different kind of storages.
+
+Through the fs.storage record, you get access to an object that
+implements the
+`fsspec.spec.AbstractFileSystem <https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem>`__
+interface and therefore give you an unified interface to access your
+data whatever the storage protocol you decide to use.
+
+The list of supported protocols depends on the installed fsspec
+implementations. By default, the addon will install the following
+protocols:
+
+-  LocalFileSystem
+-  MemoryFileSystem
+-  ZipFileSystem
+-  TarFileSystem
+-  FTPFileSystem
+-  CachingFileSystem
+-  WholeFileSystem
+-  SimplCacheFileSystem
+-  ReferenceFileSystem
+-  GenericFileSystem
+-  DirFileSystem
+-  DatabricksFileSystem
+-  GitHubFileSystem
+-  JupiterFileSystem
+-  OdooFileSystem
+
+The OdooFileSystem is the one that allows you to store your data into a
+directory mounted into your Odoo's storage directory. This is the
+default FS Storage when creating a new fs.storage record.
+
+Others protocols are available through the installation of additional
+python packages:
+
+-  DropboxDriveFileSystem -> pip install fsspec[dropbox]
+-  HTTPFileSystem -> pip install fsspec[http]
+-  HTTPSFileSystem -> pip install fsspec[http]
+-  GCSFileSystem -> pip install fsspec[gcs]
+-  GSFileSystem -> pip install fsspec[gs]
+-  GoogleDriveFileSystem -> pip install gdrivefs
+-  SFTPFileSystem -> pip install fsspec[sftp]
+-  HaddoopFileSystem -> pip install fsspec[hdfs]
+-  S3FileSystem -> pip install fsspec[s3]
+-  WandbFS -> pip install wandbfs
+-  OCIFileSystem -> pip install fsspec[oci]
+-  AsyncLocalFileSystem -> pip install 'morefs[asynclocalfs]
+-  AzureDatalakeFileSystem -> pip install fsspec[adl]
+-  AzureBlobFileSystem -> pip install fsspec[abfs]
+-  DaskWorkerFileSystem -> pip install fsspec[dask]
+-  GitFileSystem -> pip install fsspec[git]
+-  SMBFileSystem -> pip install fsspec[smb]
+-  LibArchiveFileSystem -> pip install fsspec[libarchive]
+-  OSSFileSystem -> pip install ossfs
+-  WebdavFileSystem -> pip install webdav4
+-  DVCFileSystem -> pip install dvc
+-  XRootDFileSystem -> pip install fsspec-xrootd
+
+This list of supported protocols is not exhaustive or could change in
+the future depending on the fsspec releases. You can find more
+information about the supported protocols on the `fsspec
+documentation <https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem>`__.
+
+**Table of contents**
+
+.. contents::
+   :local:
+
+Usage
+=====
+
+Configuration
+-------------
+
+When you create a new backend, you must specify the following:
+
+-  The name of the backend. This is the name that will be used to
+   identify the backend into Odoo
+
+-  The code of the backend. This code will identify the backend into the
+   store_fname field of the ir.attachment model. This code must be
+   unique. It will be used as scheme. example of the store_fname field:
+   ``odoofs://abs34Tg11``.
+
+-  The protocol used by the backend. The protocol refers to the
+   supported protocols of the fsspec python package.
+
+-  A directory path. This is a root directory from which the filesystem
+   will be mounted. This directory must exist.
+
+-  The protocol options. These are the options that will be passed to
+   the fsspec python package when creating the filesystem. These options
+   depend on the protocol used and are described in the fsspec
+   documentation.
+
+-  Resolve env vars. This options resolves the protocol options values
+   starting with $ from environment variables
+
+-  Check Connection Method. If set, Odoo will always check the
+   connection before using a storage and it will remove the fs
+   connection from the cache if the check fails.
+
+   -  ``Create Marker file``: create a hidden file on remote and then
+      check it exists with Use it if you have write access to the remote
+      and if it is not an issue to leave the marker file in the root
+      directory.
+   -  ``List file``: list all files from the root directory. You can use
+      it if the directory path does not contain a big list of files (for
+      performance reasons)
+
+Some protocols defined in the fsspec package are wrappers around other
+protocols. For example, the SimpleCacheFileSystem protocol is a wrapper
+around any local filesystem protocol. In such cases, you must specify
+into the protocol options the protocol to be wrapped and the options to
+be passed to the wrapped protocol.
+
+For example, if you want to create a backend that uses the
+SimpleCacheFileSystem protocol, after selecting the
+SimpleCacheFileSystem protocol, you must specify the protocol options as
+follows:
+
+.. code:: python
+
+   {
+       "directory_path": "/tmp/my_backend",
+       "target_protocol": "odoofs",
+       "target_options": {...},
+   }
+
+In this example, the SimpleCacheFileSystem protocol will be used as a
+wrapper around the odoofs protocol.
+
+Server Environment
+------------------
+
+To ease the management of the filesystem storages configuration accross
+the different environments, the configuration of the filesystem storages
+can be defined in environment files or directly in the main
+configuration file. For example, the configuration of a filesystem
+storage with the code fsprod can be provided in the main configuration
+file as follows:
+
+.. code:: ini
+
+   [fs_storage.fsprod]
+   protocol=s3
+   options={"endpoint_url": "https://my_s3_server/", "key": "KEY", "secret": "SECRET"}
+   directory_path=my_bucket
+
+To work, a storage.backend record must exist with the code fsprod into
+the database. In your configuration section, you can specify the value
+for the following fields:
+
+-  protocol
+-  options
+-  directory_path
+
+Migration from storage_backend
+------------------------------
+
+The fs_storage addon can be used to replace the storage_backend addon.
+(It has been designed to be a drop-in replacement for the
+storage_backend addon). To ease the migration, the fs.storage model
+defines the high-level methods available in the storage_backend model.
+These methods are:
+
+-  add
+-  get
+-  list_files
+-  find_files
+-  move_files
+-  delete
+
+These methods are wrappers around the methods of the
+fsspec.AbstractFileSystem class (see
+https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem).
+These methods are marked as deprecated and will be removed in a future
+version (V18) of the addon. You should use the methods of the
+fsspec.AbstractFileSystem class instead since they are more flexible and
+powerful. You can access the instance of the fsspec.AbstractFileSystem
+class using the fs property of a fs.storage record.
+
+Known issues / Roadmap
+======================
+
+-  Transactions: fsspec comes with a transactional mechanism that once
+   started, gathers all the files created during the transaction, and if
+   the transaction is committed, moves them to their final locations. It
+   would be useful to bridge this with the transactional mechanism of
+   odoo. This would allow to ensure that all the files created during a
+   transaction are either all moved to their final locations, or all
+   deleted if the transaction is rolled back. This mechanism is only
+   valid for files created during the transaction by a call to the open
+   method of the file system. It is not valid for others operations,
+   such as rm, mv_file, ... .
+
+Changelog
+=========
+
+16.0.1.1.0 (2023-12-22)
+-----------------------
+
+**Features**
+
+-  Add parameter on storage backend to resolve protocol options values
+   starting with $ from environment variables
+   (`#303 <https://github.com/OCA/storage/issues/303>`__)
+
+16.0.1.0.3 (2023-10-17)
+-----------------------
+
+**Bugfixes**
+
+-  Fix access to technical models to be able to upload attachments for
+   users with basic access
+   (`#289 <https://github.com/OCA/storage/issues/289>`__)
+
+16.0.1.0.2 (2023-10-09)
+-----------------------
+
+**Bugfixes**
+
+-  Avoid config error when using the webdav protocol. The auth option is
+   expected to be a tuple not a list. Since our config is loaded from a
+   json file, we cannot use tuples. The fix converts the list to a tuple
+   when the config is related to a webdav protocol and the auth option
+   is into the confix.
+   (`#285 <https://github.com/OCA/storage/issues/285>`__)
+
+Bug Tracker
+===========
+
+Bugs are tracked on `GitHub Issues <https://github.com/OCA/storage/issues>`_.
+In case of trouble, please check there if your issue has already been reported.
+If you spotted it first, help us to smash it by providing a detailed and welcomed
+`feedback <https://github.com/OCA/storage/issues/new?body=module:%20fs_storage%0Aversion:%2018.0%0A%0A**Steps%20to%20reproduce**%0A-%20...%0A%0A**Current%20behavior**%0A%0A**Expected%20behavior**>`_.
+
+Do not contact contributors directly about support or help with technical issues.
+
+Credits
+=======
+
+Authors
+-------
+
+* ACSONE SA/NV
+
+Contributors
+------------
+
+-  Laurent Mignon <laurent.mignon@acsone.eu>
+-  Sébastien BEAU <sebastien.beau@akretion.com>
+
+Maintainers
+-----------
+
+This module is maintained by the OCA.
+
+.. image:: https://odoo-community.org/logo.png
+   :alt: Odoo Community Association
+   :target: https://odoo-community.org
+
+OCA, or the Odoo Community Association, is a nonprofit organization whose
+mission is to support the collaborative development of Odoo features and
+promote its widespread use.
+
+This module is part of the `OCA/storage <https://github.com/OCA/storage/tree/18.0/fs_storage>`_ project on GitHub.
+
+You are welcome to contribute. To learn how please visit https://odoo-community.org/page/Contribute.

fs_storage/__init__.py

@@ -0,0 +1,7 @@
+# register protocols first
+from . import odoo_file_system
+from . import rooted_dir_file_system
+
+# then add normal imports
+from . import models
+from . import wizards

fs_storage/__manifest__.py

@@ -0,0 +1,23 @@
+# Copyright 2017 Akretion (http://www.akretion.com).
+# @author Sébastien BEAU <sebastien.beau@akretion.com>
+# License LGPL-3.0 or later (http://www.gnu.org/licenses/lgpl).
+
+{
+    "name": "Filesystem Storage Backend",
+    "summary": "Implement the concept of Storage with amazon S3, sftp...",
+    "version": "18.0.1.0.0",
+    "category": "FS Storage",
+    "website": "https://github.com/OCA/storage",
+    "author": " ACSONE SA/NV, Odoo Community Association (OCA)",
+    "license": "LGPL-3",
+    "development_status": "Beta",
+    "installable": True,
+    "depends": ["base", "base_sparse_field", "server_environment"],
+    "data": [
+        "views/fs_storage_view.xml",
+        "security/ir.model.access.csv",
+        "wizards/fs_test_connection.xml",
+    ],
+    "demo": ["demo/fs_storage_demo.xml"],
+    "external_dependencies": {"python": ["fsspec>=2024.5.0"]},
+}

fs_storage/demo/fs_storage_demo.xml

@@ -0,0 +1,8 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<odoo noupdate="1">
+    <record id="fs_storage_demo" model="fs.storage">
+        <field name="name">Odoo Filesystem Backend</field>
+        <field name="protocol">odoofs</field>
+        <field name="code">odoofs</field>
+    </record>
+</odoo>