Python 3.9 fixes, auth option for webdav, fixed stored filename

[wpichler]

Hi all,

three small fixes for some small problems...

[FIX] Small fix to reformat the auth option as tuple for protocol webdav. It is not possible to specify a tuple in json !
[FIX] Fixed the stored filename to include the path. Without this you are not able to store files in custom sub directories
[FIX] Added future import annotations to be python 3.9 compatible

[OCA-git-bot]

Hi @lmignon,
some modules you are maintaining are being modified, check this out!

[lmignon]

@wpichler The name should not contains the directory path defined on the backend.

[lmignon]

If you nest your filesystem into 'rooted_dir' filesystem, the path given as parameter to the rooted_dir must not appear into the store_fname. (When you define a directory_path on your backend,
your filesystem is nested into a rooted_dir filesystem. This nesting could be done on multiple levels)

[wpichler]

@lmignon I do see problems here - there is a function _get_fs_path - which does allow to optimize the filesystem path (by adding a sub directory). I did use this function (with a monkey patch) to create record depended file system paths (so the pdf generated for the invoice gets as path "Out-Invoice/2023/January/filename.pdf" - gets stored in nextcloud - so available in a nice directory structure for the end user).

The function _get_fs_path get used in _storage_file_write

_storage_file_read does use _fs_parse_store_fname - this function does work with the store_fname - and will not use _get_fs_path !

_file_open - does use _get_fs_parts - this does use _fs_parse_store_fname - so also no _get_fs_path

So the problem i got was - the file was stored in "some/path/filename" - but the system could not read the file - because the store_fname does not had the path - and the read functions do not use the _get_fs_path function to restore the path for reading.

So - why not store the path with store_fname ? I did this change - and it does work like a charm.

[lmignon]

I think we can manage-it by determining which part of the path is related to the nested rooted_dir filesystem and only remove this part of the path from the new_filename_with_path... A rooted_dir has a path attribute and a fs attribute...

something like.....

# rooted_dir path computation...
rooted_paths = []
rooted_dir_fs = fs

while rooted_dir_fs and hasattr(rooted_dir_fs, "path"):
    rooted_paths.append(rooted_dir_fs.path)
    rooted_dir_fs = getattr(rooted_dir_fs, "fs", None)
    
rooted_paths.reverse()
rooted_path = "/".join(rooted_paths)

new_filename = new_filename_with_path.replace(rooted_path, "")

In any cases, we must add unit tests for such a cases.

[wpichler]

I've applied your code (with comments that code is coming from your side....) - tested it - does work fine for me.

[lmignon]

We need unit tests. IMO, the code could be extracted into a model method on fs.storage.

[lmignon]

By default, all oca addons are tested with py 3.10 and odoo 16 itself is tested on py 3.10. You should use py 3.10

[wpichler]

yeah - but odoo 16 does also work with python 3.9 - even the default docker Dockerfile does use debian bullseye - which is based on python 3.9.
And other modules in the repo do already use this future import - so it should be consistent across the modules.

[lmignon]

@wpichler To move forward with the 2 validated commits regarding the support for python3.9 and the fix for webdav, I cherry picked the two commits into #286, fixed pre-commit and added a release note. Your 2 fixes will be included into the next release.

[lmignon]

The part regarding the path preservation has to be refined. (unittests, ...)

[len-foss]

Hello Laurent!

First, this is a great implementation of the storage concept and I hope there will be little issue with porting modules.

It's been working like a charm, except for the paths issue, which breaks everything.

I started working on an implementation of your proposed solution, putting that code snippet in a function. In our test case we had a rooted directory part before the filesystem path - basically for a thumbnail the rooted path is "odoo/attachments" and the added path will be something like "53/2b". But, it turns out that the 'rooted_path' was never a problem in our tests.
In other words, the diff you suggest, although more complicated, does exactly the same thing.

Are you fine with the simple diff if there's a test case checking for the rooted path issue?

[len-foss]

By the way @wpichler, actually commit ef68e0d wasn't necessary, you could simply state:
{"base_url": "https://username:password@host.tld/remote.php/webdav"}
As you can see in the documentation:
https://www.python-httpx.org/advanced/#authentication

Although it's not very intuitive so it might be good to have some doc in the future.

[lmignon]

Are you fine with the simple diff if there's a test case checking for the rooted path issue?

No sure what you suggest. Nevertheless a test case with the fix has more chance to be merged than a fix without tests 😏