Fix glob path in get_file_list

[bencwallace]

The 2023.12.0 release of fsspec introduced a change to the globbing of **. As a result, get_file_list is broken:

from embedding_reader.get_file_list import get_file_list
get_file_list(".", "py")

raises

ValueError: Invalid pattern: '**' can only be an entire path component

This PR fixes this issue by changing **.{file_format} to **/*.{file_format}, which should be backwards-compatible.

[bencwallace]

Looks like the tests for older versions of Python are failing due to the way older versions of fsspec (2023.1.0 or below) handle globbing. Thoughts on bumping the fsspec dependency?

[rom1504]

"Thoughts on bumping the fsspec dependency?" do you mean dropping support for python less than 3.8 ?

[bencwallace]

Hadn't considered that. I suppose requiring a newer version of fsspec would require Python>=3.8.
Instead, I should be able to fix the errors in older versions (without compromising newer versions using the latest fsspec release) by merging the results of globbing *.{file_format} and **/*.{file_format}.

[bencwallace]

This works in both Python 3.7 and 3.8

Suggested change

	glob_pattern = path.rstrip("/") + f"/**/*.{file_format}"
	glob_pattern_root = path.rstrip("/") + f"/*.{file_format}"
	glob_pattern_sub = path.rstrip("/") + f"/**/*.{file_format}"
	file_path_root = fs.glob(glob_pattern_root)
	file_path_sub = fs.glob(glob_pattern_sub)
	file_paths = list(set(file_path_root + file_path_sub))

[bencwallace]

What do you think about the changes above? They support newer and older versions of fsspec without altering the behavior.

[rom1504]

sorry for the delay here

let's drop support for 3.6 and 3.7 ; there are other reasons to do this anyway

[loretoparisi]

@rom1504 I think the issue people are having on auto-faiss it may be caused by the drop of support mentioned here. In fact I'm using Python 3.7.5. so I did something like

if sys.version_info >= (3,8,0): 
        # LP: python > 3.8.x
        glob_pattern = path.rstrip("/") + f"/**/*.{file_format}"
    else:
        # LP: python <= 3.7.x
         glob_pattern = path.rstrip("/") + f"/**.{file_format}"

in get_file_list._get_file_list to make it working back, beucase unfortunately I cannot upgrade the python version in the short term.