fsspec.open() doesn't work with square brackets in a filename

[anmyachev]

For example, fsspec.open("filename[test].csv").

Is this by design or a bug? (python builtin open function works in this case, at least for python 3.11)

[martindurant]

fsspec.open uses the same code path as open_files, and allows glob patterns in the string. These are not generally used in filepaths.... So in this case, the [test] is interpreted as a range of possible characters, but only one.
Interestingly, the following does work:

fsspec.open("filename*.csv")

There is probably a way to escape the special characters, but we are laying glob on regex, so I'm not sure how. Also, passing a list containing just the one match (with open_files) doesn't work either, since fsspec also considers a list of glob-strings.

[anmyachev]

@martindurant thanks for the quick answer!

fsspec.open uses the same code path as open_files, and allows glob patterns in the string. These are not generally used in filepaths...

Indeed, this is rare. Then I assume the current implementation will remain as is?

There is probably a way to escape the special characters, but we are laying glob on regex, so I'm not sure how.

Unfortunately me too.

[martindurant]

I'd be happy to find a way through, especially since open() is supposed to work on single paths, never return a list.

[Skylion007]

@martindurant Doesn't re.escape do everything you want here?

[martindurant]

@Skylion007 , you can try, but we are converting from glob-strings to re, and [..] is valid wildcarding in both.

[Skylion007]

@martindurant Do we actually want to support globbing in this code path, seems to odd to me to want to do that. Easy solution seems to be use glob.escape and re.escape if we want to match both filenames literally, right? Or at least we can add a flag to do so?

It seems to me like this codepath is designed to work the same as builtins.open so we should probably match call convention. Also, it's not listed anywhere in the doc that open is suppose to support globs https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.open

[martindurant]

True, but at the moment, open() calls open_files(...)[0], and the latter does support globbing. It is not totally unreasonable to want to open("file*") and get back the first file that matches, although this is not explicitly documented.

[martindurant]

Besides which, I am not sure that the escaped character would not be caught by the contains_magic checks anyway; and don't forget that they would need to be un-escaped again before passing to the actual backend.

[Skylion007]

But it's also totally reasonable to want to get file[final].csv. Currently there is not a way to access this type of filename at all though... It also prevents fsspec open from otherwise being a nearly perfect replacement of builtins.open() so that you can just monkeypatch it in.

[Skylion007]

@martindurant Figured it out! It was a bug in an arg not being respected in open_files