Add case_sensitive argument to pathlib.PurePath.match()

[barneygale]

In #81079 we added a case_sensitive argument to pathlib.Path.glob() and rglob(). It would be good to have this functionality available in the closely-related PurePath.match() method. The rationale is much the same: this argument is useful when dealing with case-sensitive filesystems on Windows, and case-insensitive filesystems on Posix. Also, it would allow this method to be used from glob() to implement recursive globbing efficiently, which is important for #102613.

Linked PRs

• gh-104484: Add parameter @case_sensitive to pathlib.PurePath.match() function #104565

[barneygale]

I'd like to land #101398 before I start work on this.

[thirumurugan-git]

Hi @barneygale,
I believe it's a relatively simple problem that I can help fix. May I have the opportunity to work on it?

[thirumurugan-git]

@barneygale kindly review this PR

[barneygale]

Added in #104565 / dcdc90d. Thank you, @thirumurugan-git!

[eryksun]

It should be noted in the docs that Path.match() has changed in 3.12 to use re.IGNORECASE instead of normalizing the path and pattern with os.path.normcase(). On Windows, os.path.normcase() is based on the system's lowercase mapping, while re.IGNORECASE is based on Python's lowercase mapping. The two differ for 200 BMP characters and 260 non-BMP characters. Thus some patterns may match a path in 3.11 but not in 3.12, and vice versa.

[barneygale]

It used str.lower() in 3.11:

cpython/Lib/pathlib.py

Lines 189 to 190 in d1bfefd

	def casefold_parts(self, parts):
	return [p.lower() for p in parts]

[eryksun]

I lost track of changes in 3.12. I thought it always used normcase() for case folding in match() as well as the hash and comparison methods. Instead, this issue restores the use of str.lower() in match(). However, the hash and comparison methods will be inconsistent on Windows because they still use _str_normcase based on os.path.normcase(). Thus match() could be true while __eq__() is false, or vice versa, given component names that contain any of the 460 characters for which str.lower() and ntpath.normcase() are different.

[barneygale]

Here's a table showing how __eq__(), match() and glob() are implemented in 3.11 and 3.12:

	3.11	3.12
__eq__()	str.lower()	os.path.normcase()
match()	str.lower()	re.IGNORECASE
glob()	re.IGNORECASE	re.IGNORECASE

You can see that things were already inconsistent in 3.11. For 3.12, we've:

1. Brought __eq__() more in line with Windows' filename comparisons
2. Brought match() in line with its sister method glob()

I wouldn't mind changing __eq__() and friends back to using lower() if you thought that best, @eryksun!

[eryksun]

@barneygale

AFAIK, str.lower() and re.IGNORECASE are consistent. The inconsistency is with ntpath.normcase() on Windows. It's probably for the best if the pure comparison methods revert to using str.lower() for the sake of consistency, not only with glob() and match(), but also with using PureWindowsPath on POSIX. Maybe platform-dependent comparisons could be implemented on Path.

On Windows, ntpath.normcase() is based on LCMapStringEx(). It turns out that this function implements a case mapping for some non-BMP characters. WinAPI CompareStringOrdinal(), on the other hand, has no case mapping for non-BMP characters, which is consistent with Microsoft's filesystems. Thus I'd prefer for a platform-dependent comparison to use CompareStringOrdinal() instead of LCMapStringEx().

Given that any tree in a Linux ext4 filesystem can be made case insensitive, and any tree in a Windows NTFS filesystem can be made case sensitive (even a non-empty directory if the registry setting allows it), I'd prefer for a platform-dependent comparison to support partially realized behavior in this regard. Maybe the Path constructor could allow overriding the default case sensitivity for all of a path or a subtree of a path.

[barneygale]

Thank you Eryk. I've logged a new issue here: #104947. Will get it fixed pronto!

AFAIK, str.lower() and re.IGNORECASE are consistent.

Seemingly not in all circumstances:

>>> s = 'ı'
>>> t = 'i'
>>> s.lower() == t.lower()
False
>>> import re
>>> re.match(s, t, flags=re.IGNORECASE)
<re.Match object; span=(0, 1), match='i'>

Given that any tree in a Linux ext4 filesystem can be made case insensitive, and any tree in a Windows NTFS filesystem can be made case sensitive (even a non-empty directory if the registry setting allows it), I'd prefer for a platform-dependent comparison to support partially realized behavior in this regard. Maybe the Path constructor could allow overriding the default case sensitivity for all of a path or a subtree of a path.

This is on my to-do list :)

[barneygale]

Re-resolving this ticket per the above.