Skip to content

Commit

Permalink
Add Python 3.13 compatibility (#474)
Browse files Browse the repository at this point in the history
* Py 313 compatibility

* type ignore

* Update CI versions

* Fix version checks

* Add tests for new methods

* mypy fix

* Right mypy line

* Keep 3.8 for now

* Keep as native paths until last chance

* Allow pathlike for glob and rglob

* fix tests

* Pass on <py3.13

* update history note

* Code review comments

* omit glob.py from coverage

* reword history

* Add NotImplementedError for the missing methods in the CloudPath class
  • Loading branch information
pjbull authored Oct 18, 2024
1 parent 68774bf commit 5f5e054
Show file tree
Hide file tree
Showing 17 changed files with 371 additions and 67 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/docs-master.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: 3.8
python-version: 3.11
cache: "pip" # caching pip dependencies
cache-dependency-path: |
pyproject.toml
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/docs-preview.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: 3.8
python-version: 3.11
cache: "pip" # caching pip dependencies
cache-dependency-path: |
pyproject.toml
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: 3.8
python-version: 3.11

- name: Install dependencies
run: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ jobs:
strategy:
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
python-version: [3.8, 3.9, "3.10", "3.11", "3.12"]
python-version: [3.8, 3.9, "3.10", "3.11", "3.12", "3.13"]

steps:
- uses: actions/checkout@v4
Expand Down
12 changes: 12 additions & 0 deletions HISTORY.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,18 @@

- Added support for custom schemes in CloudPath and Client subclases. (Issue [#466](https://github.com/drivendataorg/cloudpathlib/issues/466), PR [#467](https://github.com/drivendataorg/cloudpathlib/pull/467))
- Fixed `ResourceNotFoundError` on Azure gen2 storage accounts with HNS enabled and issue that some Azure credentials do not have `account_name`. (Issue [#470](https://github.com/drivendataorg/cloudpathlib/issues/470), Issue [#476](https://github.com/drivendataorg/cloudpathlib/issues/476), PR [#478](https://github.com/drivendataorg/cloudpathlib/pull/478))
- Added support for Python 3.13 (Issue [#472](https://github.com/drivendataorg/cloudpathlib/issues/472), [PR #474](https://github.com/drivendataorg/cloudpathlib/pull/474)):
- [`.full_match` added](https://docs.python.org/3.13/library/pathlib.html#pathlib.PurePath.full_match)
- [`.from_uri` added](https://docs.python.org/3.13/library/pathlib.html#pathlib.Path.from_uri)
- [`follow_symlinks` kwarg added to `is_file`](https://docs.python.org/3.13/library/pathlib.html#pathlib.Path.is_file) added as no-op
- [`follow_symlinks` kwarg added to `is_dir`](https://docs.python.org/3.13/library/pathlib.html#pathlib.Path.is_dir) added as no-op
- [`newline` kwarg added to `read_text`](https://docs.python.org/3.13/library/pathlib.html#pathlib.Path.read_text)
- [`recurse_symlinks` kwarg added to `glob`](https://docs.python.org/3.13/library/pathlib.html#pathlib.Path.glob) added as no-op
- [`pattern` parameter for `glob` can be PathLike](https://docs.python.org/3.13/library/pathlib.html#pathlib.Path.glob)
- [`recurse_symlinks` kwarg added to `rglob`](https://docs.python.org/3.13/library/pathlib.html#pathlib.Path.rglob) added as no-op
- [`pattern` parameter for `rglob` can be PathLike](https://docs.python.org/3.13/library/pathlib.html#pathlib.Path.rglob)
- [`.parser` property added](https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.parser)


## v0.19.0 (2024-08-29)

Expand Down
6 changes: 0 additions & 6 deletions cloudpathlib/azure/azblobpath.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,12 +39,6 @@ class AzureBlobPath(CloudPath):
def drive(self) -> str:
return self.container

def is_dir(self) -> bool:
return self.client._is_file_or_dir(self) == "dir"

def is_file(self) -> bool:
return self.client._is_file_or_dir(self) == "file"

def mkdir(self, parents=False, exist_ok=False):
self.client._mkdir(self, parents=parents, exist_ok=exist_ok)

Expand Down
104 changes: 76 additions & 28 deletions cloudpathlib/cloudpath.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,11 @@
PosixPath,
PurePosixPath,
WindowsPath,
_PathParents,
)

import shutil
import sys
from types import MethodType
from typing import (
BinaryIO,
Literal,
Expand Down Expand Up @@ -56,21 +56,29 @@
else:
from typing_extensions import Self

if sys.version_info >= (3, 12):

if sys.version_info < (3, 12):
from pathlib import _posix_flavour # type: ignore[attr-defined] # noqa: F811
from pathlib import _make_selector as _make_selector_pathlib # type: ignore[attr-defined] # noqa: F811
from pathlib import _PathParents # type: ignore[attr-defined]

def _make_selector(pattern_parts, _flavour, case_sensitive=True): # noqa: F811
return _make_selector_pathlib(tuple(pattern_parts), _flavour)

elif sys.version_info[:2] == (3, 12):
from pathlib import _PathParents # type: ignore[attr-defined]
from pathlib import posixpath as _posix_flavour # type: ignore[attr-defined]
from pathlib import _make_selector # type: ignore[attr-defined]
else:
from pathlib import _posix_flavour # type: ignore[attr-defined]
from pathlib import _make_selector as _make_selector_pathlib # type: ignore[attr-defined]
elif sys.version_info >= (3, 13):
from pathlib._local import _PathParents
import posixpath as _posix_flavour # type: ignore[attr-defined] # noqa: F811

def _make_selector(pattern_parts, _flavour, case_sensitive=True):
return _make_selector_pathlib(tuple(pattern_parts), _flavour)
from .legacy.glob import _make_selector # noqa: F811


from cloudpathlib.enums import FileCacheMode

from . import anypath

from .exceptions import (
ClientMismatchError,
CloudPathFileExistsError,
Expand Down Expand Up @@ -194,7 +202,12 @@ def __init__(cls, name: str, bases: Tuple[type, ...], dic: Dict[str, Any]) -> No
and getattr(getattr(Path, attr), "__doc__", None)
):
docstring = getattr(Path, attr).__doc__ + " _(Docstring copied from pathlib.Path)_"
getattr(cls, attr).__doc__ = docstring

if isinstance(getattr(cls, attr), (MethodType)):
getattr(cls, attr).__func__.__doc__ = docstring
else:
getattr(cls, attr).__doc__ = docstring

if isinstance(getattr(cls, attr), property):
# Properties have __doc__ duplicated under fget, and at least some parsers
# read it from there.
Expand Down Expand Up @@ -383,16 +396,6 @@ def drive(self) -> str:
"""For example "bucket" on S3 or "container" on Azure; needs to be defined for each class"""
pass

@abc.abstractmethod
def is_dir(self) -> bool:
"""Should be implemented without requiring a dir is downloaded"""
pass

@abc.abstractmethod
def is_file(self) -> bool:
"""Should be implemented without requiring that the file is downloaded"""
pass

@abc.abstractmethod
def mkdir(self, parents: bool = False, exist_ok: bool = False) -> None:
"""Should be implemented using the client API without requiring a dir is downloaded"""
Expand Down Expand Up @@ -427,24 +430,44 @@ def as_uri(self) -> str:
def exists(self) -> bool:
return self.client._exists(self)

def is_dir(self, follow_symlinks=True) -> bool:
return self.client._is_file_or_dir(self) == "dir"

def is_file(self, follow_symlinks=True) -> bool:
return self.client._is_file_or_dir(self) == "file"

@property
def fspath(self) -> str:
return self.__fspath__()

def _glob_checks(self, pattern: str) -> None:
if ".." in pattern:
@classmethod
def from_uri(cls, uri: str) -> Self:
return cls(uri)

def _glob_checks(self, pattern: Union[str, os.PathLike]) -> str:
if isinstance(pattern, os.PathLike):
if isinstance(pattern, CloudPath):
str_pattern = str(pattern.relative_to(self))
else:
str_pattern = os.fspath(pattern)
else:
str_pattern = str(pattern)

if ".." in str_pattern:
raise CloudPathNotImplementedError(
"Relative paths with '..' not supported in glob patterns."
)

if pattern.startswith(self.cloud_prefix) or pattern.startswith("/"):
if str_pattern.startswith(self.cloud_prefix) or str_pattern.startswith("/"):
raise CloudPathNotImplementedError("Non-relative patterns are unsupported")

if self.drive == "":
raise CloudPathNotImplementedError(
".glob is only supported within a bucket or container; you can use `.iterdir` to list buckets; for example, CloudPath('s3://').iterdir()"
)

return str_pattern

def _build_subtree(self, recursive):
# build a tree structure for all files out of default dicts
Tree: Callable = lambda: defaultdict(Tree)
Expand Down Expand Up @@ -488,9 +511,9 @@ def _glob(self, selector, recursive: bool) -> Generator[Self, None, None]:
yield (self / str(p)[len(self.name) + 1 :])

def glob(
self, pattern: str, case_sensitive: Optional[bool] = None
self, pattern: Union[str, os.PathLike], case_sensitive: Optional[bool] = None
) -> Generator[Self, None, None]:
self._glob_checks(pattern)
pattern = self._glob_checks(pattern)

pattern_parts = PurePosixPath(pattern).parts
selector = _make_selector(
Expand All @@ -505,9 +528,9 @@ def glob(
)

def rglob(
self, pattern: str, case_sensitive: Optional[bool] = None
self, pattern: Union[str, os.PathLike], case_sensitive: Optional[bool] = None
) -> Generator[Self, None, None]:
self._glob_checks(pattern)
pattern = self._glob_checks(pattern)

pattern_parts = PurePosixPath(pattern).parts
selector = _make_selector(
Expand Down Expand Up @@ -812,8 +835,13 @@ def read_bytes(self) -> bytes:
with self.open(mode="rb") as f:
return f.read()

def read_text(self, encoding: Optional[str] = None, errors: Optional[str] = None) -> str:
with self.open(mode="r", encoding=encoding, errors=errors) as f:
def read_text(
self,
encoding: Optional[str] = None,
errors: Optional[str] = None,
newline: Optional[str] = None,
) -> str:
with self.open(mode="r", encoding=encoding, errors=errors, newline=newline) as f:
return f.read()

def is_junction(self):
Expand Down Expand Up @@ -904,6 +932,19 @@ def is_relative_to(self, other: Self) -> bool:
def name(self) -> str:
return self._dispatch_to_path("name")

def full_match(self, pattern: str, case_sensitive: Optional[bool] = None) -> bool:
if sys.version_info < (3, 13):
raise NotImplementedError("full_match requires Python 3.13 or higher")

# strip scheme from start of pattern before testing
if pattern.startswith(self.anchor + self.drive):
pattern = pattern[len(self.anchor + self.drive) :]

# remove drive, which is kept on normal dispatch to pathlib
return PurePosixPath(self._no_prefix_no_drive).full_match( # type: ignore[attr-defined]
pattern, case_sensitive=case_sensitive
)

def match(self, path_pattern: str, case_sensitive: Optional[bool] = None) -> bool:
# strip scheme from start of pattern before testing
if path_pattern.startswith(self.anchor + self.drive + "/"):
Expand All @@ -916,6 +957,13 @@ def match(self, path_pattern: str, case_sensitive: Optional[bool] = None) -> boo

return self._dispatch_to_path("match", path_pattern, **kwargs)

@property
def parser(self) -> Self:
if sys.version_info < (3, 13):
raise NotImplementedError("parser requires Python 3.13 or higher")

return self._dispatch_to_path("parser")

@property
def parent(self) -> Self:
return self._dispatch_to_path("parent")
Expand Down
6 changes: 0 additions & 6 deletions cloudpathlib/gs/gspath.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,12 +32,6 @@ class GSPath(CloudPath):
def drive(self) -> str:
return self.bucket

def is_dir(self) -> bool:
return self.client._is_file_or_dir(self) == "dir"

def is_file(self) -> bool:
return self.client._is_file_or_dir(self) == "file"

def mkdir(self, parents=False, exist_ok=False):
# not possible to make empty directory on cloud storage
pass
Expand Down
Loading

0 comments on commit 5f5e054

Please sign in to comment.