Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: john-kurkowski/tldextract
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 5.1.1
Choose a base ref
...
head repository: john-kurkowski/tldextract
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: 5.1.2
Choose a head ref
  • 10 commits
  • 15 files changed
  • 3 contributors

Commits on Feb 27, 2024

  1. Fix Ruff deprecation

    john-kurkowski committed Feb 27, 2024
    Copy the full SHA
    63b4809 View commit details
  2. Accept newer Black output

    john-kurkowski committed Feb 27, 2024
    Copy the full SHA
    931c001 View commit details
  3. Add CI for PyPy3.9 and PyPy3.10 (#316)

    * Use latest PyPy versions
    * Use setup-python@v5
    elliotwutingfeng authored Feb 27, 2024
    Copy the full SHA
    4a4e0ec View commit details

Commits on Feb 28, 2024

  1. Remove socket.inet_pton (#318)

    Closes #317.
    
    - Remove **socket.inet_pton** which does not reject IPv4 addresses and IPv6 (dual) addresses with leading IPv4 zeroes on macOS.
    - Add **test_looks_like_ipv6**
    elliotwutingfeng authored Feb 28, 2024
    Copy the full SHA
    1f77877 View commit details

Commits on Mar 6, 2024

  1. Dedupe GitHub Actions workflows (#321)

    This is a minor optimization of the following.
    
    1. Only run the workflow 1x instead of 2x on a PR that's from a branch
       instead of a fork.
        * This is done by only triggering on push events when they're to the
          master branch. If you want to trigger the workflow on other
          branches, be sure to open a PR.
    2. Do not run the workflow on tags.
        * Tags are made on existing commits on the master branch, which
          already has workflow coverage, per the previous point.
    
    Optimizing this isn't a big deal for an open source project on GitHub,
    where GitHub Actions usage is free. However, it is a big deal for
    private repository development, where usage is limited (where I've been
    doing some testing).
    john-kurkowski authored Mar 6, 2024
    Copy the full SHA
    6632577 View commit details

Commits on Mar 8, 2024

  1. Use non-capturing groups for IPv4 address detection (#323)

    elliotwutingfeng authored Mar 8, 2024
    Copy the full SHA
    9f16a0c View commit details

Commits on Mar 15, 2024

  1. Automate release process (#325)

    * Add script to interactively release the current work tree, tagged, to PyPI and GitHub
    * Add release-only dependencies
    * Typecheck scripts folder
        * The folder is not a package, so tell mypy `--explicit-package-bases`
    * Test script happy path
        * Add snapshot library dependency
            * To unit test network and subprocess input, and verbose CLI output
        * Skip test on Windows
            * Snapshot paths are platform-dependent. Maintainers don't yet release using Windows, besides.
    ekcorso authored Mar 15, 2024
    Copy the full SHA
    a2eb188 View commit details

Commits on Mar 16, 2024

  1. Only install deps during tox, during CI (#326)

    tox installs deps itself. No need to do it first.
    
    Saves some install time.
    john-kurkowski authored Mar 16, 2024
    Copy the full SHA
    6466f89 View commit details

Commits on Mar 19, 2024

  1. Copy the full SHA
    537021f View commit details
  2. 5.1.2

    john-kurkowski committed Mar 19, 2024
    Copy the full SHA
    c74ef97 View commit details
Showing with 644 additions and 66 deletions.
  1. +12 −4 .github/workflows/ci.yml
  2. +10 −0 CHANGELOG.md
  3. +1 −1 LICENSE
  4. +8 −2 pyproject.toml
  5. +238 −0 scripts/release.py
  6. +244 −0 tests/__snapshots__/test_release.ambr
  7. +18 −14 tests/main_test.py
  8. +1 −0 tests/test_cache.py
  9. +95 −0 tests/test_release.py
  10. +1 −0 tests/test_trie.py
  11. +0 −1 tldextract/__main__.py
  12. +3 −4 tldextract/cache.py
  13. +6 −28 tldextract/remote.py
  14. +5 −10 tldextract/tldextract.py
  15. +2 −2 tox.ini
16 changes: 12 additions & 4 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
name: build
on: [push, pull_request]
on:
pull_request: {}
push:
branches:
- "master"
tags-ignore:
- "**"
jobs:
test:
strategy:
@@ -14,6 +20,8 @@ jobs:
{python-version: "3.11", toxenv: "py311"},
{python-version: "3.12", toxenv: "py312"},
{python-version: "pypy3.8", toxenv: "pypy38"},
{python-version: "pypy3.9", toxenv: "pypy39"},
{python-version: "pypy3.10", toxenv: "pypy310"},
]
include:
- os: ubuntu-latest
@@ -27,13 +35,13 @@ jobs:
- name: Check out repository
uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.language.python-version }}
check-latest: true
- name: Install Python requirements
run: |
pip install --upgrade pip
pip install --upgrade --editable '.[testing]'
pip install --upgrade tox
- name: Test
run: tox
env:
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -3,6 +3,16 @@
After upgrading, update your cache file by deleting it or via `tldextract
--update`.

## 5.1.2 (2024-03-18)

* Bugfixes
* Remove `socket.inet_pton`, to fix platform-dependent IP parsing ([#318](https://github.com/john-kurkowski/tldextract/issues/318))
* Use non-capturing groups for IPv4 address detection, for a slight speed boost ([#323](https://github.com/john-kurkowski/tldextract/issues/323))
* Misc.
* Add CI for PyPy3.9 and PyPy3.10 ([#316](https://github.com/john-kurkowski/tldextract/issues/316))
* Add script to automate package release process ([#325](https://github.com/john-kurkowski/tldextract/issues/325))
* Update LICENSE copyright years

## 5.1.1 (2023-11-16)

* Bugfixes
2 changes: 1 addition & 1 deletion LICENSE
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
BSD 3-Clause License

Copyright (c) 2020, John Kurkowski
Copyright (c) 2013-2024, John Kurkowski
All rights reserved.

Redistribution and use in source and binary forms, with or without
10 changes: 8 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -41,6 +41,10 @@ dependencies = [
]

[project.optional-dependencies]
release = [
"build",
"twine",
]
testing = [
"black",
"mypy",
@@ -49,6 +53,7 @@ testing = [
"pytest-mock",
"responses",
"ruff",
"syrupy",
"tox",
"types-filelock",
"types-requests",
@@ -79,12 +84,13 @@ write_to = "tldextract/_version.py"
version = {attr = "setuptools_scm.get_version"}

[tool.mypy]
explicit_package_bases = true
strict = true

[tool.pytest.ini_options]
addopts = "--doctest-modules"

[tool.ruff]
[tool.ruff.lint]
select = [
"A",
"B",
@@ -101,5 +107,5 @@ ignore = [
"E501", # line too long; if Black does its job, not worried about the rare long line
]

[tool.ruff.pydocstyle]
[tool.ruff.lint.pydocstyle]
convention = "pep257"
238 changes: 238 additions & 0 deletions scripts/release.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,238 @@
"""
This script automates the release process for a Python package.
It will:
- Add a git tag for the given version.
- Remove the previous dist folder.
- Create a build.
- Ask the user to verify the build.
- Upload the build to PyPI.
- Push all git tags to the remote.
- Create a draft release on GitHub using the version notes in CHANGELOG.md.
Prerequisites:
- This must be run from the root of the repository.
- The repo must have a clean git working tree.
- The user must have the GITHUB_TOKEN environment variable set to a valid GitHub personal access token.
- The user will need credentials for the PyPI repository, which the user will be prompted for during the upload step. The user will need to paste the token manually from a password manager or similar.
- The CHANGELOG.md file must already contain an entry for the version being released.
- Install requirements with: pip install --upgrade --editable '.[release]'
"""

from __future__ import annotations

import os
import re
import subprocess
import sys
from pathlib import Path

import requests


def add_git_tag_for_version(version: str) -> None:
"""Add a git tag for the given version."""
subprocess.run(["git", "tag", "-a", version, "-m", version], check=True)
print(f"Version {version} tag added successfully.")


def remove_previous_dist() -> None:
"""Check for dist folder, and if it exists, remove it."""
subprocess.run(["rm", "-rf", Path("dist")], check=True)
print("Previous dist folder removed successfully.")


def create_build() -> None:
"""Create a build."""
subprocess.run(["python", "-m", "build"], check=True)
print("Build created successfully.")


def verify_build(is_test: str) -> None:
"""Verify the build.
Print the archives in dist/ and ask the user to manually inspect and
confirm they contain the expected files, e.g. source files and test files.
"""
build_files = os.listdir("dist")
if len(build_files) != 2:
print(
"WARNING: dist folder contains incorrect number of files.", file=sys.stderr
)
print("Contents of dist folder:")
subprocess.run(["ls", "-l", Path("dist")], check=True)
print("Contents of tar files in dist folder:")
for build_file in build_files:
subprocess.run(["tar", "tvf", Path("dist") / build_file], check=True)
confirmation = input("Does the build look correct? (y/n): ")
if confirmation == "y":
print("Build verified successfully.")
upload_build_to_pypi(is_test)
push_git_tags()
else:
raise Exception("Could not verify. Build was not uploaded.")


def generate_github_release_notes_body(token: str, version: str) -> str:
"""Generate and grab release notes URL from Github."""
response = requests.post(
"https://api.github.com/repos/john-kurkowski/tldextract/releases/generate-notes",
headers={
"Accept": "application/vnd.github+json",
"Authorization": f"Bearer {token}",
"X-GitHub-Api-Version": "2022-11-28",
},
json={"tag_name": version},
)

try:
response.raise_for_status()
except requests.exceptions.HTTPError as err:
print(
f"WARNING: Failed to generate release notes from Github: {err}",
file=sys.stderr,
)
return ""
return str(response.json()["body"])


def get_release_notes_url(body: str) -> str:
"""Parse the release notes content to get the changelog URL."""
url_pattern = re.compile(r"\*\*Full Changelog\*\*: (.*)$")
match = url_pattern.search(body)
if match:
return match.group(1)
else:
print(
"WARNING: Failed to parse release notes URL from GitHub response.",
file=sys.stderr,
)
return ""


def get_changelog_release_notes(release_notes_url: str, version: str) -> str:
"""Get the changelog release notes.
Uses a regex starting on a heading beginning with the version number
literal, and matching until the next heading. Using regex to match markup
is brittle. Consider a Markdown-parsing library instead.
"""
with open("CHANGELOG.md") as file:
changelog_text = file.read()
pattern = re.compile(rf"## {re.escape(version)}[^\n]*(.*?)## ", re.DOTALL)
match = pattern.search(changelog_text)
if match:
return str(match.group(1)).strip()
else:
print(
f"WARNING: Failed to parse changelog release notes. Manually copy this version's notes from the CHANGELOG.md file to {release_notes_url}.",
file=sys.stderr,
)
return ""


def create_release_notes_body(token: str, version: str) -> str:
"""Compile the release notes."""
github_release_body = generate_github_release_notes_body(token, version)
release_notes_url = get_release_notes_url(github_release_body)
changelog_notes = get_changelog_release_notes(release_notes_url, version)
full_release_notes = f"{changelog_notes}\n\n**Full Changelog**: {release_notes_url}"
return full_release_notes


def create_github_release_draft(token: str, version: str) -> None:
"""Create a release on GitHub."""
release_body = create_release_notes_body(token, version)
response = requests.post(
"https://api.github.com/repos/john-kurkowski/tldextract/releases",
headers={
"Accept": "application/vnd.github+json",
"Authorization": f"Bearer {token}",
"X-GitHub-Api-Version": "2022-11-28",
},
json={
"tag_name": version,
"name": version,
"body": release_body,
"draft": True,
"prerelease": False,
},
)

try:
response.raise_for_status()
except requests.exceptions.HTTPError as err:
print(
f"WARNING: Failed to create release on Github: {err}",
file=sys.stderr,
)
return
print(f'Release created successfully: {response.json()["html_url"]}')


def upload_build_to_pypi(is_test: str) -> None:
"""Upload the build to PyPI."""
repository: list[str | Path] = (
[] if is_test == "n" else ["--repository", "testpypi"]
)
upload_command = ["twine", "upload", *repository, Path("dist") / "*"]
subprocess.run(
upload_command,
check=True,
)


def push_git_tags() -> None:
"""Push all git tags to the remote."""
subprocess.run(["git", "push", "--tags", "origin", "master"], check=True)


def check_for_clean_working_tree() -> None:
"""Check for a clean git working tree."""
git_status = subprocess.run(
["git", "status", "--porcelain"], capture_output=True, text=True
)
if git_status.stdout:
print(
"Git working tree is not clean. Please commit or stash changes.",
file=sys.stderr,
)
sys.exit(1)


def get_env_github_token() -> str:
"""Check for the GITHUB_TOKEN environment variable."""
github_token = os.environ.get("GITHUB_TOKEN")
if not github_token:
print("GITHUB_TOKEN environment variable not set.", file=sys.stderr)
sys.exit(1)
return github_token


def get_is_test_response() -> str:
"""Ask the user if this is a test release."""
while True:
is_test = input("Is this a test release? (y/n): ")
if is_test in ["y", "n"]:
return is_test
else:
print("Invalid input. Please enter 'y' or 'n.'")


def main() -> None:
"""Run the main program."""
check_for_clean_working_tree()
github_token = get_env_github_token()
is_test = get_is_test_response()
version_number = input("Enter the version number: ")

add_git_tag_for_version(version_number)
remove_previous_dist()
create_build()
verify_build(is_test)
create_github_release_draft(github_token, version_number)


if __name__ == "__main__":
main()
244 changes: 244 additions & 0 deletions tests/__snapshots__/test_release.ambr
Original file line number Diff line number Diff line change
@@ -0,0 +1,244 @@
# serializer version: 1
# name: test_happy_path
dict({
'input': _CallList([
_Call(
'',
tuple(
'Is this a test release? (y/n): ',
),
dict({
}),
),
_Call(
'',
tuple(
'Enter the version number: ',
),
dict({
}),
),
_Call(
'',
tuple(
'Does the build look correct? (y/n): ',
),
dict({
}),
),
]),
'listdir': _CallList([
_Call(
'',
tuple(
'dist',
),
dict({
}),
),
]),
'requests': _CallList([
_Call(
'',
tuple(
'https://api.github.com/repos/john-kurkowski/tldextract/releases/generate-notes',
),
dict({
'headers': dict({
'Accept': 'application/vnd.github+json',
'Authorization': 'Bearer fake-token',
'X-GitHub-Api-Version': '2022-11-28',
}),
'json': dict({
'tag_name': '5.0.1',
}),
}),
),
_Call(
'',
tuple(
'https://api.github.com/repos/john-kurkowski/tldextract/releases',
),
dict({
'headers': dict({
'Accept': 'application/vnd.github+json',
'Authorization': 'Bearer fake-token',
'X-GitHub-Api-Version': '2022-11-28',
}),
'json': dict({
'body': '''
* Bugfixes
* Indicate MD5 not used in a security context (FIPS compliance) ([#309](https://github.com/john-kurkowski/tldextract/issues/309))
* Misc.
* Increase typecheck aggression

**Full Changelog**: fake-body
''',
'draft': True,
'name': '5.0.1',
'prerelease': False,
'tag_name': '5.0.1',
}),
}),
),
]),
'subprocess': _CallList([
_Call(
'',
tuple(
list([
'git',
'status',
'--porcelain',
]),
),
dict({
'capture_output': True,
'text': True,
}),
),
_Call(
'',
tuple(
list([
'git',
'tag',
'-a',
'5.0.1',
'-m',
'5.0.1',
]),
),
dict({
'check': True,
}),
),
_Call(
'',
tuple(
list([
'rm',
'-rf',
PosixPath('dist'),
]),
),
dict({
'check': True,
}),
),
_Call(
'',
tuple(
list([
'python',
'-m',
'build',
]),
),
dict({
'check': True,
}),
),
_Call(
'',
tuple(
list([
'ls',
'-l',
PosixPath('dist'),
]),
),
dict({
'check': True,
}),
),
_Call(
'',
tuple(
list([
'tar',
'tvf',
PosixPath('dist/archive1'),
]),
),
dict({
'check': True,
}),
),
_Call(
'',
tuple(
list([
'tar',
'tvf',
PosixPath('dist/archive2'),
]),
),
dict({
'check': True,
}),
),
_Call(
'',
tuple(
list([
'tar',
'tvf',
PosixPath('dist/archive3'),
]),
),
dict({
'check': True,
}),
),
_Call(
'',
tuple(
list([
'twine',
'upload',
'--repository',
'testpypi',
PosixPath('dist/*'),
]),
),
dict({
'check': True,
}),
),
_Call(
'',
tuple(
list([
'git',
'push',
'--tags',
'origin',
'master',
]),
),
dict({
'check': True,
}),
),
]),
})
# ---
# name: test_happy_path.1
'''
Version 5.0.1 tag added successfully.
Previous dist folder removed successfully.
Build created successfully.
Contents of dist folder:
Contents of tar files in dist folder:
Build verified successfully.
Release created successfully: https://github.com/path/to/release

'''
# ---
# name: test_happy_path.2
'''
WARNING: dist folder contains incorrect number of files.

'''
# ---
32 changes: 18 additions & 14 deletions tests/main_test.py
Original file line number Diff line number Diff line change
@@ -4,6 +4,7 @@

import logging
import os
import sys
import tempfile
from collections.abc import Sequence
from pathlib import Path
@@ -17,7 +18,7 @@
import tldextract
import tldextract.suffix_list
from tldextract.cache import DiskCache
from tldextract.remote import inet_pton, lenient_netloc, looks_like_ip
from tldextract.remote import lenient_netloc, looks_like_ip, looks_like_ipv6
from tldextract.suffix_list import SuffixListNotFound
from tldextract.tldextract import ExtractResult

@@ -152,21 +153,24 @@ def test_lenient_netloc() -> None:
)


@pytest.mark.skipif(not inet_pton, reason="inet_pton unavailable")
def test_looks_like_ip_with_inet_pton() -> None:
"""Test preferred function to check if a string looks like an IP address."""
assert looks_like_ip("1.1.1.1", inet_pton) is True
assert looks_like_ip("a.1.1.1", inet_pton) is False
assert looks_like_ip("1.1.1.1\n", inet_pton) is False
assert looks_like_ip("256.256.256.256", inet_pton) is False
def test_looks_like_ip() -> None:
"""Test function to check if a string looks like an IPv4 address."""
assert looks_like_ip("1.1.1.1") is True
assert looks_like_ip("1.1.1.01") is False
assert looks_like_ip("a.1.1.1") is False
assert looks_like_ip("1.1.1.1\n") is False
assert looks_like_ip("256.256.256.256") is False


def test_looks_like_ip_without_inet_pton() -> None:
"""Test fallback function to check if a string looks like an IP address."""
assert looks_like_ip("1.1.1.1", None) is True
assert looks_like_ip("a.1.1.1", None) is False
assert looks_like_ip("1.1.1.1\n", None) is False
assert looks_like_ip("256.256.256.256", None) is False
def test_looks_like_ipv6() -> None:
"""Test function to check if a string looks like an IPv6 address."""
assert looks_like_ipv6("::") is True
assert looks_like_ipv6("aBcD:ef01:2345:6789:aBcD:ef01:aaaa:2288") is True
assert looks_like_ipv6("aBcD:ef01:2345:6789:aBcD:ef01:127.0.0.1") is True
assert looks_like_ipv6("ZBcD:ef01:2345:6789:aBcD:ef01:127.0.0.1") is False
if sys.version_info >= (3, 8, 12): # noqa: UP036
assert looks_like_ipv6("aBcD:ef01:2345:6789:aBcD:ef01:127.0.0.01") is False
assert looks_like_ipv6("aBcD:ef01:2345:6789:aBcD:") is False


def test_similar_to_ip() -> None:
1 change: 1 addition & 0 deletions tests/test_cache.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
"""Test the caching functionality."""

from __future__ import annotations

import sys
95 changes: 95 additions & 0 deletions tests/test_release.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
"""Test the library maintainer release script."""

from __future__ import annotations

import dataclasses
import sys
from collections.abc import Iterator
from typing import Any
from unittest import mock

import pytest
from syrupy.assertion import SnapshotAssertion

from scripts import release


@dataclasses.dataclass
class Mocks:
"""Collection of all mocked objects used in the release script."""

input: mock.Mock
listdir: mock.Mock
requests: mock.Mock
subprocess: mock.Mock

@property
def mock_calls(self) -> dict[str, Any]:
"""A dict of _all_ calls to this class's mock objects."""
return {
k.name: getattr(self, k.name).mock_calls for k in dataclasses.fields(self)
}


@pytest.fixture
def mocks() -> Iterator[Mocks]:
"""Stub network and subprocesses."""
with mock.patch("builtins.input") as mock_input, mock.patch(
"os.listdir"
) as mock_listdir, mock.patch("requests.post") as mock_requests, mock.patch(
"subprocess.run"
) as mock_subprocess:
yield Mocks(
input=mock_input,
listdir=mock_listdir,
requests=mock_requests,
subprocess=mock_subprocess,
)


@pytest.mark.skipif(
sys.platform == "win32", reason="Snapshot paths are different on Windows"
)
def test_happy_path(
capsys: pytest.CaptureFixture[str],
mocks: Mocks,
monkeypatch: pytest.MonkeyPatch,
snapshot: SnapshotAssertion,
) -> None:
"""Test the release script happy path.
Simulate user input for a typical, existing release.
This one test case covers most lines of the release script, without
actually making network requests or running subprocesses. For an
infrequently used script, this coverage is useful without being too brittle
to change.
"""
monkeypatch.setenv("GITHUB_TOKEN", "fake-token")

mocks.input.side_effect = ["y", "5.0.1", "y"]

mocks.listdir.return_value = ["archive1", "archive2", "archive3"]

def mock_post(*args: Any, **kwargs: Any) -> mock.Mock:
"""Return _one_ response JSON that happens to match expectations for multiple requests."""
return mock.Mock(
json=mock.Mock(
return_value={
"body": "Body start **Full Changelog**: fake-body",
"html_url": "https://github.com/path/to/release",
}
),
)

mocks.requests.side_effect = mock_post

mocks.subprocess.return_value.stdout = ""

release.main()

out, err = capsys.readouterr()

assert mocks.mock_calls == snapshot
assert out == snapshot
assert err == snapshot
1 change: 1 addition & 0 deletions tests/test_trie.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
"""Trie tests."""

from itertools import permutations

from tldextract.tldextract import Trie
1 change: 0 additions & 1 deletion tldextract/__main__.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
"""tldextract __main__."""


from .cli import main

if __name__ == "__main__":
7 changes: 3 additions & 4 deletions tldextract/cache.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
"""Helpers."""

from __future__ import annotations

import errno
@@ -37,8 +38,7 @@ def md5(*args: bytes) -> hashlib._Hash:


def get_pkg_unique_identifier() -> str:
"""
Generate an identifier unique to the python version, tldextract version, and python instance.
"""Generate an identifier unique to the python version, tldextract version, and python instance.
This will prevent interference between virtualenvs and issues that might arise when installing
a new version of tldextract
@@ -65,8 +65,7 @@ def get_pkg_unique_identifier() -> str:


def get_cache_dir() -> str:
"""
Get a cache dir that we have permission to write to.
"""Get a cache dir that we have permission to write to.
Try to follow the XDG standard, but if that doesn't work fallback to the package directory
http://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html
34 changes: 6 additions & 28 deletions tldextract/remote.py
Original file line number Diff line number Diff line change
@@ -3,19 +3,13 @@
from __future__ import annotations

import re
from collections.abc import Callable
from ipaddress import AddressValueError, IPv6Address
from urllib.parse import scheme_chars

inet_pton: Callable[[int, str], bytes] | None
try:
from socket import AF_INET, AF_INET6, inet_pton # Availability: Unix, Windows.
except ImportError:
inet_pton = None

IP_RE = re.compile(
r"^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.)"
r"{3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$"
r"^(?:(?:[0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.)"
r"{3}(?:[0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$",
re.ASCII,
)

scheme_chars_set = set(scheme_chars)
@@ -59,32 +53,16 @@ def _schemeless_url(url: str) -> str:
return url[double_slashes_start + 2 :]


def looks_like_ip(
maybe_ip: str, pton: Callable[[int, str], bytes] | None = inet_pton
) -> bool:
"""Check whether the given str looks like an IP address."""
def looks_like_ip(maybe_ip: str) -> bool:
"""Check whether the given str looks like an IPv4 address."""
if not maybe_ip[0].isdigit():
return False

if pton is not None:
try:
pton(AF_INET, maybe_ip)
return True
except OSError:
return False
return IP_RE.fullmatch(maybe_ip) is not None


def looks_like_ipv6(
maybe_ip: str, pton: Callable[[int, str], bytes] | None = inet_pton
) -> bool:
def looks_like_ipv6(maybe_ip: str) -> bool:
"""Check whether the given str looks like an IPv6 address."""
if pton is not None:
try:
pton(AF_INET6, maybe_ip)
return True
except OSError:
return False
try:
IPv6Address(maybe_ip)
except AddressValueError:
15 changes: 5 additions & 10 deletions tldextract/tldextract.py
Original file line number Diff line number Diff line change
@@ -75,8 +75,7 @@ class ExtractResult:

@property
def registered_domain(self) -> str:
"""
Joins the domain and suffix fields with a dot, if they're both set.
"""Joins the domain and suffix fields with a dot, if they're both set.
>>> extract('http://forums.bbc.co.uk').registered_domain
'bbc.co.uk'
@@ -89,8 +88,7 @@ def registered_domain(self) -> str:

@property
def fqdn(self) -> str:
"""
Returns a Fully Qualified Domain Name, if there is a proper domain/suffix.
"""Returns a Fully Qualified Domain Name, if there is a proper domain/suffix.
>>> extract('http://forums.bbc.co.uk/path/to/file').fqdn
'forums.bbc.co.uk'
@@ -103,8 +101,7 @@ def fqdn(self) -> str:

@property
def ipv4(self) -> str:
"""
Returns the ipv4 if that is what the presented domain/url is.
"""Returns the ipv4 if that is what the presented domain/url is.
>>> extract('http://127.0.0.1/path/to/file').ipv4
'127.0.0.1'
@@ -123,8 +120,7 @@ def ipv4(self) -> str:

@property
def ipv6(self) -> str:
"""
Returns the ipv6 if that is what the presented domain/url is.
"""Returns the ipv6 if that is what the presented domain/url is.
>>> extract('http://[aBcD:ef01:2345:6789:aBcD:ef01:127.0.0.1]/path/to/file').ipv6
'aBcD:ef01:2345:6789:aBcD:ef01:127.0.0.1'
@@ -334,8 +330,7 @@ def update(

@property
def tlds(self, session: requests.Session | None = None) -> list[str]:
"""
Returns the list of tld's used by default.
"""Returns the list of tld's used by default.
This will vary based on `include_psl_private_domains` and `extra_suffixes`
"""
4 changes: 2 additions & 2 deletions tox.ini
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[tox]
envlist = py{38,39,310,311,312,py38},codestyle,lint,typecheck
envlist = py{38,39,310,311,312,py38,py39,py310},codestyle,lint,typecheck

[testenv]
commands = pytest {posargs}
@@ -18,5 +18,5 @@ extras = testing

[testenv:typecheck]
basepython = python3.8
commands = mypy --show-error-codes tldextract tests
commands = mypy --show-error-codes scripts tldextract tests
extras = testing