Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refresh part 2 #581

Merged
merged 38 commits into from
Dec 24, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
b4c9c4b
:art: move charset_normalizer into a src directory
Ousret Dec 24, 2024
5257d5e
:fire: remove .codecov.yml in favor of local coverage
Ousret Dec 24, 2024
540c839
:fire: remove bin/integration.py and bin/serve.py
Ousret Dec 24, 2024
b0aecd8
:fire: remove setup.cfg in favor of pyproject.toml
Ousret Dec 24, 2024
1775d46
:pencil: update copyright in docs + email
Ousret Dec 24, 2024
1bcb2cb
:fire: remove shebang in bin/*
Ousret Dec 24, 2024
ada467f
:pencil: replace "chaos" in --threshold help by "noise"
Ousret Dec 24, 2024
4a84793
:fire: remove commented code about "logging.addLevelName"
Ousret Dec 24, 2024
e64eebd
:wrench: add complete coverage configuration
Ousret Dec 24, 2024
47cc1c6
:fire: remove unused "range_scan" util function
Ousret Dec 24, 2024
5b19afb
:wrench: exclude some "non-production" and defensive behavior in code…
Ousret Dec 24, 2024
dc5d1eb
:sparkle: add noxfile
Ousret Dec 24, 2024
d22eb70
:pencil: update readme with fresh data
Ousret Dec 24, 2024
48229e9
:wrench: exclude lines where ensure handler cleanup (logging/defensiv…
Ousret Dec 24, 2024
dc5e4cb
:wrench: adjust workflows with added noxfile
Ousret Dec 24, 2024
a690b15
:wrench: display environment details in coverage nox session
Ousret Dec 24, 2024
54127ca
:pencil: update changelog
Ousret Dec 24, 2024
01eb8af
:wrench: update MANIFEST.in
Ousret Dec 24, 2024
e235e10
:fire: remove old guide UPGRADE.md (from v1 to v2)
Ousret Dec 24, 2024
6558c8a
:wrench: update pre-commit config
Ousret Dec 24, 2024
53b7931
:art: reformat files
Ousret Dec 24, 2024
6e3e7ab
:pencil: fix typo in readme
Ousret Dec 24, 2024
32d1cd3
:wrench: fix deprecation warning "'count' is passed as positional arg…
Ousret Dec 24, 2024
7ec1fe2
:wrench: fix posargs nox in CI
Ousret Dec 24, 2024
ef35d97
:arrow_up: support pypi attestation via updated pypa/gh-action-pypi-p…
Ousret Dec 24, 2024
f8b55de
:wrench: fix bad generator annotatio in utils.cut_sequence_chunks fun…
Ousret Dec 24, 2024
070a1ff
:wrench: set coverage min accepted to 92%
Ousret Dec 24, 2024
691ae03
:art: reformat files
Ousret Dec 24, 2024
0ebfccf
:art: reformat files
Ousret Dec 24, 2024
ec99326
Merge branch 'master' into refresh-part-2
Ousret Dec 24, 2024
3582e62
:pencil: update utils.iana_name function to reflect his purpose
Ousret Dec 24, 2024
1b06bc0
Merge branch 'refresh-part-2' of github.com:jawah/charset_normalizer …
Ousret Dec 24, 2024
14b4649
:bug: output(...) replace declarative mark using non iana compliant e…
Ousret Dec 24, 2024
1ec7c06
:wrench: update changelog
Ousret Dec 24, 2024
f11f571
:wrench: include noxfile in sdist
Ousret Dec 24, 2024
deed205
:wrench: update LICENSE copyright
Ousret Dec 24, 2024
7ce1ef1
:wrench: use ubuntu-22.04 for cibuildwheel in continuous deployment w…
Ousret Dec 24, 2024
f9d6b8c
:lock: add CODEOWNERS
Ousret Dec 24, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 0 additions & 8 deletions .codecov.yml

This file was deleted.

28 changes: 27 additions & 1 deletion .coveragerc
Original file line number Diff line number Diff line change
@@ -1,2 +1,28 @@
[run]
source=charset_normalizer
source =
charset_normalizer
# Needed for Python 3.11 and lower
disable_warnings = no-sysmon

[paths]
source =
src/charset_normalizer
*/charset_normalizer
*\charset_normalizer

[report]
omit =
src/charset_normalizer/__main__.py

exclude_lines =
except ModuleNotFoundError:
except ImportError:
pass
import
raise NotImplementedError
.* # Platform-specific.*
.*:.* # Python \d.*
.* # Abstract
.* # Defensive:
if (?:typing.)?TYPE_CHECKING:
^\s*?\.\.\.\s*$
9 changes: 9 additions & 0 deletions .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Restrict all files related to deploying to
# require lead maintainer approval.

.github/workflows/ @Ousret
.github/CODEOWNERS @Ousret
src/charset_normalizer/ @Ousret
pyproject.toml @Ousret
tests/ @Ousret
data/ @Ousret
13 changes: 5 additions & 8 deletions .github/workflows/cd.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
with:
python-version: '3.11'
python-version: '3'
- name: Update pip, install build
run: |
python -m pip install --upgrade pip
Expand All @@ -46,7 +46,7 @@ jobs:
needs: pre_flight_check
strategy:
matrix:
os: [ ubuntu-latest, windows-latest, macos-13 ]
os: [ ubuntu-22.04, windows-latest, macos-13 ]
qemu: [ '' ]
include:
# Split ubuntu job for the sake of speed-up
Expand Down Expand Up @@ -77,18 +77,15 @@ jobs:
shell: bash
- name: Setup Python
uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
- name: Update pip, wheel, setuptools, build, twine
run: |
python -m pip install -U pip wheel setuptools build twine
- name: Build wheels
uses: pypa/cibuildwheel@ee63bf16da6cddfb925f542f2c7b59ad50e93969 # v2.22.0
env:
CIBW_BUILD_FRONTEND: build
CIBW_ARCHS_MACOS: x86_64 arm64 universal2
CIBW_ARCHS_MACOS: universal2
CIBW_ENVIRONMENT: CHARSET_NORMALIZER_USE_MYPYC='1'
CIBW_TEST_REQUIRES: pytest
CIBW_TEST_COMMAND: pytest -c {package} {package}/tests
CIBW_SKIP: pp* cp36*
CIBW_SKIP: pp*
- name: Upload artifacts
uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce
with:
Expand Down Expand Up @@ -152,7 +149,7 @@ jobs:
run: |
tree dist
- name: Publish package distributions to PyPI
uses: pypa/gh-action-pypi-publish@f7600683efdcb7656dec5b29656edb7bc586e597 # release/v1
uses: pypa/gh-action-pypi-publish@67339c736fd9354cd4f8cb0b744f2b82a74b5c70 # release/v1
- name: Upload dists to GitHub Release
env:
GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
Expand Down
231 changes: 104 additions & 127 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,19 +19,15 @@ jobs:
- name: Set up Python
uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
with:
python-version: '3.11'
- name: Install dependencies
run: |
python -m pip install -U pip setuptools
python -m pip install -r dev-requirements.txt
python -m pip uninstall -y charset-normalizer
python-version: '3'
- name: Install nox
run: python -m pip install nox
- name: Pre-commit checks
run: |
pre-commit run --all
run: nox -s lint

tests:
name: ✅ Tests
runs-on: ubuntu-latest
runs-on: ubuntu-22.04

strategy:
fail-fast: false
Expand All @@ -53,18 +49,15 @@ jobs:
python-version: ${{ matrix.python-version }}
allow-prereleases: true
- name: Install dependencies
run: |
python -m pip install -U pip setuptools
python -m pip install -r dev-requirements.txt
python -m pip uninstall -y charset-normalizer
- name: Install the package
run: |
python -m build
python -m pip install ./dist/*.whl
run: python -m pip install nox
- name: Run tests
run: |
pytest
- uses: codecov/codecov-action@4fe8c5f003fae66aa5ebb77cfd3e7bfbbda0b6b0 # v3.1.5
run: nox -s test-${{ matrix.python-version }}
- name: "Upload artifact"
uses: "actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce"
with:
name: coverage-data
path: ".coverage.*"
if-no-files-found: error

detection_coverage:

Expand All @@ -79,61 +72,45 @@ jobs:
- name: Set up Python
uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
with:
python-version: '3.11'
python-version: '3'
- name: Install dependencies
run: |
python -m pip install -U pip setuptools
python -m pip install -r dev-requirements.txt
python -m pip uninstall -y charset-normalizer
- name: Install the package
run: |
python -m build
python -m pip install ./dist/*.whl
- name: Clone the complete dataset
run: |
git clone https://github.com/Ousret/char-dataset.git
run: python -m pip install nox
- name: Coverage WITH preemptive
run: |
python ./bin/coverage.py --coverage 97 --with-preemptive
run: nox -s coverage -- --coverage 97 --with-preemptive
- name: Coverage WITHOUT preemptive
run: |
python ./bin/coverage.py --coverage 95

# integration_test:
#
# needs:
# - tests
#
# name: 🔗 Integration Tests
# runs-on: ubuntu-latest
#
# steps:
# - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
# - name: Set up Python
# uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
# with:
# python-version: '3.11'
# - name: Install dependencies
# run: |
# pip install -U pip setuptools
# pip install -r dev-requirements.txt
# - name: Remove Chardet & Charset-Normalizer
# run: |
# pip uninstall -y chardet
# pip uninstall -y charset-normalizer
# - name: Install the package
# run: |
# python -m build
# pip install ./dist/*.whl
# - name: Clone the complete dataset
# run: |
# git clone https://github.com/Ousret/char-dataset.git
# - name: Start the Flask server
# run: |
# python ./bin/serve.py &
# - name: Integration Tests with Requests
# run: |
# python ./bin/integration.py
run: nox -s coverage -- --coverage 95
- name: "Upload artifact"
uses: "actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce"
with:
name: coverage-data
path: ".coverage.*"
if-no-files-found: error

integration_test:

needs:
- tests

name: 🔗 Integration Tests
runs-on: ubuntu-latest

strategy:
fail-fast: false
matrix:
downstream_project:
- niquests
- requests

steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- name: Set up Python
uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
with:
python-version: '3'
- name: Install dependencies
run: pip install nox
- name: Integration Tests with Requests
run: nox -s downstream_${{ matrix.downstream_project }}

chardet_bc:

Expand All @@ -145,22 +122,11 @@ jobs:
- name: Set up Python
uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
with:
python-version: '3.11'
python-version: '3'
- name: Install dependencies
run: |
python -m pip install -U pip setuptools
python -m pip install -r dev-requirements.txt
python -m pip uninstall -y charset-normalizer
- name: Install the package
run: |
python -m build
python -m pip install ./dist/*.whl
- name: Clone the complete dataset
run: |
git clone https://github.com/Ousret/char-dataset.git
run: pip install nox
- name: BC Coverage
run: |
python ./bin/bc.py --coverage 80
run: nox -s backward_compatibility -- --coverage 80

mypyc_test:

Expand All @@ -184,7 +150,7 @@ jobs:
os: [ ubuntu-latest, macos-latest, windows-latest ]
include:
- python-version: "3.7"
os: ubuntu-latest
os: ubuntu-22.04
- python-version: "3.7"
os: macos-13
- python-version: "3.7"
Expand All @@ -198,55 +164,66 @@ jobs:
with:
python-version: ${{ matrix.python-version }}
allow-prereleases: true
- name: Install dependencies
run: |
python -m pip install -U pip setuptools
python -m pip install -r dev-requirements.txt
python -m pip uninstall -y charset-normalizer
- name: Install the package
env:
CHARSET_NORMALIZER_USE_MYPYC: '1'
run: |
python -m pip install .
- name: Clone the complete dataset
run: |
git clone https://github.com/Ousret/char-dataset.git
- name: Coverage WITH preemptive
run: |
python ./bin/coverage.py --coverage 97 --with-preemptive
- name: Performance (Normal)
- name: Install nox
run: pip install nox
- name: Run tests with mypyc enabled
run: nox -s test_mypyc-${{ matrix.python-version }}
- name: "Upload artifact"
uses: "actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce"
with:
name: coverage-data
path: ".coverage.*"
if-no-files-found: error

coverage:
if: always()
runs-on: "ubuntu-latest"
needs:
- tests
- mypyc_test
- detection_coverage
steps:
- name: "Checkout repository"
uses: "actions/checkout@d632683dd7b4114ad314bca15554477dd762a938"

- name: "Setup Python"
uses: "actions/setup-python@f677139bbe7f9c59b41e40162b753c062f5d49a3"
with:
python-version: "3.x"

- name: "Install coverage"
run: "python -m pip install --upgrade coverage"

- name: "Download artifact"
uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
with:
name: coverage-data

- name: "Combine & check coverage"
run: |
python ./bin/performance.py
python -m coverage combine
python -m coverage html --skip-covered --skip-empty
python -m coverage report --ignore-errors --show-missing --fail-under=92

- name: "Upload report"
uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce
with:
name: coverage-report
path: htmlcov

performance:
name: ⚡ Performance Test (no MypyC)
name: ⚡ Performance Test
runs-on: ubuntu-latest

needs:
- mypyc_test
- chardet_bc
needs: coverage

steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- name: Set up Python
uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
with:
python-version: '3.11'
python-version: '3'
- name: Install dependencies
run: |
python -m pip install -U pip setuptools
python -m pip install -r dev-requirements.txt
python -m pip uninstall -y charset-normalizer
- name: Install the package
run: |
python -m build
python -m pip install ./dist/*.whl
- name: Clone the complete dataset
run: |
git clone https://github.com/Ousret/char-dataset.git
- name: Performance (Normal)
run: |
python ./bin/performance.py
- name: Performance (Medium)
run: |
python ./bin/performance.py --size-increase 2
run: pip install nox
- name: Performance Measurement
run: nox -s performance
2 changes: 1 addition & 1 deletion .github/workflows/codeql.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ jobs:
uses: github/codeql-action/init@48ab28a6f5dbc2a99bf1e0131198dd8f1df78169 # v3.28.0
with:
languages: ${{ matrix.language }}

# Autobuild attempts to build any compiled languages (C/C++, C#, Go, or Java).
# If this step fails, then you should remove it and run the build manually (see below)
- name: Autobuild
Expand Down
Loading
Loading