Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pigar -c requirements.txt download error if lib name contains hyphen or brackets (- or []) #113

Closed
mickdewald opened this issue Oct 26, 2022 · 7 comments · Fixed by #115 or #124
Closed
Labels
Milestone

Comments

@mickdewald
Copy link

Problems for downloading info if lib name contains hypen (- like python-multipart) or brackets ([], like uvicorn[standard])

If I try to list information for libraries like this:

./venv/bin/pigar -c requirements.txt I run into some errors:

❯ ./venv/bin/pigar -c requirements.txt --no
--- Logging error ---
Traceback (most recent call last):
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/core.py", line 248, in check_requirements_latest_versions
    latest = downloader.download_package(pkg).version()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/pypi.py", line 64, in download_package
    pkg_info = self.download(PKG_INFO_URL.format(name))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/pypi.py", line 54, in download
    resp.raise_for_status()
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://pypi.org/pypi/uvicorn%5Bstandard%5D/json

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/logging/__init__.py", line 1110, in emit
    msg = self.format(record)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/logging/__init__.py", line 953, in format
    return fmt.format(record)
           ^^^^^^^^^^^^^^^^^^
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/log.py", line 49, in format
    message = color(record.getMessage())
                    ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/logging/__init__.py", line 377, in getMessage
    msg = msg % self.args
          ~~~~^~~~~~~~~~~
TypeError: must be real number, not HTTPError
Call stack:
  File "/home/mick/projects/capmon-backend/./venv/bin/pigar", line 8, in <module>
    sys.exit(main())
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/__main__.py", line 174, in main
    check_requirements_latest_versions(
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/core.py", line 250, in check_requirements_latest_versions
    logger.error('checking %s failed: %e', pkg, e)
Message: 'checking %s failed: %e'
Arguments: ('uvicorn[standard]', HTTPError('404 Client Error: Not Found for url: https://pypi.org/pypi/uvicorn%5Bstandard%5D/json'))
--- Logging error ---
Traceback (most recent call last):
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/core.py", line 248, in check_requirements_latest_versions
    latest = downloader.download_package(pkg).version()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/pypi.py", line 64, in download_package
    pkg_info = self.download(PKG_INFO_URL.format(name))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/pypi.py", line 54, in download
    resp.raise_for_status()
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://pypi.org/pypi/python-jose%5Bcryptography%5D/json

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/logging/__init__.py", line 1110, in emit
    msg = self.format(record)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/logging/__init__.py", line 953, in format
    return fmt.format(record)
           ^^^^^^^^^^^^^^^^^^
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/log.py", line 49, in format
    message = color(record.getMessage())
                    ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/logging/__init__.py", line 377, in getMessage
    msg = msg % self.args
          ~~~~^~~~~~~~~~~
TypeError: must be real number, not HTTPError
Call stack:
  File "/home/mick/projects/capmon-backend/./venv/bin/pigar", line 8, in <module>
    sys.exit(main())
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/__main__.py", line 174, in main
    check_requirements_latest_versions(
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/core.py", line 250, in check_requirements_latest_versions
    logger.error('checking %s failed: %e', pkg, e)
Message: 'checking %s failed: %e'
Arguments: ('python-jose[cryptography]', HTTPError('404 Client Error: Not Found for url: https://pypi.org/pypi/python-jose%5Bcryptography%5D/json'))
--- Logging error ---
Traceback (most recent call last):
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/core.py", line 248, in check_requirements_latest_versions
    latest = downloader.download_package(pkg).version()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/pypi.py", line 64, in download_package
    pkg_info = self.download(PKG_INFO_URL.format(name))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/pypi.py", line 54, in download
    resp.raise_for_status()
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://pypi.org/pypi/passlib%5Bbcrypt%5D/json

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/logging/__init__.py", line 1110, in emit
    msg = self.format(record)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/logging/__init__.py", line 953, in format
    return fmt.format(record)
           ^^^^^^^^^^^^^^^^^^
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/log.py", line 49, in format
    message = color(record.getMessage())
                    ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/logging/__init__.py", line 377, in getMessage
    msg = msg % self.args
          ~~~~^~~~~~~~~~~
TypeError: must be real number, not HTTPError
Call stack:
  File "/home/mick/projects/capmon-backend/./venv/bin/pigar", line 8, in <module>
    sys.exit(main())
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/__main__.py", line 174, in main
    check_requirements_latest_versions(
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/core.py", line 250, in check_requirements_latest_versions
    logger.error('checking %s failed: %e', pkg, e)
Message: 'checking %s failed: %e'
Arguments: ('passlib[bcrypt]', HTTPError('404 Client Error: Not Found for url: https://pypi.org/pypi/passlib%5Bbcrypt%5D/json'))
 ==============================================
  PACKAGE                   | CURRENT | LATEST
  --------------------------+---------+-------
  fastapi                   | 0.85.0  | 0.85.1
  h11                       | 0.12.0  | 0.14.0
  pydantic                  | 1.10.2  | 1.10.2
  starlette                 | 0.20.4  | 0.21.0
  uvicorn[standard]         |         | 0.21.0
  python-socketio           |         | 5.7.2
  websockets                | 10.4    | 10.4
  aiohttp                   | 3.8.3   | 3.8.3
  httpx                     | 0.23.0  | 0.23.0
  python-jose[cryptography] |         | 0.23.0
  passlib[bcrypt]           |         | 0.23.0
  bcrypt                    | 4.0.1   | 4.0.1
  yahooquery                | 2.2.15  | 2.2.15
  yfinance                  | 0.1.84  | 0.1.84
  html5lib                  | 1.1     | 1.1
  pycoingecko               | 3.0.0   | 3.0.0
  beautifulsoup4            | 4.11.1  | 4.11.1
  cloudscraper              | 1.2.64  | 1.2.64
  motor                     | 3.1.1   | 3.1.1
  pymongo                   | 4.3.2   | 4.3.2
  beanie                    | 1.13.0  | 1.13.0
  kubernetes                | 25.3.0  | 25.3.0
  pytest                    | 7.2.0   | 7.2.0
  pytest-asyncio            |         | 0.20.1
  locust                    | 2.12.2  | 2.12.2
  loguru                    | 0.6.0   | 0.6.0
  autopep8                  | 1.7.0   | 1.7.0
  pylint                    | 2.15.5  | 2.15.5
  python-multipart          |         | 0.0.5
  email-validator           |         | 1.3.0
  configparser              | 5.3.0   | 5.3.0
  apscheduler               |         | 3.9.1
  Pillow                    | 9.2.0   | 9.2.0
  async_lru                 | 1.0.3   | 1.0.3
  colorama                  | 0.4.6   | 0.4.6
  pigar                     | 1.0.0   | 1.0.0
 ==============================================
@damnever
Copy link
Owner

damnever commented Nov 7, 2022

Try the main branch pip install git+https://github.com/damnever/pigar.git@main --upgrade

@mickdewald
Copy link
Author

mickdewald commented Nov 7, 2022

Thanks for answering. I will test this a soon I have access to my PC.
Is there a way to make this work in requirements.txt? Like
pigar.git@main?

EDIT
I think I found a good solution here
https://stackoverflow.com/a/35998253/7220665

I use this in my project to print out the libs including versions as some debug information

# https://capmon.org/api/server

{"message":"Welcome to the capmon-server v86bfabe1_2022-11-05_19:04:34","python":"3.11.0","libs":
{"fastapi":{"current":"0.85.1","latest":"0.86.0"},
"h11":{"current":"0.12.0","latest":"0.14.0"},
"pydantic":{"current":"1.10.2","latest":"1.10.2"},
...


@mickdewald
Copy link
Author

I tested your suggested solution, and here is my result:

  PACKAGE          | SPEC     | LOCAL  | LATEST
  -----------------+----------+--------+-------
  fastapi          | ==0.85.1 | 0.85.1 | 0.86.0
  h11              | ==0.12.0 | 0.12.0 | 0.14.0
  pydantic         |          | 1.10.2 | 1.10.2
  starlette        | ==0.20.4 | 0.20.4 | 0.21.0
  uvicorn          |          | 0.19.0 | 0.19.0
  python-socketio  |          |        | 5.7.2
  websockets       |          | 10.4   | 10.4
  aiohttp          |          | 3.8.3  | 3.8.3
  httpx            |          | 0.23.0 | 0.23.0
  python-jose      |          |        | 3.3.0
  passlib          |          | 1.7.4  | 1.7.4
  bcrypt           |          | 4.0.1  | 4.0.1
  yahooquery       |          | 2.2.15 | 2.2.15
  yfinance         |          | 0.1.85 | 0.1.85
  html5lib         |          | 1.1    | 1.1
  pycoingecko      |          | 3.1.0  | 3.1.0
  beautifulsoup4   |          | 4.11.1 | 4.11.1
  cloudscraper     |          | 1.2.64 | 1.2.64
  motor            |          | 3.1.1  | 3.1.1
  pymongo          |          | 4.3.2  | 4.3.2
  beanie           |          | 1.15.1 | 1.15.1
  kubernetes       |          | 25.3.0 | 25.3.0
  pytest           |          | 7.2.0  | 7.2.0
  pytest-asyncio   |          |        | 0.20.1
  locust           |          | 2.13.0 | 2.13.0
  loguru           |          | 0.6.0  | 0.6.0
  autopep8         |          | 2.0.0  | 2.0.0
  pylint           |          | 2.15.5 | 2.15.5
  python-multipart |          |        | 0.0.5
  email-validator  |          |        | 1.3.0
  configparser     |          | 5.3.0  | 5.3.0
  apscheduler      |          |        | 3.9.1
  Pillow           |          | 9.3.0  | 9.3.0
  async_lru        |          | 1.0.3  | 1.0.3
  colorama         |          | 0.4.6  | 0.4.6
 ===============================================

The lib definitions including brackets seem to be working now, but the ones with hyphens do not show a version number and there are still exceptions:

--- Logging error ---
Traceback (most recent call last):
  File "/root/.local/lib/python3.11/site-packages/packaging/requirements.py", line 102, in __init__
    req = REQUIREMENT.parseString(requirement_string)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/lib/python3.11/site-packages/pyparsing/core.py", line 1141, in parse_string
    raise exc.with_traceback(None)
pyparsing.exceptions.ParseException: Expected string_end, found '+'  (at char 3), (line:1, col:4)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.local/lib/python3.11/site-packages/pigar/helpers.py", line 131, in parse_requirements
    req = Requirement(line)
          ^^^^^^^^^^^^^^^^^
  File "/root/.local/lib/python3.11/site-packages/packaging/requirements.py", line 104, in __init__
    raise InvalidRequirement(
packaging.requirements.InvalidRequirement: Parse error at "'+https:/'": Expected string_end

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.local/lib/python3.11/site-packages/pigar/core.py", line 241, in check_requirements_latest_versions
    for req in parse_requirements(file):
  File "/root/.local/lib/python3.11/site-packages/pigar/helpers.py", line 133, in parse_requirements
    raise PraseRequirementError('line {}: {}'.format(lineno, e))
pigar.helpers.PraseRequirementError: line 226: Parse error at "'+https:/'": Expected string_end

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/logging/__init__.py", line 1110, in emit
    msg = self.format(record)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/logging/__init__.py", line 953, in format
    return fmt.format(record)
           ^^^^^^^^^^^^^^^^^^
  File "/root/.local/lib/python3.11/site-packages/pigar/log.py", line 49, in format
    message = color(record.getMessage())
                    ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/logging/__init__.py", line 377, in getMessage
    msg = msg % self.args
          ~~~~^~~~~~~~~~~
TypeError: must be real number, not PraseRequirementError
Call stack:
  File "/root/.local/bin/pigar", line 8, in <module>
    sys.exit(main())
  File "/root/.local/lib/python3.11/site-packages/pigar/__main__.py", line 174, in main
    check_requirements_latest_versions(
  File "/root/.local/lib/python3.11/site-packages/pigar/core.py", line 256, in check_requirements_latest_versions
    logger.error('parse %s failed: %e', file, e)
Message: 'parse %s failed: %e'
Arguments: ('/server/requirements.txt', PraseRequirementError('line 226: Parse error at "\'+https:/\'": Expected string_end'))

@damnever
Copy link
Owner

damnever commented Nov 8, 2022

Looks like there are some invalid lines in requirements.txt, could you post your requirements.txt file here?

@damnever damnever reopened this Nov 8, 2022
@mickdewald
Copy link
Author

So here is the requirements.txt:

fastapi==0.86.0

h11==0.12.0

pydantic==1.10.2

starlette==0.20.4

uvicorn[standard]==0.19

python-socketio==5.7.2

websockets==10.4

aiohttp==3.8.3

httpx==0.23.0

python-jose[cryptography]==3.3.0

passlib[bcrypt]==1.7.4

bcrypt==4.0.1

yahooquery==2.2.15

yfinance==0.1.84

html5lib==1.1

pycoingecko==3.1.0

beautifulsoup4==4.11.1

cloudscraper==1.2.64

motor==3.1.1

pymongo==4.3.2

beanie==1.15.1

kubernetes==25.3.0

pytest==7.2.0

pytest-asyncio==0.20.1

locust==2.12.0

loguru==0.6.0

autopep8==2.0.0

pylint==2.15.5

python-multipart==0.0.5

email-validator==1.3.0

configparser==5.3.0

apscheduler==3.9.1

Pillow==9.3.0

async_lru==1.0.3

colorama==0.4.6

git+https://github.com/damnever/pigar.git@main

And this is the error message:

--- Logging error ---
Traceback (most recent call last):
  File "/root/.local/lib/python3.11/site-packages/packaging/requirements.py", line 102, in __init__
    req = REQUIREMENT.parseString(requirement_string)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/lib/python3.11/site-packages/pyparsing/core.py", line 1141, in parse_string
    raise exc.with_traceback(None)
pyparsing.exceptions.ParseException: Expected string_end, found '+'  (at char 3), (line:1, col:4)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.local/lib/python3.11/site-packages/pigar/helpers.py", line 131, in parse_requirements
    req = Requirement(line)
          ^^^^^^^^^^^^^^^^^
  File "/root/.local/lib/python3.11/site-packages/packaging/requirements.py", line 104, in __init__
    raise InvalidRequirement(
packaging.requirements.InvalidRequirement: Parse error at "'+https:/'": Expected string_end

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.local/lib/python3.11/site-packages/pigar/core.py", line 241, in check_requirements_latest_versions
    for req in parse_requirements(file):
  File "/root/.local/lib/python3.11/site-packages/pigar/helpers.py", line 133, in parse_requirements
    raise PraseRequirementError('line {}: {}'.format(lineno, e))
pigar.helpers.PraseRequirementError: line 70: Parse error at "'+https:/'": Expected string_end

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/logging/__init__.py", line 1110, in emit
    msg = self.format(record)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/logging/__init__.py", line 953, in format
    return fmt.format(record)
           ^^^^^^^^^^^^^^^^^^
  File "/root/.local/lib/python3.11/site-packages/pigar/log.py", line 49, in format
    message = color(record.getMessage())
                    ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/logging/__init__.py", line 377, in getMessage
    msg = msg % self.args
          ~~~~^~~~~~~~~~~
TypeError: must be real number, not PraseRequirementError
Call stack:
  File "/root/.local/bin/pigar", line 8, in <module>
    sys.exit(main())
  File "/root/.local/lib/python3.11/site-packages/pigar/__main__.py", line 174, in main
    check_requirements_latest_versions(
  File "/root/.local/lib/python3.11/site-packages/pigar/core.py", line 256, in check_requirements_latest_versions
    logger.error('parse %s failed: %e', file, e)
Message: 'parse %s failed: %e'
Arguments: ('/server/requirements.txt', PraseRequirementError('line 70: Parse error at "\'+https:/\'": Expected string_end'))

But nevertheless, there is an output:

  PACKAGE          | SPEC     | LOCAL  | LATEST
  -----------------+----------+--------+-------
  fastapi          | ==0.86.0 | 0.86.0 | 0.86.0
  h11              | ==0.12.0 | 0.12.0 | 0.14.0
  pydantic         | ==1.10.2 | 1.10.2 | 1.10.2
  starlette        | ==0.20.4 | 0.20.4 | 0.21.0
  uvicorn          | ==0.19   | 0.19.0 | 0.19.0
  python-socketio  | ==5.7.2  |        | 5.7.2
  websockets       | ==10.4   | 10.4   | 10.4
  aiohttp          | ==3.8.3  | 3.8.3  | 3.8.3
  httpx            | ==0.23.0 | 0.23.0 | 0.23.0
  python-jose      | ==3.3.0  |        | 3.3.0
  passlib          | ==1.7.4  | 1.7.4  | 1.7.4
  bcrypt           | ==4.0.1  | 4.0.1  | 4.0.1
  yahooquery       | ==2.2.15 | 2.2.15 | 2.2.15
  yfinance         | ==0.1.84 | 0.1.84 | 0.1.85
  html5lib         | ==1.1    | 1.1    | 1.1
  pycoingecko      | ==3.1.0  | 3.1.0  | 3.1.0
  beautifulsoup4   | ==4.11.1 | 4.11.1 | 4.11.1
  cloudscraper     | ==1.2.64 | 1.2.64 | 1.2.64
  motor            | ==3.1.1  | 3.1.1  | 3.1.1
  pymongo          | ==4.3.2  | 4.3.2  | 4.3.2
  beanie           | ==1.15.1 | 1.15.1 | 1.15.1
  kubernetes       | ==25.3.0 | 25.3.0 | 25.3.0
  pytest           | ==7.2.0  | 7.2.0  | 7.2.0
  pytest-asyncio   | ==0.20.1 |        | 0.20.1
  locust           | ==2.12.0 | 2.12.0 | 2.13.0
  loguru           | ==0.6.0  | 0.6.0  | 0.6.0
  autopep8         | ==2.0.0  | 2.0.0  | 2.0.0
  pylint           | ==2.15.5 | 2.15.5 | 2.15.5
  python-multipart | ==0.0.5  |        | 0.0.5
  email-validator  | ==1.3.0  |        | 1.3.0
  configparser     | ==5.3.0  | 5.3.0  | 5.3.0
  apscheduler      | ==3.9.1  |        | 3.9.1
  Pillow           | ==9.3.0  | 9.3.0  | 9.3.0
  async_lru        | ==1.0.3  | 1.0.3  | 1.0.3
  colorama         | ==0.4.6  | 0.4.6  | 0.4.6
 ===============================================

Maybe the reason for the error message is the pigar requirement itself, since it's not part of the table?

@damnever
Copy link
Owner

You can remove git+https://github.com/damnever/pigar.git@main to fix the issue for now..

In the case like git+https://github.com/damnever/pigar.git@main, the first step may be to download the package, then to parse information from it, there is much work to be done and done it right.

@mickdewald
Copy link
Author

You can remove git+https://github.com/damnever/pigar.git@main to fix the issue for now..

I just use the old version for now and live with the error message. It's just for debugging, so that's fine right now

In the case like git+https://github.com/damnever/pigar.git@main, the first step may be to download the package, then to parse information from it, there is much work to be done and done it right.

I am looking forward to upcoming versions.
Thanks for this great tool!

@damnever damnever added this to the v2 milestone Nov 12, 2022
@damnever damnever linked a pull request Nov 26, 2022 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
2 participants