Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update on lock with acryl-datahub fails with cryptic error message #2324

Closed
cburroughs opened this issue Jan 12, 2024 · 4 comments · Fixed by #2325
Closed

update on lock with acryl-datahub fails with cryptic error message #2324

cburroughs opened this issue Jan 12, 2024 · 4 comments · Fixed by #2325
Assignees
Labels

Comments

@cburroughs
Copy link
Collaborator

$ python3.10 -m pex.cli lock create '--indent=2'  --manylinux manylinux2014 --interpreter-constraint $'CPython==3.10.*' --style=universal --pip-version=23.3.2 --resolver-version pip-2020-resolver --target-system=linux --target-system=mac  --prefer-binary -o tmp/ad-lock.json acryl-datahub==0.12.0.5
$ pex3 lock update '--indent=2'  --manylinux manylinux2014  --project=pydot tmp/ad-lock.json
...
Encountered 1 error updating tmp/ad-lock.json:
1.) cp311-cp311-manylinux_2_37_x86_64: 

To get some more details with PEX_VERBOSE=10:

Traceback (most recent call last):
  File "/home/ecsb/.local/lib/python3.11/site-packages/pex/result.py", line 105, in catch
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/ecsb/.local/lib/python3.11/site-packages/pex/resolve/lockfile/updater.py", line 320, in update_resolve
    assert updated_requirement.artifact == locked_requirement.artifact
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
Encountered 1 error updating tmp/ad-lock.json:
1.) cp311-cp311-manylinux_2_37_x86_64: 

With some dinosaur print debugging:

                assert updated_pin == original_pin
                if updated_requirement.artifact != locked_requirement.artifact:
                    print('up ',  updated_requirement, 'lock', locked_requirement)
                assert updated_requirement.artifact == locked_requirement.artifact
up  LockedRequirement(pin=Pin(project_name=ProjectName(raw='avro', normalized='avro'), version=Version(raw='1.11.3', normalized='1.11.3')), artifact=FileArtifact(url='https://files.pythonhosted.org/packages/58/5b/41d2dab437adbad4077acba55b05064c5eb0aea8e77145a0379564d32950/avro-1.11.3.tar.gz', fingerprint=Fingerprint(algorithm='sha256', hash='3393bb5139f9cf0791d205756ce1e39a5b58586af5b153d6a3b5a199610e9d17'), verified=True, filename='avro-1.11.3.tar.gz'), requires_dists=(Requirement(name='python-snappy', url=None, extras=frozenset(), specifier=<SpecifierSet('')>, marker=<Marker('extra == "snappy"')>), Requirement(name='typing-extensions', url=None, extras=frozenset(), specifier=<SpecifierSet('')>, marker=<Marker('python_version < "3.8"')>), Requirement(name='zstandard', url=None, extras=frozenset(), specifier=<SpecifierSet('')>, marker=<Marker('extra == "zstandard"')>)), requires_python=<SpecifierSet('>=3.6')>, additional_artifacts=()) lock LockedRequirement(pin=Pin(project_name=ProjectName(raw='avro', normalized='avro'), version=Version(raw='1.11.3', normalized='1.11.3')), artifact=FileArtifact(url='https://files.pythonhosted.org/packages/58/5b/41d2dab437adbad4077acba55b05064c5eb0aea8e77145a0379564d32950/avro-1.11.3.tar.gz', fingerprint=Fingerprint(algorithm='sha256', hash='3393bb5139f9cf0791d205756ce1e39a5b58586af5b153d6a3b5a199610e9d17'), verified=False, filename='avro-1.11.3.tar.gz'), requires_dists=(Requirement(name='python-snappy', url=None, extras=frozenset(), specifier=<SpecifierSet('')>, marker=<Marker('extra == "snappy"')>), Requirement(name='typing-extensions', url=None, extras=frozenset(), specifier=<SpecifierSet('')>, marker=<Marker('python_version < "3.8"')>), Requirement(name='zstandard', url=None, extras=frozenset(), specifier=<SpecifierSet('')>, marker=<Marker('extra == "zstandard"')>)), requires_python=<SpecifierSet('>=3.6')>, additional_artifacts=())

Which I believe simplifies to:

$ diff -u tmp/up tmp/lock
--- tmp/up	2024-01-12 10:17:17.372343836 -0500
+++ tmp/lock	2024-01-12 10:17:25.575630776 -0500
@@ -5,7 +5,7 @@
 artifact=FileArtifact(url='https://files.pythonhosted.org/packages/58/5b/41d2dab437adbad4077acba55b05064c5eb0aea8e77145a0379564d32950/avro-1.11.3.tar.gz', 
 fingerprint=Fingerprint(algorithm='sha256', 
 hash='3393bb5139f9cf0791d205756ce1e39a5b58586af5b153d6a3b5a199610e9d17'), 
-verified=True, 
+verified=False, 
 filename='avro-1.11.3.tar.gz'), 
 requires_dists=(Requirement(name='python-snappy', 
 url=None, 

Which I think shows why the assertion was triggered, but I still don't see what about this package is tickling the verified

  • Pex versions tried: 2.1.152, 2.1.156, and main
  • avro==1.11.3 instead of acryl-datahub==0.12.0.5 in the original create also fails in the same way.
@jsirois jsirois added the bug label Jan 12, 2024
@jsirois jsirois self-assigned this Jan 12, 2024
@jsirois
Copy link
Member

jsirois commented Jan 12, 2024

@cburroughs I won't have time to work on this until tomorrow afternoon. The verified flag indicates whether the artifact hash has been calculated by Pex itself (verified=True), or taken as given (verified=False); i.e.: taken from a PyPI index artifact URL or JSON API metadata.

@jsirois
Copy link
Member

jsirois commented Jan 12, 2024

Noting this fixes:

$ git diff
diff --git a/pex/resolve/locked_resolve.py b/pex/resolve/locked_resolve.py
index 241cb837..c4421063 100644
--- a/pex/resolve/locked_resolve.py
+++ b/pex/resolve/locked_resolve.py
@@ -148,7 +148,7 @@ class Artifact(object):

     url = attr.ib()  # type: str
     fingerprint = attr.ib()  # type: Fingerprint
-    verified = attr.ib()  # type: bool
+    verified = attr.ib(eq=False)  # type: bool

     def __lt__(self, other):
         # type: (Any) -> bool

An integration test is needed to prevent backslide though as well as a note explaining why this is right to ignore in __eq__.

jsirois added a commit to jsirois/pex that referenced this issue Jan 13, 2024
Previously, a `pex3 lock update` would fail whenever bystander projects
(those projects in the lock but not targeted for update via `-p`) had an
sdist primary artifact.

Fixes pex-tool#2324
jsirois added a commit that referenced this issue Jan 14, 2024
Previously, a `pex3 lock update` would fail whenever bystander projects
(those projects in the lock but not targeted for update via `-p`) had an
sdist primary artifact.

Fixes #2324
@jsirois
Copy link
Member

jsirois commented Jan 14, 2024

@cburroughs thanks for using the lock update feature and the bug report. the fix for this is now released in Pex 2.1.157: https://github.com/pantsbuild/pex/releases/tag/v2.1.157

@cburroughs
Copy link
Collaborator Author

Thanks for the quick fix!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

2 participants