Build fails/incomplete on requirements.txt schema not followed correctly #2410
Comments
Github-dm-CDE
changed the title
|
The current workaround for us is to use the older v1.7.1 version of |
|
@Github-dm-CDE can you please provide the exact Pex command line you use to build the PEX? Normally, instead of doing a poetry export then Pex build using the exported requirements, you'd just say |
|
@Github-dm-CDE I just looked at Pex's requirements.txt parsing code and and it would seem to handle Lines 264 to 269 in 36dd237 @Github-dm-CDE are you building the PEX using Pants? If so, that's where the bug is. If you can confirm you're using Pants I'll close this as an answered question and you'll have to raise an issue with the Pants project; otherwise, please provide the Pex command line you use and I'll repro and debug. |
We are building our pex with help of this small bash script: #!/bin/sh
set -e
rm -rf test_dist && mkdir test_dist
pip install -U pip
poetry install --only build
poetry export --with dev --extras sources -f requirements.txt --output test_dist/test_requirements.txt --without-hashes --with-credentials
echo "========= BUILD PEX ========="
poetry run pex -r test_dist/test_requirements.txt -o test_dist/test_sources_dependencies.pex
This is the content of our [tool.poetry]
name = "spark-sources"
version = "0"
description = "spark-sources for CXA-CK Cluster jobs"
readme = "README.md"
authors = ["<[Masked]@[Masked]>"]
[tool.poetry.dependencies]
python = "~3.9"
bda-base-dependencies = { path = "bda_base_dependencies/", develop = true, optional = true }
ck-spark-sources = { path = "sources/", develop = true, optional = true }
utils = { path = "utils/", develop = true, optional = true }
common-utils = { path = "utils/common_utils/", develop = true, optional = true }
datahub-utils = { path = "utils/datahub_utils/", develop = true, optional = true }
kafka-utils = { path = "utils/kafka_utils/", develop = true, optional = true }
logck = { path = "utils/logck/", develop = true, optional = true }
mssql-utils = { path = "utils/mssql_utils/", develop = true, optional = true }
pyspark = { version = "3.3.0", optional = true }
table-builder = { path = "table_builder/", develop = true, optional = true }
test-helpers = { path = "utils/test_helpers/", develop = true, optional = true }
zeppelin-utils = { path = "utils/zeppelin_utils/", develop = true, optional = true }
[tool.poetry.extras]
bda-base = ["bda-base-dependencies"]
pyspark = ["pyspark"]
table-builder = ["bda-base-dependencies", "common-utils", "datahub-utils", "pyspark", "table-builder", "zeppelin-utils"]
sources = [
"bda-base-dependencies",
"ck-spark-sources",
"pyspark",
"table-builder",
"utils",
]
[tool.poetry.group.build.dependencies]
pex = "^2.1.117"
[tool.poetry.group.dev.dependencies]
Flask = "^3.0.0"
freezegun = "^1.1.0"
pytest = "^8.0.0"
pytest_check = "^2.0.0"
pytest-cov = "^5.0.0"
pre-commit = "^3.3.3"
rich = "^13.4.0"
ipdb = "^0.13.9"
dm-livy-submit = { version = "^2.3.0", source = "ck-pypi" }
dm-knox-token = { version = "^2.0.1", source = "ck-pypi" }
pytest-flaskserver = "^1.1.3"
sphinx = "^7.1.2"
autodoc-pydantic = "^2.0.1"
sphinx-rtd-theme = "^2.0.0"
sphinx-autoapi = "^3.0.0"
myst-parser = "^3.0.0"
[tool.pytest.ini_options]
addopts = "-ra -v --showlocals -s"
markers = [
"local_test_only", # marks tests that should not run as system test
"system_test_only" # marks tests that should only run as system tests on cluster
]
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
[[tool.poetry.source]]
name = "dm-artifactory-pypi"
url = "https://[Masked].com/artifactory/api/pypi/pypi-remote/simple"
priority = "supplemental"
[[tool.poetry.source]]
name = "ck-pypi"
url = "https://[Masked]/api/v4/projects/13599/packages/pypi/simple/"
priority = "primary" |
|
@Github-dm-CDE where is the The more information you can provide the better. For example, PEX's are just zips. You might create both the good PEX and bad PEX, unzip them both in different directories and then run a recursive diff to see what's different. That would be very helpful information. |
|
Hi @jsirois , Now {
"bootstrap_hash": "abae6f731d4b0d818ab6d02e09294593143e0132",
"build_properties": {
"pex_version": "2.9.0"
},
"code_hash": "4155f382ffd2ab38bcf15a249b859595d933456e",
"deps_are_wheel_files": false,
"distributions": {
"certifi-2024.7.4-py3-none-any.whl": "f51955747a040ad60cc9b281d2b81748fda55a5bb5c4b6b10be65f0354609302",
"charset_normalizer-3.3.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "606334df13ed33c5ab6385635909cf91fb213ef7b11022f40e7a3161c7b2ee98",
"exceptiongroup-1.2.1-py3-none-any.whl": "f74bb5f1b9d4d177eb069806e20fd043b788e4275f0bd3405badabfc0103dac3",
"idna-3.7-py3-none-any.whl": "243156850dfb4ab00c4b9604d58d83e722e41eb0f445358feb23af5701231307",
"iniconfig-2.0.0-py3-none-any.whl": "988e40b3b55723f6802760cae3cd0319699d3e69ae7f34c7793d38fb44d579b7",
"packaging-24.1-py3-none-any.whl": "25baba288a0f0e2ebe1b0117cad380c5d4c0137aff4a2c7755f92cffd0eb38f8",
"pluggy-1.5.0-py3-none-any.whl": "6826f70172987927d47205f5bc5b1b18c380201149e5b63bc90e4d1645e7a8d9",
"pytest-8.2.2-py3-none-any.whl": "ce97a2c0ec29fd42f042f7e76e7f857fb83919d9054169b16988d628f47b43f4",
"requests-2.32.1-py3-none-any.whl": "7226b06e0a6f282f84ca65816c130920e5d01c2932e0382e9fe7afec1c993928",
"tomli-2.0.1-py3-none-any.whl": "8af30168c90e438c0488ddb0dcc8bfbdb7f54299570473943266c8f800de58dc",
"urllib3-2.2.2-py3-none-any.whl": "568f866565573953519a3419f2ab438633b3fb6e932725d0a1aecb46a35d9b12"
},
"emit_warnings": true,
"excluded": [],
"ignore_errors": false,
"includes_tools": false,
"inherit_path": "false",
"inject_args": [],
"inject_env": {},
"inject_python_args": [],
"interpreter_constraints": [],
"max_install_jobs": 1,
"overridden": [],
"pex_hash": "3f284033281410a36e3ac7a6efebbc97277d6311",
"pex_path": "",
"pex_paths": [],
"requirements": [
"pytest"
],
"strip_pex_env": true,
"venv": false,
"venv_bin_path": "false",
"venv_copies": false,
"venv_hermetic_scripts": true,
"venv_site_packages_copies": false,
"pex_root": "/home/alex/.pex"
}However there is an error importing the package, because it is not part of the Hope that helps to debug the issue. |
|
Thanks @xebab, that's a helpful repro case. When I get back to my keyboard on 7/14 I'll take a look and hopefully get a quick fix out. |
|
Ok, this is an embarrassing bug. Although part of the requirement parsing logic handles editable, there is a part that runs before it that short circuits the editable logic. I'll have a fix PR out this afternoon and hopefully a release shortly after. |
Previously, editable requirements in requirements files were not parsed properly by Pex. Although they did not trigger parse errors, PEXes created from editable requirements would fail to import those requirements at runtime despite the editable project distribution being embedded in the PEX file. Fixes pex-tool#2410
Previously, editable requirements in requirements files were not parsed properly by Pex. Although they did not trigger parse errors, PEXes created from editable requirements would fail to import those requirements at runtime despite the editable project distribution being embedded in the PEX file. Fixes #2410
|
Alright, a fix for this is now released in 2.10.1: Thanks @Github-dm-CDE and @xebab for the report and repro case respectively. |
Since python-poetry/poetry-plugin-export#258 was merged, the temporary requirements.txt format's changed. This makes local project dependencies unparseable. The build works, but the pex entrypoints do not exist / changed.
requirements.txt with [email protected]
requirements.txt with [email protected]
With the pex created based on the requierements.txt of v1.8.0, our pipeline failed:
The text was updated successfully, but these errors were encountered: