[conda] Support conda-style requirements.txt files?

[jherland]

(found while exploring potential Conda support for FawltyDeps, see e.g. #447 for more context)

I'm following the documentation at https://www.activestate.com/resources/quick-reads/how-to-manage-python-dependencies-with-conda/ to see what file format(s) conda use to encode dependency declarations.

Specifically, the following commands are mentioned here:

conda create --name my_conda_project python=3.8
conda activate my_conda_project
conda install requests
conda list -e > requirements.txt

This yields the following requirements.txt on my machine:

# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: linux-64
_libgcc_mutex=0.1=main
_openmp_mutex=5.1=1_gnu
brotli-python=1.0.9=py38h6a678d5_8
ca-certificates=2024.7.2=h06a4308_0
certifi=2024.7.4=py38h06a4308_0
charset-normalizer=3.3.2=pyhd3eb1b0_0
idna=3.7=py38h06a4308_0
ld_impl_linux-64=2.38=h1181459_1
libffi=3.4.4=h6a678d5_1
libgcc-ng=11.2.0=h1234567_1
libgomp=11.2.0=h1234567_1
libstdcxx-ng=11.2.0=h1234567_1
ncurses=6.4=h6a678d5_0
openssl=3.0.14=h5eee18b_0
pip=24.2=py38h06a4308_0
pysocks=1.7.1=py38h06a4308_0
python=3.8.19=h955ad1f_0
readline=8.2=h5eee18b_0
requests=2.32.3=py38h06a4308_0
setuptools=72.1.0=py38h06a4308_0
sqlite=3.45.3=h5eee18b_0
tk=8.6.14=h39e8969_0
urllib3=2.2.2=py38h06a4308_0
wheel=0.43.0=py38h06a4308_0
xz=5.4.6=h5eee18b_1
zlib=1.2.13=h5eee18b_1

This file is NOT compatible with pip (see e.g. https://stackoverflow.com/questions/50777849/from-conda-create-requirements-txt-for-pip3) and it does not follow https://pip.pypa.io/en/stable/reference/requirements-file-format/.

Instead, the Conda documentation states that this requirements.txt can be installed with this conda command:

conda install -n my_conda_project --file requirements.txt

Currently, attempting to parse the above requirements.txt file with FawltyDeps yields nothing at all.

We can add som debug log messages to our requirements.txt parsing code to print out invalid lines from the parse:

diff --git a/fawltydeps/extract_declared_dependencies.py b/fawltydeps/extract_declared_dependencies.py
index 18c7576..3aa382b 100644
--- a/fawltydeps/extract_declared_dependencies.py
+++ b/fawltydeps/extract_declared_dependencies.py
@@ -57,10 +57,15 @@ def parse_requirements_txt(path: Path) -> Iterator[DeclaredDependency]:
     https://pip.pypa.io/en/stable/reference/requirements-file-format/.
     """
     source = Location(path)
-    for dep in RequirementsFile.from_file(path).requirements:
+    parsed = RequirementsFile.from_file(path)
+    for dep in parsed.requirements:
         if dep.name:
             yield DeclaredDependency(dep.name, source)
 
+    if parsed.invalid_lines and logger.isEnabledFor(logging.DEBUG):
+        error_messages = "\n".join(line.dumps() for line in parsed.invalid_lines)
+        logger.debug(f"Invalid lines found in {source}:\n{error_messages}")
+
 
 def parse_setup_py(path: Path) -> Iterator[DeclaredDependency]:  # noqa: C901
     """Extract dependencies (package names) from setup.py.

And with this, we get the following debug output from the parser for each line in the above requirements.txt:

DEBUG:fawltydeps.extract_declared_dependencies:Invalid lines found in requirements.txt:
# Invalid requirement: = is not a valid operator. Did you mean == ?: Expected package name at the start of dependency specifier
    _libgcc_mutex=0.1=main
    ^
_libgcc_mutex=0.1=main
# Invalid requirement: = is not a valid operator. Did you mean == ?: Expected package name at the start of dependency specifier
    _openmp_mutex=5.1=1_gnu
    ^
_openmp_mutex=5.1=1_gnu
# Invalid requirement: = is not a valid operator. Did you mean == ?: Expected end or semicolon (after name and no valid version specifier)
    brotli-python=1.0.9=py38h6a678d5_8
                 ^
brotli-python=1.0.9=py38h6a678d5_8
[and so on...]

I don't know how prevalent this requirements.txt format is compared to e.g. the environment.yml format that Conda is also known to use. I suspect that we will need to somehow add support for this different requirements.txt format if we want to support Conda fully.

[jherland]

Further discussion on #447 and #452 indicate that this weird requirements.txt format is mostly used as a "lockfile" format (if used at all) in the Conda community. Apparently nobody is using this format to manually curate their direct dependency declarations. As such, it is not interesting for FawltyDeps to parse these files.

Instead, Conda users are more likely to manually write an environment.yml file and then use tools like conda-lock to produce corresponding lockfiles.

Closing this for now, but feel free to reopen if the these conclusions prove untrue.