Extra PyPI packages almost always causes dependency conflicts

[koshy1123]

Specifying extra pypi packages for a composer environment, and trying to replicate it in a local build almost always results in a dependency conflict.

A reproducible example is to start with a composer-2.5.3-airflow-2.6.3 and add pypi packages:

apache-airflow-providers-amazon==8.15.0
apache-airflow-providers-slack==8.5.1

But depending on when you created the environment, sub-packages will have drifted, leading to incompatible builds in a local setting. I see logs that indicate that when trying to spin up the local container, eg:

pydantic 2.4.2 has requirement typing-extensions>=4.6.1, but you have typing-extensions 4.4.0

You can get around this by specifying tje requirements.txt to be an exact copy of the current configuration generated by list-packages. That leads to more stable builds locally, and confidence that the local environment more closely represents the composer environment in the GCP project.

What does the dev team think about that? Happy to contribute if this sounds like a good idea.

[koshy1123]

Here(ish) is where I think the change can happen:

composer-local-dev/composer_local_dev/environment.py

Lines 539 to 546 in 89c54a5

	def pypi_packages_to_requirements(self):
	"""Create requirements file using environment PyPi packagest list."""
	reqs = sorted(
	f"{key}{value}" for key, value in self.pypi_packages.items()
	)
	reqs_lines = "\n".join(reqs)
	with open(self.env_dir_path / "requirements.txt", "w") as fp:
	fp.write(reqs_lines)

[ahidalgob]

Hi, koshy1123, thanks for raising this and offering to help.

I will take a better look into this soon but just thinking about your idea, this won't work for upgrading/downgrading pypi packages that are already installed, right?

[koshy1123]

I will take a better look into this soon but just thinking about your idea, this won't work for upgrading/downgrading pypi packages that are already installed, right?

Thanks for your response @ahidalgob - just to clarify, do you mean the use case where someone wants to actually test upgrading/downgrading their pypi packages in the local build before making that change in their composer environment? Then yeah, this won't work :/