Managing dependencies using tools like pipenv or poetry

[ysk24ok]

Is your feature request related to a problem? Please describe.
Currently dependencies are managed in sdk/python/setup.py and installation in CI is done by pip install -e "sdk/python" via Makefile targets.
But users who use pip-compile and pipenv can suffer installing feast package (e.g see #1912) because the dependency check is loose.

Describe the solution you'd like
Using pipenv. Installing dependencies in CI will be done by pipenv install -s sdk/python, which generates Pipfile.lock. After the lock file is generated, re-installing dependencies can be done by pipenv sync.

• pros
  • Dependencies are checked and version conflicts can be found when we execute pipenv install.
  • Using pipenv can be a good small start compared to poetry.
• cons
  • Locking might take long (but only when dependencies are update).
  • Commands in Makefile or GitHub Actions yaml files must be updated, e.g. pytest -> pipenv run pytest

Using pip-compile might work. Just I'm not familiar with pip-compile.

Describe alternatives you've considered
Using poetry. Installing dependencies in CI will be done by poetry install using pyproject.toml, which generates poetry.lock. After the lock file is generated, re-installing dependencies can be done by poetry install, this time using the lock file.

• pros
  • Dependencies are checked and version conflicts can be found when we execute poetry install.
  • poetry can not only check dependencies but also build and publish wheel packages.
  • poetry is a tool which aims to eliminate pipenv's pain. See https://github.com/python-poetry/poetry#what-about-pipenv.
• cons
  • poetry manages all settings in pyproject.toml. Using poetry can be a big change compared to pipenv.
  • Commands in Makefile or GitHub Actions yaml must be updated, e.g. pytest -> poetry run pytest

Additional context

#1912

Which do maintainers think is better? After the decision is made, I can give it a shot and create a pull request.

[adchia]

Hey! Sorry for the late response here.

Might be worth discussing this in a community call, but this does seem like it could solve some of the recent issues around install issues.

Seeing a draft PR for this would be really interesting to see!

[ysk24ok]

Seeing a draft PR for this would be really interesting to see!

Thank you for your response. I'd be glad to give it a shot.

[ysk24ok]

@adchia I've created #2093. I'd appreciate if you could take a look and give me some feedback.