Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move spark to extra requirements file #45

Merged
merged 2 commits into from
Jun 28, 2022
Merged

Conversation

winfried-ripken
Copy link
Contributor

@winfried-ripken winfried-ripken commented Jun 28, 2022

Description

Move spark requirement such that it can be installed separately.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring including code style reformatting
  • Other (please describe):

Checklist:

  • I have read the contributing guideline doc (external contributors only)
  • Lint and unit tests pass locally with my changes
  • I have kept the PR small so that it can be easily reviewed
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • All dependency changes have been reflected in the pip requirement files.

@@ -0,0 +1 @@
pyspark==3.2.0

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how about naming the flavor preprocessing ? so that if the preprocessing pipeline needs more dependencies in the future, they go here. Now it feels that as soon as we add more dependencies to this flavor, the name doesnt make much sense anymore.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right! Turns out it was already there.. my bad

@winfried-ripken winfried-ripken merged commit 5cbe3ac into main Jun 28, 2022
@winfried-ripken winfried-ripken deleted the winnie-move-spark branch June 28, 2022 12:19
@github-actions github-actions bot locked and limited conversation to collaborators Jun 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants