-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add csv driver option to specify csv read args #93
Conversation
- Allows for more flexible usage of CsvDriver in catalog
CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅ |
I have read the CLA Document and I hereby sign the CLA |
requirements.txt
Outdated
@@ -462,62 +462,6 @@ google-resumable-media==2.3.3 \ | |||
googleapis-common-protos==1.56.4 \ | |||
--hash=sha256:8eb2cbc91b69feaf23e32452a7ae60e791e09967d81d4fcc7fc388182d1bd394 \ | |||
--hash=sha256:c25873c47279387cfdcbdafa36149887901d36202cb645a0e4f29686bf6e4417 | |||
greenlet==1.1.2 \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
doesn't a change here mean that requirements.in
also got changed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Dang, I didn't do any requirements changes and did not commit that on purpose. Will revert.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done, thanks for the catch!
I don't understand, why the test fails due to changes introduced by |
should be solved now. weirdly, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks Max, LGTM!
Description
Adds a
read_csv_kwargs
argument toCsvDriver
initialization which is used in allread_csv
calls in the class.Does not break backward compatibility, as
get_df
andget_iter
still allow to specify kwargs forread_csv
which will take precedence over the ones given at initialization.This makes the creation of new catalog entries based on the
CsvDriver
much easier as dataset-specific read options (such as seperator, dtypes, etc.) can be specified in thedriver_kwargs
.Type of change
Checklist: