Create simple storage writer #826
Conversation
mrchtr
requested review from
RobbeSneyders,
GeorgesLorre and
PhilippeMoussalli
There was a problem hiding this comment.
Thanks @mrchtr!
Could you add it to the integration test as well?
|
The component is missing a README which makes the docs build fail. Not sure why this is not caught by the pre-commit hook? |
The readme was generated, but I have forgotten to push it. |
Co-authored-by: Philippe Moussalli <[email protected]>
Co-authored-by: Philippe Moussalli <[email protected]>
Co-authored-by: Philippe Moussalli <[email protected]>
components/write_to_file/Dockerfile
Outdated
| @@ -0,0 +1,28 @@ | |||
| FROM --platform=linux/amd64 python:3.8-slim as base | |||
There was a problem hiding this comment.
We might want to start using newer python versions since 3.8 will reach end-of-life before you know it. (https://devguide.python.org/versions/)
| ### Produces | ||
|
|
||
|
|
||
| **This component does not produce data.** |
There was a problem hiding this comment.
Correct bit it might confuse users.
Maybe word it like: This component does not produce a dataframe for downstream processing.
components/write_to_file/README.md
Outdated
| | argument | type | description | default | | ||
| | -------- | ---- | ----------- | ------- | | ||
| | path | str | Path to store the dataset, whether it's a local path or a cloud storage bucket, must be specified. A separate filename will be generated for each partition. If you are using the local runner and export the data to a local directory, ensure that you mount the path to the directory using the `--extra-volumes` argument. | / | | ||
| | format | str | Format for storing the dataframe can be either `csv` or `parquet`. | csv | |
There was a problem hiding this comment.
does the CSV write headers ? Does it use comma as a delimiter ? Should we allow for specifying this ?
There was a problem hiding this comment.
CSV is written with headers and comma separated. I'll add to the description.
I would not make this configurable. I see this one more as an example component. For more custom export methods I would recommend implementing a python component instead, no?
|
Wondering: did you test it with a remote path ? |
I've tested it quickly and it is working :) |
Added `write_to_file` to both getting started guides. #826 already added the component to the example pipeline. fix #825 --------- Co-authored-by: Robbe Sneyders <[email protected]>
Write component that writes a dask dataframe to a file, either csv or parquet.
Fix #824