Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow Customizing File Write Behavior of export_to_file Operator #2195

Open
9 tasks
jordanyakerirt opened this issue Nov 22, 2024 · 0 comments · May be fixed by #2196
Open
9 tasks

Allow Customizing File Write Behavior of export_to_file Operator #2195

jordanyakerirt opened this issue Nov 22, 2024 · 0 comments · May be fixed by #2196
Labels
feature New feature or request

Comments

@jordanyakerirt
Copy link

Please describe the feature you'd like to see
The current implementation of the export_to_file operator doesn't provide any mechanism for overriding or customizing the behavior of the file output. Instead the operator simply calls one of a number of Pandas.DataFrame.to_* functions with default functions that SDK users are locked in to.

Describe the solution you'd like
Allow the passing in of a dictionary of parameters that get conveyed to the underlying Pandas.DataFrame functionality.

Are there any alternatives to this feature?
No. The current write implementation is hard-coded to a specific set of parameters in the FileType.create_from_dataframe functionality.

Acceptance Criteria

  • All checks and tests in the CI should pass
  • Unit tests (90% code coverage or more, once available)
  • Integration tests (if the feature relates to a new database or external service)
  • Example DAG
  • Docstrings in reStructuredText for each of methods, classes, functions and module-level attributes (including Example DAG on how it should be used)
  • Exception handling in case of errors
  • Logging (are we exposing useful information to the user? e.g. source and destination)
  • Improve the documentation (README, Sphinx, and any other relevant)
  • How to use Guide for the feature (example)
@jordanyakerirt jordanyakerirt added the feature New feature or request label Nov 22, 2024
@jordanyakerirt jordanyakerirt linked a pull request Nov 22, 2024 that will close this issue
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant