Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SqlToS3Operator - Add feature to partition SQL table #30460

Merged
merged 7 commits into from
Apr 18, 2023
Merged

SqlToS3Operator - Add feature to partition SQL table #30460

merged 7 commits into from
Apr 18, 2023

Conversation

utkarsharma2
Copy link
Contributor

Added feature to partition SQL table. Internally used pandas groupby() method to partition the generated dataframe. User can pass the parameters to groupby() by passing values in groupby_kwargs.

closes: #30382

@boring-cyborg boring-cyborg bot added area:providers provider:amazon-aws AWS/Amazon - related issues labels Apr 4, 2023
@utkarsharma2 utkarsharma2 changed the title SqlToS3Operator - Added feature to partition SQL table SqlToS3Operator - Add feature to partition SQL table Apr 4, 2023
phanikumv
phanikumv reviewed Apr 4, 2023
View reviewed changes
Copy link
Contributor

@phanikumv phanikumv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WDYT about including this additional feature in the docs ?

@utkarsharma2
Copy link
Contributor Author

@phanikumv Yup, missed it. Added them now.

@utkarsharma2
Copy link
Contributor Author

@eladkal @o-nikolas Please review

@utkarsharma2
Copy link
Contributor Author

@o-nikolas Can you please review?

@hussein-awala hussein-awala mentioned this pull request Apr 17, 2023
Closed
2 tasks
@potiuk potiuk merged commit d7cef58 into apache:main Apr 18, 2023
wookiist pushed a commit to wookiist/airflow that referenced this pull request Apr 19, 2023
@utkarsharma2
SqlToS3Operator - Add feature to partition SQL table (apache#30460)
8642ec8
This was referenced Apr 21, 2023
Closed
Closed
@eladkal eladkal mentioned this pull request Apr 26, 2023
Closed
19 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@phanikumv phanikumv phanikumv left review comments

@vincbeck vincbeck vincbeck approved these changes

@potiuk potiuk potiuk approved these changes

@eladkal eladkal Awaiting requested review from eladkal eladkal is a code owner

@o-nikolas o-nikolas Awaiting requested review from o-nikolas o-nikolas is a code owner

Assignees
No one assigned
Labels
area:providers provider:amazon-aws AWS/Amazon - related issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

SqlToS3Operator not able to write data with partition_cols provided.
4 participants
@utkarsharma2 @potiuk @phanikumv @vincbeck