-
Notifications
You must be signed in to change notification settings - Fork 43
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement transfer for BigQuery - read/write #1829
Conversation
for more information, see https://pre-commit.ci
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## main #1829 +/- ##
==========================================
+ Coverage 85.72% 86.66% +0.93%
==========================================
Files 124 125 +1
Lines 6485 6612 +127
Branches 643 648 +5
==========================================
+ Hits 5559 5730 +171
+ Misses 791 741 -50
- Partials 135 141 +6
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 8 files with indirect coverage changes Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report in Codecov by Sentry. |
…/astro-sdk into bigquery_dataprovider
S3Hook of apache-airflow-providers-amazon==3.2.0 doesn't support None in transfer_config_args needs {}
universal_transfer_operator/example_dags/example_universal_transfer_operator.py
Show resolved
Hide resolved
universal_transfer_operator/example_dags/example_dag_fivetran.py
Outdated
Show resolved
Hide resolved
universal_transfer_operator/example_dags/example_universal_transfer_operator.py
Outdated
Show resolved
Hide resolved
...transfer_operator/src/universal_transfer_operator/data_providers/database/google/bigquery.py
Outdated
Show resolved
Hide resolved
…ta_providers/database/google/bigquery.py Co-authored-by: Phani Kumar <[email protected]>
Co-authored-by: Phani Kumar <[email protected]>
for more information, see https://pre-commit.ci
# Description ## What is the current behavior? <!-- Please describe the current behavior that you are modifying. --> As part of this [PR 1829](#1829), a bug was introduced. This is not handling the case when the File path is a file pattern or directory. <img width="1680" alt="Screenshot 2023-03-24 at 12 23 45 AM" src="https://user-images.githubusercontent.com/8670962/227315288-b11bc232-65a0-458f-a217-b74c9b882b07.png"> In case the destination dataset is a folder it doesn't pass the filename from the source at all here: https://github.com/astronomer/astro-sdk/blame/1b35bb9443d57ef7834259a3cd3c26695ea5fcf8/universal_transfer_operator/src/universal_transfer_operator/data_providers/filesystem/base.py#L131 instead of https://github.com/astronomer/astro-sdk/blame/bf189de90e4ad9f2db77c7ac46a7193bde9a9c92/universal_transfer_operator/src/universal_transfer_operator/data_providers/filesystem/base.py#L123 <!-- Issues are required for both bug fixes and features. Reference it using one of the following: closes: #ISSUE related: #ISSUE --> closes: #1870 ## What is the new behavior? <!-- Please describe the behavior or changes that are being added by this PR. --> - It should check if the provided file is a directory or file pattern and then add the actual file name from the source dataset from `FileStream` ``` destination_file = self.dataset.path # check if destination dataset is folder or file pattern if self.dataset.is_pattern(): destination_file = os.path.join(self.dataset.path, os.path.basename(source_ref.actual_filename)) ``` - Add the test <img width="1719" alt="Screenshot 2023-03-24 at 12 44 01 AM" src="https://user-images.githubusercontent.com/8670962/227320470-48bfc6b7-54d9-49eb-b6d5-17efd129f9d1.png"> ## Does this introduce a breaking change? No ### Checklist - [x] Created tests which fail without the change (if possible) - [x] Extended the README / documentation, if necessary --------- Co-authored-by: utkarsh sharma <[email protected]>
# Description ## What is the current behavior? <!-- Please describe the current behavior that you are modifying. --> As part of this [PR 1829](astronomer/astro-sdk#1829), a bug was introduced. This is not handling the case when the File path is a file pattern or directory. <img width="1680" alt="Screenshot 2023-03-24 at 12 23 45 AM" src="https://user-images.githubusercontent.com/8670962/227315288-b11bc232-65a0-458f-a217-b74c9b882b07.png"> In case the destination dataset is a folder it doesn't pass the filename from the source at all here: https://github.com/astronomer/astro-sdk/blame/1b35bb9443d57ef7834259a3cd3c26695ea5fcf8/universal_transfer_operator/src/universal_transfer_operator/data_providers/filesystem/base.py#L131 instead of https://github.com/astronomer/astro-sdk/blame/bf189de90e4ad9f2db77c7ac46a7193bde9a9c92/universal_transfer_operator/src/universal_transfer_operator/data_providers/filesystem/base.py#L123 <!-- Issues are required for both bug fixes and features. Reference it using one of the following: closes: #ISSUE related: #ISSUE --> closes: #1870 ## What is the new behavior? <!-- Please describe the behavior or changes that are being added by this PR. --> - It should check if the provided file is a directory or file pattern and then add the actual file name from the source dataset from `FileStream` ``` destination_file = self.dataset.path # check if destination dataset is folder or file pattern if self.dataset.is_pattern(): destination_file = os.path.join(self.dataset.path, os.path.basename(source_ref.actual_filename)) ``` - Add the test <img width="1719" alt="Screenshot 2023-03-24 at 12 44 01 AM" src="https://user-images.githubusercontent.com/8670962/227320470-48bfc6b7-54d9-49eb-b6d5-17efd129f9d1.png"> ## Does this introduce a breaking change? No ### Checklist - [x] Created tests which fail without the change (if possible) - [x] Extended the README / documentation, if necessary --------- Co-authored-by: utkarsh sharma <[email protected]>
Please describe the feature you'd like to see
DataProvider
for Bigquery - read/write methodsAcceptance Criteria
closes: #1732
closes: #1785
closes: #1730