Implement transfer for BigQuery - read/write #1829
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rajaths010494
changed the title
for more information, see https://pre-commit.ci
9 tasks
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## main #1829 +/- ##
==========================================
+ Coverage 85.72% 86.66% +0.93%
==========================================
Files 124 125 +1
Lines 6485 6612 +127
Branches 643 648 +5
==========================================
+ Hits 5559 5730 +171
+ Misses 791 741 -50
- Partials 135 141 +6
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 8 files with indirect coverage changes Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report in Codecov by Sentry. |
…/astro-sdk into bigquery_dataprovider
2 tasks
utkarsharma2
requested review from
pankajastro,
feluelle,
tatiana and
utkarsharma2
as code owners
S3Hook of apache-airflow-providers-amazon==3.2.0 doesn't support None in transfer_config_args needs {}
phanikumv
reviewed
Mar 13, 2023
universal_transfer_operator/example_dags/example_universal_transfer_operator.py
Show resolved
Hide resolved
phanikumv
reviewed
Mar 14, 2023
universal_transfer_operator/example_dags/example_dag_fivetran.py
Outdated
Show resolved
Hide resolved
universal_transfer_operator/example_dags/example_universal_transfer_operator.py
Outdated
Show resolved
Hide resolved
...transfer_operator/src/universal_transfer_operator/data_providers/database/google/bigquery.py
Outdated
Show resolved
Hide resolved
phanikumv
reviewed
Mar 14, 2023
…ta_providers/database/google/bigquery.py Co-authored-by: Phani Kumar <[email protected]>
Co-authored-by: Phani Kumar <[email protected]>
for more information, see https://pre-commit.ci
utkarsharma2
approved these changes
Mar 16, 2023
This was referenced Mar 23, 2023
sunank200
added a commit
that referenced
this pull request
Mar 24, 2023
# Description ## What is the current behavior? <!-- Please describe the current behavior that you are modifying. --> As part of this [PR 1829](#1829), a bug was introduced. This is not handling the case when the File path is a file pattern or directory. <img width="1680" alt="Screenshot 2023-03-24 at 12 23 45 AM" src="https://user-images.githubusercontent.com/8670962/227315288-b11bc232-65a0-458f-a217-b74c9b882b07.png"> In case the destination dataset is a folder it doesn't pass the filename from the source at all here: https://github.com/astronomer/astro-sdk/blame/1b35bb9443d57ef7834259a3cd3c26695ea5fcf8/universal_transfer_operator/src/universal_transfer_operator/data_providers/filesystem/base.py#L131 instead of https://github.com/astronomer/astro-sdk/blame/bf189de90e4ad9f2db77c7ac46a7193bde9a9c92/universal_transfer_operator/src/universal_transfer_operator/data_providers/filesystem/base.py#L123 <!-- Issues are required for both bug fixes and features. Reference it using one of the following: closes: #ISSUE related: #ISSUE --> closes: #1870 ## What is the new behavior? <!-- Please describe the behavior or changes that are being added by this PR. --> - It should check if the provided file is a directory or file pattern and then add the actual file name from the source dataset from `FileStream` ``` destination_file = self.dataset.path # check if destination dataset is folder or file pattern if self.dataset.is_pattern(): destination_file = os.path.join(self.dataset.path, os.path.basename(source_ref.actual_filename)) ``` - Add the test <img width="1719" alt="Screenshot 2023-03-24 at 12 44 01 AM" src="https://user-images.githubusercontent.com/8670962/227320470-48bfc6b7-54d9-49eb-b6d5-17efd129f9d1.png"> ## Does this introduce a breaking change? No ### Checklist - [x] Created tests which fail without the change (if possible) - [x] Extended the README / documentation, if necessary --------- Co-authored-by: utkarsh sharma <[email protected]>
sunank200
added a commit
to astronomer/apache-airflow-providers-transfers
that referenced
this pull request
Mar 24, 2023
# Description ## What is the current behavior? <!-- Please describe the current behavior that you are modifying. --> As part of this [PR 1829](astronomer/astro-sdk#1829), a bug was introduced. This is not handling the case when the File path is a file pattern or directory. <img width="1680" alt="Screenshot 2023-03-24 at 12 23 45 AM" src="https://user-images.githubusercontent.com/8670962/227315288-b11bc232-65a0-458f-a217-b74c9b882b07.png"> In case the destination dataset is a folder it doesn't pass the filename from the source at all here: https://github.com/astronomer/astro-sdk/blame/1b35bb9443d57ef7834259a3cd3c26695ea5fcf8/universal_transfer_operator/src/universal_transfer_operator/data_providers/filesystem/base.py#L131 instead of https://github.com/astronomer/astro-sdk/blame/bf189de90e4ad9f2db77c7ac46a7193bde9a9c92/universal_transfer_operator/src/universal_transfer_operator/data_providers/filesystem/base.py#L123 <!-- Issues are required for both bug fixes and features. Reference it using one of the following: closes: #ISSUE related: #ISSUE --> closes: #1870 ## What is the new behavior? <!-- Please describe the behavior or changes that are being added by this PR. --> - It should check if the provided file is a directory or file pattern and then add the actual file name from the source dataset from `FileStream` ``` destination_file = self.dataset.path # check if destination dataset is folder or file pattern if self.dataset.is_pattern(): destination_file = os.path.join(self.dataset.path, os.path.basename(source_ref.actual_filename)) ``` - Add the test <img width="1719" alt="Screenshot 2023-03-24 at 12 44 01 AM" src="https://user-images.githubusercontent.com/8670962/227320470-48bfc6b7-54d9-49eb-b6d5-17efd129f9d1.png"> ## Does this introduce a breaking change? No ### Checklist - [x] Created tests which fail without the change (if possible) - [x] Extended the README / documentation, if necessary --------- Co-authored-by: utkarsh sharma <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Please describe the feature you'd like to see
DataProviderfor Bigquery - read/write methodsAcceptance Criteria
closes: #1732
closes: #1785
closes: #1730