Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement transfer for GCS/S3 to BigQuery #1794

Closed
wants to merge 22 commits into from

Conversation

rajaths010494
Copy link
Contributor

@rajaths010494 rajaths010494 commented Feb 23, 2023

Please describe the feature you'd like to see

  • Add DataProvider for Bigquery - read/write methods
  • Add non-native transfer implementation for GCS to BigQuery
  • Add non-native transfer implementation for S3 to BigQuery
  • Add non-native transfer example DAG for BigQuery to Sqlite
  • Add non-native transfer example DAG for BigQuery to Snowflake
  • Add example DAG
  • Add tests with 90% coverage

Acceptance Criteria

  • All checks and tests in the CI should pass
  • Unit tests (90% code coverage or more, once available)
  • Integration tests (if the feature relates to a new database or external service)
  • Example DAG
  • Docstrings in reStructuredText for each of methods, classes, functions and module-level attributes (including Example DAG on how it should be used)
  • Exception handling in case of errors
  • Logging (are we exposing useful information to the user? e.g. source and destination)
  • Improve the documentation (README, Sphinx, and any other relevant)
  • How to use Guide for the feature (example)

@rajaths010494 rajaths010494 changed the title Implement transfer for GCS/s3 to BigQuery Implement transfer for GCS/S3 to BigQuery Feb 23, 2023
@rajaths010494 rajaths010494 changed the base branch from main to SnowflakeDatabaseDataProvider February 23, 2023 10:20
Copy link
Collaborator

@dimberman dimberman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great start! few nit comments but looking good overall.

@rajaths010494
Copy link
Contributor Author

Added read operations for bigquery as well.

@utkarsharma2 utkarsharma2 force-pushed the SnowflakeDatabaseDataProvider branch 2 times, most recently from 14b9fb4 to 275608e Compare March 2, 2023 15:58
Base automatically changed from SnowflakeDatabaseDataProvider to main March 6, 2023 13:09
@rajaths010494
Copy link
Contributor Author

closing this as this PR #1829 will cover along with tests

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants