Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

📝 Destination bigquery: mark service account as required for cloud #12768

Merged
merged 14 commits into from
May 15, 2022
Merged

📝 Destination bigquery: mark service account as required for cloud #12768

merged 14 commits into from
May 15, 2022

Conversation

tuliren
Copy link
Contributor

@tuliren tuliren commented May 11, 2022 •
edited
Loading

What

TODO

  • Add screenshots in the documentation to show case service account, required roles, IAM, and HMAC key.
  • List common mistakes that can lead to permission issues.
  • Add a sample "service account key json" in the documentation.
  • Add extra connection check message if possible.
    • The check already has specific error message for GCS permission issues.

🚨 User Impact 🚨

  • Previously users may get the wrong impression that the "service account key json" field is optional on cloud.
  • Now it will be clear that this field is required on cloud.

@github-actions github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels May 11, 2022
@tuliren
Copy link
Contributor Author

tuliren commented May 15, 2022 •
edited
Loading

BigQuery connector documentation screenshot - add "Cloud Storage Admin" role to BigQuery service account.

Screen Shot 2022-05-14 at 22 32 13

@tuliren
Copy link
Contributor Author

tuliren commented May 15, 2022

BigQuery connector documentation screenshot - create a dedicated role for gcs access

Screen Shot 2022-05-14 at 22 39 27

@tuliren
Copy link
Contributor Author

tuliren commented May 15, 2022 •
edited
Loading

BigQuery connector documentation screenshot - create hmac key for the bigquery service account

Screen Shot 2022-05-14 at 22 48 04

@tuliren
Copy link
Contributor Author

tuliren commented May 15, 2022

BigQuery connector documentation screenshot - create a service account with the bigquery user and data editor roles

Screen Shot 2022-05-14 at 22 55 49

@tuliren
Copy link
Contributor Author

tuliren commented May 15, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2326393760
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2326393760
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_config/transform.py                         159     31    81%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_catalog/utils.py                             34      7    79%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 534    345    35%
-------------------------------------------------------------------------------------
TOTAL                                                              1320    550    58%

@tuliren
Copy link
Contributor Author

tuliren commented May 15, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/2326393949
✅ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/2326393949
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_config/transform.py                         159     31    81%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_catalog/utils.py                             34      7    79%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 534    345    35%
-------------------------------------------------------------------------------------
TOTAL                                                              1320    550    58%

@tuliren
Copy link
Contributor Author

tuliren commented May 15, 2022 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-bigquery run-tests=false

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2326610333
🚀 Successfully published connectors/destination-bigquery
🚀 Auto-bumped version for connectors/destination-bigquery
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2326610333

@tuliren
Copy link
Contributor Author

tuliren commented May 15, 2022 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-bigquery-denormalized run-tests=false

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/2326610855
🚀 Successfully published connectors/destination-bigquery-denormalized
🚀 Auto-bumped version for connectors/destination-bigquery-denormalized
✅ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/2326610855

@tuliren tuliren merged commit e00f487 into master May 15, 2022
@tuliren tuliren deleted the liren/bigquery-doc-update branch May 15, 2022 08:28
@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets May 15, 2022 08:29 Inactive
@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets May 15, 2022 08:29 Inactive
@octavia-squidington-iii octavia-squidington-iii mentioned this pull request May 17, 2022
Merged
suhomud pushed a commit that referenced this pull request May 23, 2022
@tuliren @octavia-squidington-iii @suhomud
📝 Destination bigquery: mark service account as required for cloud (#…
d9cdc99 
…12768)

* Update spec

* Update doc

* Bump version and update changelog

* Modify wording

* Add sample service account key json

* Add screenshots and common permission issues

* Refactor service account helper method

* Update log message

* Update version date in changelog

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@tuliren @octavia-squidington-iii