Skip to content

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gcs_to_bq: Support schema in different bucket #12520

Closed
maroshmka opened this issue Nov 20, 2020 · 1 comment
Closed

gcs_to_bq: Support schema in different bucket #12520

maroshmka opened this issue Nov 20, 2020 · 1 comment
Labels
good first issue kind:feature Feature Requests provider:google Google (including GCP) related issues

Comments

@maroshmka
Copy link
Contributor

Description

Schema path (schema_object) for table in GCSToBigQueryOperator is downloading schema from source bucket.

New feature should support support also pulling schemas from different bucket.

Use case / motivation

I guess there are lot of use-cases. To mention one concrete:

  • we want to have schemas at 1 place, for us its terraform, where we define all resources and ACLs
  • those schemas are exported to GCS by CI
  • we want to use them in loads to BQ tables

In case there is a schema change, we make 1 change (in terraform) and it works. Otherwise we would need to do terraform change, apply, then either update schema_fields in operator and release it or to allow adding fields on table load (which we don't necessarily want all the time).

Related Issues

Not directly, but at least - #8280 - polish system tests to operator while at it

@maroshmka
Copy link
Contributor Author

Implemented draft, if its ok to have this feature, I can finish the system tests :)

@eladkal eladkal added the provider:google Google (including GCP) related issues label Jan 9, 2021
@apache apache locked and limited conversation to collaborators Sep 7, 2022
@eladkal eladkal converted this issue into discussion #26220 Sep 7, 2022

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Labels
good first issue kind:feature Feature Requests provider:google Google (including GCP) related issues
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants