Support BigQuery custom schema's for external data using CSV / NDJSON

[ffung]

external_data_configuration.schema for CSV and NDJSON formats

Fixes hashicorp/terraform-provider-google#6693

Release Note Template for Downstream PRs (will be copied)

bigquery: added support for BigQuery custom schemas for external data using CSV / NDJSON 

[modular-magician]

Hello! I am a robot who works on Magic Modules PRs.

I have detected that you are a community contributor, so your PR will be assigned to someone with a commit-bit on this repo for initial review.

Thanks for your contribution! A human will be with you soon.

@danawillow, please review this PR or find an appropriate assignee.

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 132 insertions(+), 10 deletions(-))
Terraform Beta: Diff ( 3 files changed, 132 insertions(+), 10 deletions(-))

[danawillow]

I don't totally follow why not reading back the external schema requires us to do ForceNew. Couldn't we still allow updating that field even while not detecting drift on it?

[ffung]

The ForceNew is not required in terms of reading back the data, I just consider this better UX wrt the expectations of the user, e.g. if you change the schema, the most probable course I think is to recreate the table with that schema. Afaik you cannot change the schema of an existing table in place.

The external_data_configuration.schema is only used as an input parameter for creating the table, when we read back the table, this field is always empty. There is a computed schema returned on top level, which reflects the effective schema of the created table, however this value is calculated by combining the schema provided here and any other field/type mappings it can infer by autodetection and/or inferred from the source_uri_prefix. I wanted to avoid having to determine if external_data_configuration.schema is accurately reflected in the computed schema and reimplement BQ's logic in doing so, so I just assume after creation this is successfully reflected, hence I just ignore this field by making sure there are no changes with what is configured.

Perhaps there's a smarter way to do this?

[danawillow]

That makes sense. If it's not clear that it can be updated and users wouldn't necessarily expect it to be, we can leave it as ForceNew. Worst thing that happens is someone files an issue to ask for update support.

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 138 insertions(+), 18 deletions(-))
Terraform Beta: Diff ( 3 files changed, 138 insertions(+), 18 deletions(-))

[danawillow]

Looks like some information got repeated here.

Can you also add a note that we don't detect drift on this field?

[ffung]

Yes good suggestion, I expanded a bit on the description

[danawillow]

That makes sense. If it's not clear that it can be updated and users wouldn't necessarily expect it to be, we can leave it as ForceNew. Worst thing that happens is someone files an issue to ask for update support.

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 136 insertions(+), 18 deletions(-))
Terraform Beta: Diff ( 3 files changed, 136 insertions(+), 18 deletions(-))

[danawillow]

Looks good, thanks for the contribution @ffung!