Bug in Resource "google_bigquery_table": hive_partition_options does not properly create the hive fields

[nick-hf]

Terraform Version

Terraform v0.12.28

...

Terraform Configuration Files

resource "google_bigquery_table" "xxx" {
project = xxx
dataset_id = xxx
table_id = xxx

external_data_configuration {
autodetect = false
source_format = "CSV"
max_bad_records = 0
source_uris = [
"xxx"
]
csv_options {
quote = """
field_delimiter = ","
skip_leading_rows = 1
allow_quoted_newlines = true
}
hive_partitioning_options {
mode = "STRINGS"
source_uri_prefix = xxx
}
}

schema = ...
}

Expected Behavior

Table xxx should be created, with hive partitioned fields that follow the source_uri.

Actual Behavior

Table xxx is created, but without the hive partitioned fields generated.

Steps to Reproduce

Create an external table with this resource with a hive-formatted cloud storage backing.

Additional Context

This may be an API problem, as we have seen the same behavior with the bq cmd line interface.

[edwardmedia]

@nick-hf can you share the debug log? Thanks

[nick-hf]

There's no debug log to be shared. Terraform thinks it created the table properly, there's no error output at all. In fact, the created table in the BQ UI thinks it has hive partitioning set up, and reports the proper prefix URI and everything. But, the partition fields don't exist. Using the exact same configuration via the BQ UI works fine.

As I said, it also happens with the BQ command line tool, so there may be something outside just the Terraform module having problems. I don't appear to be the only person experiencing this: https://stackoverflow.com/questions/60838904/creating-external-table-from-gcs-with-hive-partition-information-in-bigquery-usi

[edwardmedia]

@nick-hf I understand. With the debug login, we can verify if the field is properly set as wanted from Terraform side, and review the responses from API's.

[nick-hf]

Plan:

# module.tech.module.clients.module.xxx.module.data.google_bigquery_table.xxx will be created
  + resource "google_bigquery_table" "xxx" {
      + creation_time       = (known after apply)
      + dataset_id          = "xxx"
      + etag                = (known after apply)
      + expiration_time     = (known after apply)
      + id                  = (known after apply)
      + last_modified_time  = (known after apply)
      + location            = (known after apply)
      + num_bytes           = (known after apply)
      + num_long_term_bytes = (known after apply)
      + num_rows            = (known after apply)
      + project             = "xxx"
      + schema              = jsonencode(
            [
              ...
            ]
        )
      + self_link           = (known after apply)
      + table_id            = "xxx"
      + type                = (known after apply)

      + external_data_configuration {
          + autodetect      = false
          + compression     = "NONE"
          + max_bad_records = 0
          + schema          = (known after apply)
          + source_format   = "CSV"
          + source_uris     = [
              + "gs://xxx/*",
            ]

          + csv_options {
              + allow_jagged_rows     = false
              + allow_quoted_newlines = true
              + encoding              = "UTF-8"
              + field_delimiter       = ","
              + quote                 = "\""
              + skip_leading_rows     = 1
            }

          + hive_partitioning_options {
              + mode              = "STRINGS"
              + source_uri_prefix = "gs://xxx/"
            }
        }
    }

Relevant bits from the apply:

2020/07/23 19:25:29 [WARN] Provider "registry.terraform.io/-/google" produced an invalid plan for module.tech.module.clients.module.xxx.module.data.google_bigquery_table.xxx, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .external_data_configuration[0].compression: planned value cty.StringVal("NONE") does not match config value cty.NullVal(cty.String)
      - .external_data_configuration[0].csv_options[0].allow_jagged_rows: planned value cty.False does not match config value cty.NullVal(cty.Bool)
      - .external_data_configuration[0].csv_options[0].encoding: planned value cty.StringVal("UTF-8") does not match config value cty.NullVal(cty.String)
module.tech.module.clients.module.xxx.module.data.google_bigquery_table.xxx: Creating...

2020/07/23 19:25:29 [DEBUG] module.tech.module.clients.module.xxx.module.data.google_bigquery_table.xxx: applying the planned Create change

2020/07/23 19:25:29 [WARN] Provider "registry.terraform.io/-/google" produced an unexpected new value for module.tech.module.clients.module.xxx.module.data.google_bigquery_table.xxx, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .description: was null, but now cty.StringVal("")
      - .friendly_name: was null, but now cty.StringVal("")
      - .external_data_configuration[0].ignore_unknown_values: was null, but now cty.False
module.tech.module.clients.module.xxx.module.data.google_bigquery_table.xxx: Creation complete after 1s [id=projects/xxx/datasets/xxx/tables/xxx]

[nick-hf]

Looks like this was fixed here #6693

Thanks for your assistance

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!