azurerm_data_factory_delimited_text type properties don't default correctly

[easkay-castlight]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.12.31
+ provider.azurerm v2.77.0

Affected Resource(s)

• azurerm_data_factory_dataset_delimited_text

Terraform Configuration Files

# This config hasn't been applied
provider azurerm {
  version = "2.77.0"
  features {}
}

resource azurerm_resource_group example {
  name     = "example"
  location = "uksouth"
}

resource azurerm_storage_account example {
  name                      = "example"
  resource_group_name       = azurerm_resource_group.example.name
  location                  = azurerm_resource_group.example.location
  account_kind              = "StorageV2"
  account_tier              = "Standard"
  account_replication_type  = "GRS"
  enable_https_traffic_only = true
}

resource azurerm_storage_container example {
  name                  = "example"
  storage_account_name  = azurerm_storage_account.example.name
  container_access_type = "private"
}

resource azurerm_data_factory example {
  name                = "example"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
}

resource azurerm_data_factory_linked_service_azure_blob_storage example {
  name                     = "example"
  resource_group_name      = azurerm_resource_group.example.name
  data_factory_name        = azurerm_data_factory.example.name
  integration_runtime_name = "AutoResolveIntegrationRuntime"
  connection_string        = azurerm_storage_account.example.primary_connection_string
}

resource azurerm_data_factory_dataset_delimited_text example {
  name                = "example"
  resource_group_name = azurerm_resource_group.example.name
  data_factory_name   = azurerm_data_factory.example.name
  linked_service_name = azurerm_data_factory_linked_service_azure_blob_storage.example.name
  parameters          = {
    "subfolder" = "" # Map value sets parameter default value
  }

  first_row_as_header = true

  schema_column {
    name = "example"
    type = "String"
  }

  azure_blob_storage_location {
    container            = azurerm_storage_container.example.name
    path                 = "@dataset().subfolder"
    dynamic_path_enabled = true
  }
}

Debug Output

N/A

Panic Output

N/A

Expected Behaviour

The defaults for the dataset should have been set according to the docs:

• column_delimiter - (Optional) The column delimiter. Defaults to ,.
• row_delimiter - (Optional) The row delimiter. Defaults to any of the following values on read: \r\n, \r, \n, and \n or \r\n on write by mapping data flow and Copy activity respectively.
• encoding - (Optional) The encoding format for the file.
• quote_character - (Optional) The quote character. Defaults to ".
• escape_character - (Optional) The escape character. Defaults to \.

Which looks like this in Data Factory:

With the following JSON definition:

{
... // ellipsised
    "properties": {
        ... // ellipsised
        "type": "DelimitedText",
        "typeProperties": {
            "location": {...}, // ellipsised
            "columnDelimiter": ",",
            "escapeChar": "\\",
            "firstRowAsHeader": true,
            "quoteChar": "\""
        },
        "schema": [...] // ellipsised
    },
    "type": "Microsoft.DataFactory/factories/datasets"
}

Actual Behaviour

The pertinent keys in the JSON object (columnDelimiter, escapeChar, quoteChar, rowDelimiter) are set as empty strings instead of being left absent, or set to sensible defaults. This means that there are no defaults.

Which looks like this in Data Factory:

With the following JSON definition:

{
... // ellipsised
    "properties": {
        ... // ellipsised
        "type": "DelimitedText",
        "typeProperties": {
            "location": {...}, // ellipsised
            "columnDelimiter": "",
            "rowDelimiter": "",
            "escapeChar": "",
            "firstRowAsHeader": true,
            "quoteChar": ""
        },
        "schema": [],
        "structure": [...] // ellipsised
    },
    "type": "Microsoft.DataFactory/factories/datasets"
}

Steps to Reproduce

1. terraform apply

Important Factoids

N/A

References

N/A

[github-actions]

This functionality has been released in v3.5.0 of the Terraform Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.