Changes are planned unnecessarily due to whitespace diffs when using file() function

[nick-doyle]

Terraform Version

0.12.20

Terraform Configuration Files

resource "azurerm_api_management_api_operation" "my-api-op" {
  operation_id        = "my-api-op"
  api_name            = "${azurerm_api_management_api.my-api.name}"
  api_management_name = "${azurerm_api_management_api.my-api.api_management_name}"
  resource_group_name = "${azurerm_api_management_api.my-api.resource_group_name}"
  display_name        = "Add Item"
  method              = "POST"
  url_template        = "/items"
  description         = "Add an item"

  request {
    representation {
      content_type = "application/json"
      sample       = file("${path.module}/add-item-request-sample.json")
    }
  }

  response {
    description = "Item added"
    status_code = 200

    representation {
      content_type = "application/json"
      sample       = file("${path.module}/add-item-response-sample.json")
    }
  }
}

Debug Output

Crash Output

Expected Behavior

Bearing in mind I am not actually making any Terraform config changes at this moment in time, when I run a 'terraform plan' I expect:

Plan: 0 to add, 0 to change, 0 to destroy.

Actual Behavior

'terraform plan' gives:

# module.api-app.azurerm_api_management_api_operation.my-api-op will be updated in-place
  ~ resource "azurerm_api_management_api_operation" "my-api-op" {
        api_management_name = "my-apim"
        api_name            = "my-api"
        description         = "Add an item"
        display_name        = "Add Item"
        id                  = "my-api-op"
        method              = "POST"
        operation_id        = "add-item"
        resource_group_name = "my-resource-group"
        url_template        = "/items"

      ~ request {
            description = ""

          ~ representation {
                content_type = "application/json"
              ~ sample       = jsonencode( # whitespace changes
                    {
                        <contents of file 'add-item-request-sample.json' appear here>
                    }
                )
            }
        }

      ~ response {
            description = "The item was added"
            status_code = 201

          ~ representation {
                content_type = "application/json"
              ~ sample       = jsonencode( # whitespace changes
                    {
                        <contents of file 'add-item-response-sample.json' appear here>
                    }
                )
            }
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Note the lines which start with 'sample'. Those lines show '# whitespace change' when there have been no changes made at all.

Steps to Reproduce

This problem, whereby Terraform is unnecessarily applying changes due to perceived whitespace change, occurs when two different team members are working on two different operating systems (Unix and Windows).

1. On a Unix/Mac machine, make some Terraform config change and terraform apply
2. On the same Unix/Mac machine, do terraform plan and observe that no changes are detected - correctly.
3. Switch to a Windows machine, check out the latest config used in steps 1&2 and run terraform plan. Observe that it wants to make a change due to whitespace differences, as per output shown above.

The same problem occurs if working the other way round (start with Windows, switch to Unix/Mac).

Additional Context

The problem appears to be down to the way the Unix/Windows versions of Terraform each handle whitespace slightly differently when the file() function runs and a 'jsonencode' statement is produced. It happens for any resource type.

This unnecessary output clutters everything when reviewing changes and results in many more 'apply' commands being run by team members and by CI.

References

[apparentlymart]

Hi @nick-doyle,

Terraform Core itself treats a file loaded with file as an opaque series of unicode characters and will, by default, compare it character-by-character to test equality, because it has no additional context on what the meaning of that data is or how it will be parsed by the provider.

A common cause of this problem is that the version control system is configured to rewrite these files to use Windows-style line endings (\r\n) when cloning on a Windows system, and vice-versa. From Terraform's perspective then, the file is different and it can't assume without other information that those newline changes are harmless. Reconfiguring your version control system to not rewrite the files as they move between platforms, so that they remain byte-for-byte identical, is likely the best workaround for the moment.

However, fortunately there is a mechanism for Terraform to get a better result here: The provider itself can include a normalization rule that allows it to signal to Terraform that even though the string is not character-for-character equal, it's still "equal enough" to not need updating. On the provider development side this is called DiffSuppressFunc, and the SDK already includes helpers for recognizing when two JSON strings are equivalent even if they have irrelevant whitespace differences.

For that reason, I'm going to ask our bot to move this issue over to the Azure provider repository in the hope that the provider could be updated to include such a rule for this sample argument, which should then cause these whitespace differences to be ignored.

[ghost]

This issue has been automatically migrated to hashicorp/terraform-provider-azurerm#5500 because it looks like an issue with that provider. If you believe this is not an issue with the provider, please reply to hashicorp/terraform-provider-azurerm#5500.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.