Support for managed virtual networking in azure data factory

[dgcaron]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

When you create an Azure data factory you can specify that the Integration Runtime needs to be placed on a dedicated VNET (managed virtual network)

it would be helpfull if we can express this setting while creating the data factory. As of right now, we need to create a second IR that has this setting.

New or Affected Resource(s)

azurerm_data_factory

Potential Terraform Configuration

The property would be named similarly to Azure Synapse Workspace (azurerm_synapse_workspace)

resource "azurerm_data_factory" "example" {
  name                = "example"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name

  managed_virtual_network_enabled  = true
}

References

• https://azure.microsoft.com/nl-nl/blog/azure-data-factory-managed-virtual-network/
• https://docs.microsoft.com/en-us/azure/data-factory/managed-virtual-network-private-endpoint

[ghost]

This also needs to be supported in the resource "azurerm_data_factory_integration_runtime_azure". If you do not specify this when creating the data factory, you can create an Azure Managed Virtual Network Integration runtime.

[drjwelch]

I am facing this issue right now yet it appears to be the recommended architecture for almost 1 year (https://medium.com/microsoftazure/azure-sql-database-network-settings-private-link-vnet-service-endpoint-and-azure-data-factory-b0f72b5d2af2)

When you say "As of right now, we need to create a second IR that has this setting." - do you mean this must be done in the portal? I can see no way to enable this setting via terraform.

[ghost]

I am also facing this issue now and yes, I create it in the portal and import it into Terraform. Terraform doesn't detect the Managed Virtual Network configuration so the plan is clean. The portal will not allow you to change from the default Public to Managed Virtual Network so you can not create the runtime with Terraform and edit it in the portal. It must be created in the portal.

I also see no way to create the necessary Managed Private Endpoints other than the portal.

[cbellee]

Hi All, I’ll look into implementing the managed virtual network & managed private endpoint capabilities. On initial inspection the required resource types are available in Azure Go SDK.

[willfell]

This would make life a lot easier at the moment.

[dkutetsky]

@mark-b-greene , @drjwelch for time being, you can use azurerm_resource_group_template_deployment resource and ARM templates (it could be exported from azure data factory UI).

Some sort of this:

ADF managed vnet:

resource "azurerm_resource_group_template_deployment" "azure_managed_virtual_network" {
  name = "adf-managed-vnet-runtime"
  resource_group_name = "resource-group-name"
  deployment_mode = "Incremental"

  template_content = file("${path.module}/templates/azure_vnet_managed_runtime.json")

  parameters_content = jsonencode({
    "factoryName" = {
      value = "factory-name"
    },
    "runtimeName" = {
      value = "adf-managed-vnet-runtime"
    }
    "runtimeLocation" = {
      value = "runtime location"
    }
  })
}

azure_vnet_managed_runtime.json:

azure_vnet_managed_runtime.json:
{
  "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "factoryName": {
      "type": "String",
      "metadata": "ADF name"
    },
    "runtimeName": {
      "type": "String",
      "metadata": "ADF runtime name"
    },
    "runtimeLocation": {
      "type": "String",
      "metadata": "ADF runtime location"
    }
  },
  "variables": {
    "factoryId": "[concat('Microsoft.DataFactory/factories/', parameters('factoryName'))]"
  },
  "resources": [
    {
      "name": "[concat(parameters('factoryName'), '/', parameters('runtimeName'))]",
      "type": "Microsoft.DataFactory/factories/integrationRuntimes",
      "apiVersion": "2018-06-01",
      "properties": {
        "type": "Managed",
        "typeProperties": {
          "computeProperties": {
            "location": "[parameters('runtimeLocation')]",
            "dataFlowProperties": {
              "computeType": "General",
              "coreCount": 8,
              "timeToLive": 10,
              "cleanup": true
            }
          }
        },
        "managedVirtualNetwork": {
          "type": "ManagedVirtualNetworkReference",
          "referenceName": "default"
        }
      },
      "dependsOn": [
        "[concat(variables('factoryId'), '/managedVirtualNetworks/default')]"
      ]
    },
    {
      "name": "[concat(parameters('factoryName'), '/default')]",
      "type": "Microsoft.DataFactory/factories/managedVirtualNetworks",
      "apiVersion": "2018-06-01",
      "properties": {
        "preventDataExfiltration": false
      },
      "dependsOn": []
    }
  ]
}

Managed endpoint:

resource "azurerm_resource_group_template_deployment" "private_endpoint" {
  name = "private-endpoint"
  resource_group_name = "resource_group_name"
  deployment_mode = "Incremental"

  template_content = file("${path.module}/templates/adf_managed_endpoint.json")

  parameters_content = jsonencode({
    "name" = {
      value = "<private-endpoint>"
    },
    "factoryName" = {
      value = "<factory-name>"
    }
    "linkedResourceId" = {
      value = "<linked-resource-id>"
    }
    "linkedResourceGroupId" = {
      value = "blob"
    }
    "linkedResourceFqdn" = {
      value = ["<linked-resource-fqdn>"]
    }
  })

  depends_on = [azurerm_resource_group_template_deployment.azure_managed_virtual_network]
}

adf_managed_endpoint.json:

{
  "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "factoryName": {
      "type": "String",
      "metadata": "ADF name"
    },
    "name": {
      "type": "String",
      "metadata": "Managed private endpoint name"
    },
    "linkedResourceId": {
      "type": "String",
      "metadata": "The linked resource id"
    },
    "linkedResourceGroupId": {
      "type": "String",
      "metadata": "The GroupId of the linked resource"
    },
    "linkedFqdn": {
      "type": "Array",
      "metadata": "The linked resource FQDN"
    }
  },
  "resources": [
    {
      "name": "[concat(parameters('factoryName'), '/default/', parameters('name'))]",
      "type": "Microsoft.DataFactory/factories/managedVirtualNetworks/managedPrivateEndpoints",
      "apiVersion": "2018-06-01",
      "properties": {
        "privateLinkResourceId": "[parameters('linkedResourceId')]",
        "groupId": "[parameters('linkedResourceGroupId')]",
        "fqdns": "[parameters('linkedFqdn')]"
      }
    }
  ]
}

[github-actions]

This functionality has been released in v2.68.0 of the Terraform Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[dkutetsky]

This actually, doesn't work - created Integrated Runtime has Virtual network configuration=Disable.

resource "azurerm_data_factory" "adf" {
  name                   = "some-adf"
  location               = "westeurope"
  resource_group_name    = azurerm_resource_group.rg.name
  public_network_enabled = false
  managed_virtual_network_enabled  = true

  identity {
    type = "SystemAssigned"
  }
}

resource "azurerm_resource_group" "rg" {
  name     = "adf-rg"
  location = "westeurope"
}

[michael-kolpin-bt]

This actually, doesn't work - created Integrated Runtime has Virtual network configuration=Disable.

resource "azurerm_data_factory" "adf" {
  name                   = "some-adf"
  location               = "westeurope"
  resource_group_name    = azurerm_resource_group.rg.name
  public_network_enabled = false
  managed_virtual_network_enabled  = true

  identity {
    type = "SystemAssigned"
  }
}

resource "azurerm_resource_group" "rg" {
  name     = "adf-rg"
  location = "westeurope"
}

This still seems to be wrongly deploying the ADF instance. Should this ticket be opened again or is there already a fix to be deployed?

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.