Command: terraform destroy with VRF-Lite isn't completing

[raymonddejong]

When using terraform destroy to clean up the VRF-Lite topology it fails to remove the Default Tier-0, it's uplinks and the two Segments it is using to connect to the physical network.

You can reproduce using my repository here: https://github.com/raymonddejong/terraform-nsxt/tree/master/VRF-Lite

Output below.

Plan: 0 to add, 0 to change, 35 to destroy.

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

nsxt_policy_tier0_gateway_interface.vrf_uplink2: Destroying... [id=e5f69fdb-cce6-46ec-8224-894298a6d1e3]
nsxt_policy_bgp_neighbor.vrf_blue_router_b: Destroying... [id=57a6ce24-5636-40b8-9d42-0583ebfbdfd5]
nsxt_policy_tier0_gateway_interface.vrf_uplink1: Destroying... [id=c8b5a31c-aaf3-4948-85d7-5196f1a0aab5]
nsxt_policy_bgp_neighbor.vrf_blue_router_a: Destroying... [id=fad02cf5-f509-4a05-b2b0-a3a847f175db]
nsxt_policy_bgp_neighbor.vrf_red_router_a: Destroying... [id=a57e7e7b-d250-4f48-9d23-d97bfb87dffb]
nsxt_policy_bgp_neighbor.router_a: Destroying... [id=e5dfe0da-cd61-4088-8c71-bde1f489b947]
nsxt_policy_vlan_segment.vlan106: Destroying... [id=465d8ebd-1a43-470e-8dd4-b08df9f2b09a]
nsxt_policy_bgp_neighbor.router_b: Destroying... [id=5a6f19fc-fa14-4de5-a831-ebac71e68293]
nsxt_policy_segment.blue_segment_1: Destroying... [id=b15a572c-3223-48bb-b3e2-ad84a85f50fe]
nsxt_policy_segment.red_segment_2: Destroying... [id=34661b35-d747-4eba-940e-4b51f98c5ad2]
nsxt_policy_tier0_gateway_interface.blue_vrf_uplink1: Destroying... [id=65f01049-0c8c-4c2d-aa6c-f904331efd4d]
nsxt_policy_vlan_segment.vlan103: Destroying... [id=83f6a5d0-29b8-4f34-b2a3-cc011ae2022d]
nsxt_policy_bgp_neighbor.router_a: Destruction complete after 2s
nsxt_policy_bgp_neighbor.vrf_blue_router_b: Destruction complete after 2s
nsxt_policy_bgp_neighbor.vrf_red_router_a: Destruction complete after 2s
nsxt_policy_bgp_neighbor.router_b: Destruction complete after 2s
nsxt_policy_bgp_neighbor.vrf_blue_router_a: Destruction complete after 2s
nsxt_policy_tier0_gateway_interface.red_vrf_uplink2: Destroying... [id=24a1288e-2680-4241-bf40-b525508176be]
nsxt_policy_tier0_gateway_interface.blue_vrf_uplink2: Destroying... [id=3173b6b4-7d8a-4c61-bafc-143993d77dbb]
nsxt_policy_tier0_gateway_interface.red_vrf_uplink1: Destroying... [id=a852ac94-7558-45b7-86b7-4762d17ba650]
nsxt_policy_tier0_gateway_interface.blue_vrf_uplink1: Destruction complete after 1s
nsxt_policy_group.red_servers: Destroying... [id=b45f72ac-ed24-4291-817d-1b96788c2c4e]
nsxt_policy_bgp_neighbor.vrf_red_router_b: Destroying... [id=f2735dbd-2ee5-4bf7-8367-beaf8349e7ce]
nsxt_policy_vlan_segment.vlan104: Destroying... [id=6c6571d3-01f8-4479-a0b9-586a9e716017]
nsxt_policy_tier0_gateway_interface.blue_vrf_uplink2: Destruction complete after 0s
nsxt_policy_security_policy.allow_blue: Destroying... [id=f0f5d1f7-4c03-4f0e-8ba4-ac9340f7ccb4]
nsxt_policy_tier0_gateway_interface.red_vrf_uplink2: Destruction complete after 0s
nsxt_policy_vlan_segment.vlan105: Destroying... [id=096972e7-e501-4153-94e6-3cf3d1a4a558]
nsxt_policy_tier0_gateway_interface.red_vrf_uplink1: Destruction complete after 0s
nsxt_policy_segment.red_segment_1: Destroying... [id=3964dee1-f293-4ff8-96c5-80028c3d9d1c]
nsxt_policy_bgp_neighbor.vrf_red_router_b: Destruction complete after 0s
nsxt_policy_vlan_segment.vrf_trunk_b: Destroying... [id=3918aaf2-639f-45a8-a936-e2908a4f5086]
nsxt_policy_vlan_segment.vlan106: Destruction complete after 2s
nsxt_policy_vlan_segment.vrf_trunk_a: Destroying... [id=eee25da1-cd72-4c6c-b86c-d10224d854da]
nsxt_policy_segment.red_segment_2: Destruction complete after 2s
nsxt_policy_segment.blue_segment_1: Destruction complete after 2s
nsxt_policy_tier1_gateway.tier1_gw_blue: Destroying... [id=6a996a50-49a6-469f-892b-0d1c3fc3af18]
nsxt_policy_group.red_servers: Destruction complete after 1s
nsxt_policy_vlan_segment.vlan103: Destruction complete after 2s
nsxt_policy_tier1_gateway.tier1_gw_blue: Destruction complete after 1s
nsxt_policy_tier0_gateway.blue_vrf: Destroying... [id=6355b9be-92ca-4cf2-a1d1-8c8e58da24b9]
nsxt_policy_security_policy.allow_blue: Destruction complete after 2s
nsxt_policy_group.web_servers: Destroying... [id=3cff0d74-5e24-4c8d-87db-3b46b3e2029f]
nsxt_policy_service.service_tcp8443: Destroying... [id=3b3b32c7-f43d-4abd-a90f-7bc01c082891]
nsxt_policy_group.db_servers: Destroying... [id=d6fc113e-bf8d-4251-bf67-fa133c730b08]
nsxt_policy_group.app_servers: Destroying... [id=09fd6ae1-9e80-41b2-83e5-2dacdd730b93]
nsxt_policy_vlan_segment.vlan104: Destruction complete after 2s
nsxt_policy_group.blue_servers: Destroying... [id=f8ed6c4f-4e17-47b5-a6e3-4349f7c9e3da]
nsxt_policy_service.service_tcp8443: Destruction complete after 1s
nsxt_policy_tier0_gateway.blue_vrf: Destruction complete after 3s
nsxt_policy_vlan_segment.vrf_trunk_b: Destruction complete after 4s
nsxt_policy_segment.red_segment_1: Destruction complete after 4s
nsxt_policy_tier1_gateway.tier1_gw_red: Destroying... [id=eec166ad-3168-4c4a-9abd-737200b717e2]
nsxt_policy_group.web_servers: Destruction complete after 2s
nsxt_policy_vlan_segment.vrf_trunk_a: Destruction complete after 4s
nsxt_policy_group.db_servers: Destruction complete after 2s
nsxt_policy_group.app_servers: Destruction complete after 2s
nsxt_policy_vlan_segment.vlan105: Destruction complete after 4s
nsxt_policy_group.blue_servers: Destruction complete after 3s
nsxt_policy_tier1_gateway.tier1_gw_red: Destruction complete after 3s
nsxt_policy_tier0_gateway.red_vrf: Destroying... [id=63cba0a3-c982-45e6-a5a8-a2cd3258d12b]
nsxt_policy_tier0_gateway.red_vrf: Destruction complete after 1s

Error:  Failed to delete Tier0 Interface c8b5a31c-aaf3-4948-85d7-5196f1a0aab5: The object path=[/infra/tier-0s/2afe6343-0021-48b4-9f0e-685bee410160/locale-services/default/interfaces/c8b5a31c-aaf3-4948-85d7-5196f1a0aab5] cannot be deleted as either it has children or it is being referenced by other objects path=[/infra/tier-0s/6355b9be-92ca-4cf2-a1d1-8c8e58da24b9,/infra/tier-0s/63cba0a3-c982-45e6-a5a8-a2cd3258d12b] (code 500030)

Error:  Failed to delete Tier0 Interface e5f69fdb-cce6-46ec-8224-894298a6d1e3: The object path=[/infra/tier-0s/2afe6343-0021-48b4-9f0e-685bee410160/locale-services/default/interfaces/e5f69fdb-cce6-46ec-8224-894298a6d1e3] cannot be deleted as either it has children or it is being referenced by other objects path=[/infra/tier-0s/6355b9be-92ca-4cf2-a1d1-8c8e58da24b9,/infra/tier-0s/63cba0a3-c982-45e6-a5a8-a2cd3258d12b] (code 500030)

As you can see on the screenshots below these are the objects not being removed:
Tier-0

Tier-0 Uplinks

Segments:

[annakhm]

Hi @raymonddejong, thanks for filing this. There is in fact a dependency between parent interface and VRF interface, which terraform doesn't know about. The way to address this for now is to specify the dependency explicitly on each vrf interface, for example:
depends_on = [nsxt_policy_tier0_gateway_interface.vrf_uplink1]

I have added this note to the docs here #478

[raymonddejong]

Hi @annakhm, I tested this just now and this only fixed a part of the problem. I still got:
Error: Failed to delete Tier0 094b26ac-df5a-49de-ba45-73a3f2498478: The object path=[/infra/tier-0s/094b26ac-df5a-49de-ba45-73a3f2498478/locale-services/default/bgp] cannot be deleted as either it has children or it is being referenced by other objects path=[/infra/tier-0s/fc932fe9-771c-45a6-bdb5-4c614d28cf69/locale-services/default/bgp,/infra/tier-0s/fc932fe9-771c-45a6-bdb5-4c614d28cf69,/infra/tier-0s/1f39a38d-dfcf-4cde-a73f-092984f6ee56,/infra/tier-0s/1f39a38d-dfcf-4cde-a73f-092984f6ee56/locale-services/default/bgp] (code 500030)
Now the Uplink interfaces are successfully deleted but the default Tier-0 isn't.

I tried to apply the same trick by adding depends_on = [nsxt_policy_tier0_gateway.vrf_tier0_gw] in the VRF tier-0 but no luck.

[annakhm]

Unfortunately this is platform issue, we hope to fix it with retry mechanism once its available in the SDK we are using.