Importing T1 information failing on VMC-NSX-T

[chanakanissanka]

Describe the bug

Hi There,
We have pre-created T1’s, the CGW and the MGW. They are pre-created with some default rules to allow outbound internet access. These rules need to be imported into our Terraform code. We are getting import errors and appreciate some help on resolving those.

Reproduction steps

1. created the resource
2. MGW definition
   resource "nsxt_policy_gateway_policy" "mgw_policy" {
   category = "LocalGatewayRules"
   display_name = "default"
   domain = "mgw"
   }

CGW definition
resource "nsxt_policy_gateway_policy" "cgw_policy" {
category = "LocalGatewayRules"
display_name = "default"
domain = "cgw"
}

2. Issue the command bellow
   terraform import nsxt_policy_gateway_policy.cgw_policy cgw/default

terraform import nsxt_policy_gateway_policy.mgw_policy mgw/default

3.Getting the error bellow due to the resource we created above. Can we get some help here ? what am I missing here ?
sxt_policy_gateway_policy.cgw_policy: Import prepared!
Prepared nsxt_policy_gateway_policy for import
data.nsxt_policy_security_policy.predefined: Read complete after 0s [id=default-layer3-section]
╷
│ Error: Resource already managed by Terraform
│
│ Terraform is already managing a remote object for nsxt_policy_gateway_policy.cgw_policy. To import to this address you must first remove the existing object from the state.

...

Expected behavior

Import to be ok and I can get those into my tf state file

Additional context

No response

[2ez4szliu]

Hi @chanakanissanka,
I think the command you use is correct. I see that the provider is reporting this error

│ Error: Resource already managed by Terraform

This only happens when the object you are importing is already in the terraform.tfstate file, which means it was created or already imported by the provider.
So in step 1 when you create the resource, did you do that via NSX UI/API or did you create it using the provider?

The step for importing a resource from NSX is

1. Create the resource via NSX API/UI
2. Create resource definition .tf file
3. Import resource with terraform import command

I think one way to fix the situation is to go to the terraform.tfstate file, find the resource name mgw_policy and remove it from state. Once it's removed you should be able to import it with terraform import command.

Please let me know if it works.
Thanks,
Shizhao

[chanakanissanka]

@2ez4szliu Thanks for the quick reply, right after I opened this case and tried again and finally make it happen,
nsxt_policy_security_policy.IDMZIN: Refreshing state... [id=65baa84d-e7b4-429c-b585-9131221ddfa9]

Import successful!

The resources that were imported are shown above. These resources are now in
Your Terraform swill henceforth be managed by Terraform.

Thanks for that,
I got few distributed firewall policies with 3000+ rules in it. I'm wondering what would be the way of doing it...some advice much appreciated at this stage

[2ez4szliu]

@chanakanissanka I would suggest creating several policy sections for your distributed firewall, each contains no more than 1000 rules because the maximum number of rules in one policy supported by nsxt provider is 1000.

[chanakanissanka]

Importing those rules into the state seems to be messy and hairy. Is there any other way to refer to pre-created rules, like a resource?
Something like this ?
data "nsxt_policy_security_policy" "default_l3" {
is_default = true
category = "Application"
}

resource "nsxt_policy_predefined_security_policy" "test" {
path = data.nsxt_policy_security_policy.default_l3.path

tag {
scope = "color"
tag = "orange"
}

rule {
display_name = "allow_icmp"
destination_groups = [nsxt_policy_group.cats.path, nsxt_policy_group.dogs.path]
action = "ALLOW"
services = [nsxt_policy_service.icmp.path]
logged = true
}

rule {
display_name = "allow_udp"
source_groups = [nsxt_policy_group.fish.path]
sources_excluded = true
scope = [nsxt_policy_group.aquarium.path]
action = "ALLOW"
services = [nsxt_policy_service.udp.path]
logged = true
disabled = true
}

default_rule {
action = "DROP"
}

}

[2ez4szliu]

Hi @chanakanissanka,

Is there any other way to refer to pre-created rules, like a resource?

Do you mean a resource for managing all DFW rules? From the template you provided seems you are creating a nsxt_policy_predefined_security_policy resource and define the rules in the policy's template, I assume you want a separate resource for DFWRule, probably something like this:

resource "nsxt_policy_distributed_firewall_rule" rule1 {
display_name = "allow_udp"
source_groups = [nsxt_policy_group.fish.path]
sources_excluded = true
scope = [nsxt_policy_group.aquarium.path]
action = "ALLOW"
services = [nsxt_policy_service.udp.path]
logged = true
disabled = true
}

and then create security policy like this:

resource "nsxt_policy_predefined_security_policy" "test" {
path = data.nsxt_policy_security_policy.default_l3.path
rules = [nsxt_policy_distributed_firewall_rule.rule1, nsxt_policy_distributed_firewall_rule.rule2]
}

Is this what you are asking for?
Thanks,
Shizhao

[2ez4szliu]

a correction to the above comment: I don't think we can first create the rule then use it in security policy template to create the policy because we cannot create the rule without the policy, but I understand that you want a separate resource to refer to the rules in a policy.

If you only want to refer to a specific rule, I think a data source for DFW rule might be able to do that.
I'm not sure if we can have a resource to manage the rules in a security policy and do update on the rules without changing the security policy resource.
cc @annakhm to see if she has some other insights on this.

[chanakanissanka]

@2ez4szliu thanks for the reploy so far mate,
I have added a snap of our VMC-NSX-t and what we want is a mechanism to manage this via terraform. These are pre-created via RestNSX as part of migrations. If we are to have pipeline with terraform how best we make sure these exists rules are aware to our state ? Hope that will clarify ?

[2ez4szliu]

@chanakanissanka I think these rules are already aware by the state once you have imported the policy with terraform, you can view its state byterraform state show nsxt_policy_gateway_policy.mgw_policy, but as you mentioned earlier you find this to be a little messy, so what I'm thinking is a data source for a rule.

you can create data source like:
data "nsxt_policy_rule" "rule1" {
dispaly_name = "rule1"
domain = "default"
gateway = "mgw"
}
with this data source you can get the state of an individual rule without having to look at the entire policy. Does this satisfy your need?
However if you want to modify a rule in a policy, I would still recommend doing it in the policy resource instead of having a separate resource for each rule.

Shizhao

[chanakanissanka]

Hi There, thanks again
I tried this got this "The provider vmware/nsxt does not support data source "nsxt_policy_rule". My provier as follows, Any changes required ?
terraform {
required_providers {
nsxt = {
source = "vmware/nsxt"
}

}

}

[2ez4szliu]

Hi @chanakanissanka sorry for the confusion, this data source is currently not supported by nsxt provider. I just want to check with you if this would sufficient for you to managing your rule before we start to implement it.

[chanakanissanka]

@2ez4szliu I see. That will be ok since we not gonna update all rules at once, wonder normally how log will it take to have this feature ?

[2ez4szliu]

We can include this feature in the next release v3.4.1, which is currently due on Jan 11 2024 according to the milestone v3.4.1

https://github.com/vmware/terraform-provider-nsxt/milestone/1

[chanakanissanka]

@2ez4szliu appreciate these details. Just to summarise what we discussed,
Currently, if we want to refer to pre-created rules under pre-created policies into terraform, there is no other way, and the data source for each rule would be the approach which will be released as per the milestone above. Would this be a fair assumption to be made at this stage?

[2ez4szliu]

@chanakanissanka The above is correct, I will let you know if anything changes

[chanakanissanka]

@2ez4szliu One last thing to clarify

1. Can I import DFW policy sets and rules created by clickops into terraform?
2. Is there a limit on the amount of rules within a policy that we can import

[2ez4szliu]

For 1. As long as these policies and rules are successfully created on NSX side then we can import them into terraform
2. it's 1000

[chanakanissanka]

@2ez4szliu, you are right, and we managed to successfully import a few of those policy sets rules into the state, now the state is messy (with so many rules ) and needs to find a way to update this into code so the plan will not make noise.