fix(panos_security_rule): state merged with existing values

[alperenkose]

Description

When state: merged is used with panos_security_rule for existing entries, it adds default values to existing configuration, like "any" being added to the list of source_zone, or source_ip. And similar issue exist for all the fields with default values in the module.

This PR removes "default" values from sdk_params in module which results in None (null) values for non-provided arguments, and arguments with null values in the invocation are not updated on the device for updating an existing rule. Whereas if rule doesn't exist a new record is being created with default values provided in the newly introduced default_values helper param. If no default values exist in default_values, it fallbacks to SDK default values, and if none found it is set to None.

PR also introduces preset_values for module parameters, which are available fixed selections for list type parameters like "any" or "application-default" for service param. With the addition of preset_values, it fixes updating the list parameters to or from the preset values or user defined entries.

Motivation and Context

Fixes issue #564 but we will keep it open for a while when we have the change in development branch.
fixes #563, fixes #467
Also fixes #551 regarding panos_template mode xpath error on second run

How Has This Been Tested?

Tested manually on vmseries.

Similar issue exists for modules using "default" values in the sdk_params like panos_zone , panos_pbf_rule or others which causes existing values to revert to default if not provided with the merged state. These will be fixed on a separate PR.

Types of changes

• Bug fix (non-breaking change which fixes an issue)

Checklist

• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes if appropriate.
• All new and existing tests passed.

[adambaumeister]

I believe I understand the intent and functionality of these changes - from what I can see this solution is good and shouldn't add too much additional handling within the Ansible collection. I also like that we don't have to fix the entire thing at once or change the underlying modules considerably.

[alperenkose]

We need to clarify how to document module default values while they are not in spec but actually applied with present or merged states while creating new resources.

[adambaumeister]

This looks good now and based on our feedback from the Red Hat team it's the only path forward.

[dmurarasu]

This seems to have broken the fix from #314 and now all security rules show as if they had changes even when no changes were made.

[alperenkose]

Hi @dmurarasu could you please send an example task and output of the issue?

[dmurarasu]

Hi @alperenkose, sorry, I probably should have raised an issue, details below
This is ran on a PA-440 unit running 11.1.2

ansible@a801c38859c8:/ansible$ ansible-galaxy collection install paloaltonetworks.panos
Starting galaxy collection install process
Process install dependency map
Starting collection install process
Downloading https://galaxy.ansible.com/api/v3/plugin/ansible/content/published/collections/artifacts/paloaltonetworks-panos-2.21.2.tar.gz to /home/ansible/.ansible/tmp/ansible-local-14wf7mps93/tmpev8pz_j5/paloaltonetworks-panos-2.21.2-14miom3s
Installing 'paloaltonetworks.panos:2.21.2' to '/home/ansible/.ansible/collections/ansible_collections/paloaltonetworks/panos'
paloaltonetworks.panos:2.21.2 was installed successfully

I'm just running a few rules in this example, just to see the changed output; the rules are already in place but I'll run it twice to show it's the same every time.

ansible@a801c38859c8:/ansible$ ansible-playbook -i local.inventory config_playbook.yml --tags=security_rules
PLAY [palo] ***********************************************************************************************************************

TASK [palo_configure : Create security rules] *************************************************************************************
changed: [pa01] => (item={'rule_name': 'Dynamic Block List Inbound', 'source_zone': ['any'], 'source_ip': ['Cisco Talos', 'panw-torexit-ip-list', 'panw-bulletproof-ip-list', 'panw-highrisk-ip-list', 'panw-known-ip-list'], 'destination_zone': ['any'], 'destination_ip': ['any'], 'application': ['any'], 'service': ['any'], 'action': 'drop', 'location': 'top', 'description': 'Blocking known bad IPs inbound'})
changed: [pa01] => (item={'rule_name': 'Dynamic Block List Outbound', 'source_zone': ['any'], 'source_ip': ['any'], 'destination_zone': ['any'], 'destination_ip': ['Cisco Talos', 'panw-torexit-ip-list', 'panw-bulletproof-ip-list', 'panw-highrisk-ip-list', 'panw-known-ip-list'], 'application': ['any'], 'service': ['any'], 'action': 'drop', 'location': 'after', 'existing_rule': 'Dynamic Block List Inbound', 'description': 'Blocking known bad IPs outbound'})
changed: [pa01] => (item={'rule_name': 'GP Connection', 'source_zone': ['External_LAN'], 'source_ip': ['any'], 'destination_zone': ['External_LAN'], 'destination_ip': ['any'], 'application': ['ipsec', 'panos-global-protect', 'ssl'], 'service': ['application-default'], 'action': 'allow', 'description': 'Permit GP Gateway Login', 'group_profile': 'Inbound Network Security', 'tag': ['Global Protect'], 'location': 'after', 'existing_rule': 'Dynamic Block List Outbound'})

PLAY RECAP ************************************************************************************************************************
pa01 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

ansible@a801c38859c8:/ansible$ ansible-playbook -i local.inventory config_playbook.yml --tags=security_rules
PLAY [palo] ***********************************************************************************************************************

TASK [palo_configure : Create security rules] *************************************************************************************
changed: [pa01] => (item={'rule_name': 'Dynamic Block List Inbound', 'source_zone': ['any'], 'source_ip': ['Cisco Talos', 'panw-torexit-ip-list', 'panw-bulletproof-ip-list', 'panw-highrisk-ip-list', 'panw-known-ip-list'], 'destination_zone': ['any'], 'destination_ip': ['any'], 'application': ['any'], 'service': ['any'], 'action': 'drop', 'location': 'top', 'description': 'Blocking known bad IPs inbound'})
changed: [pa01] => (item={'rule_name': 'Dynamic Block List Outbound', 'source_zone': ['any'], 'source_ip': ['any'], 'destination_zone': ['any'], 'destination_ip': ['Cisco Talos', 'panw-torexit-ip-list', 'panw-bulletproof-ip-list', 'panw-highrisk-ip-list', 'panw-known-ip-list'], 'application': ['any'], 'service': ['any'], 'action': 'drop', 'location': 'after', 'existing_rule': 'Dynamic Block List Inbound', 'description': 'Blocking known bad IPs outbound'})
changed: [pa01] => (item={'rule_name': 'GP Connection', 'source_zone': ['External_LAN'], 'source_ip': ['any'], 'destination_zone': ['External_LAN'], 'destination_ip': ['any'], 'application': ['ipsec', 'panos-global-protect', 'ssl'], 'service': ['application-default'], 'action': 'allow', 'description': 'Permit GP Gateway Login', 'group_profile': 'Inbound Network Security', 'tag': ['Global Protect'], 'location': 'after', 'existing_rule': 'Dynamic Block List Outbound'})

PLAY RECAP ************************************************************************************************************************
pa01 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

Just to show it was ok before

ansible@a801c38859c8:/ansible$ ansible-galaxy collection install paloaltonetworks.panos:2.20.0
Starting galaxy collection install process
Process install dependency map
Starting collection install process
Downloading https://galaxy.ansible.com/api/v3/plugin/ansible/content/published/collections/artifacts/paloaltonetworks-panos-2.20.0.tar.gz to /home/ansible/.ansible/tmp/ansible-local-37472fsj_7b/tmpx42tjxmv/paloaltonetworks-panos-2.20.0-oqng52gi
Installing 'paloaltonetworks.panos:2.20.0' to '/home/ansible/.ansible/collections/ansible_collections/paloaltonetworks/panos'
paloaltonetworks.panos:2.20.0 was installed successfully

ansible@a801c38859c8:/ansible$ ansible-playbook -i local.inventory config_playbook.yml --tags=security_rules

PLAY [palo] ***********************************************************************************************************************

TASK [palo_configure : Create security rules] *************************************************************************************
ok: [pa01] => (item={'rule_name': 'Dynamic Block List Inbound', 'source_zone': ['any'], 'source_ip': ['Cisco Talos', 'panw-torexit-ip-list', 'panw-bulletproof-ip-list', 'panw-highrisk-ip-list', 'panw-known-ip-list'], 'destination_zone': ['any'], 'destination_ip': ['any'], 'application': ['any'], 'service': ['any'], 'action': 'drop', 'location': 'top', 'description': 'Blocking known bad IPs inbound'})
ok: [pa01] => (item={'rule_name': 'Dynamic Block List Outbound', 'source_zone': ['any'], 'source_ip': ['any'], 'destination_zone': ['any'], 'destination_ip': ['Cisco Talos', 'panw-torexit-ip-list', 'panw-bulletproof-ip-list', 'panw-highrisk-ip-list', 'panw-known-ip-list'], 'application': ['any'], 'service': ['any'], 'action': 'drop', 'location': 'after', 'existing_rule': 'Dynamic Block List Inbound', 'description': 'Blocking known bad IPs outbound'})
ok: [pa01] => (item={'rule_name': 'GP Connection', 'source_zone': ['External_LAN'], 'source_ip': ['any'], 'destination_zone': ['External_LAN'], 'destination_ip': ['any'], 'application': ['ipsec', 'panos-global-protect', 'ssl'], 'service': ['application-default'], 'action': 'allow', 'description': 'Permit GP Gateway Login', 'group_profile': 'Inbound Network Security', 'tag': ['Global Protect'], 'location': 'after', 'existing_rule': 'Dynamic Block List Outbound'})

PLAY RECAP ************************************************************************************************************************
pa01 : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

When it detects changes all the time and I run it with verbosity the only changes that I can see in the before and after section:

before:
"group_profile": [
"Inbound Network Security"
],
"group_tag": null,
"hip_profiles": null,

after:
"group_profile": "Inbound Network Security",
"group_tag": null,
"hip_profiles": [
"any"
],

Commiting and rerunning just shows the same again.

Full output below:

ansible@a801c38859c8:/ansible$ ansible-playbook -i local.inventory config_playbook.yml --tags=security_rules -vvvv
ansible-playbook [core 2.15.1]
config file = None
configured module search path = ['/home/ansible/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python3.10/dist-packages/ansible
ansible collection location = /home/ansible/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/local/bin/ansible-playbook
python version = 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] (/usr/bin/python3)
jinja version = 3.1.2
libyaml = True
No config file found; using defaults
setting up inventory plugins
Loading collection ansible.builtin from
host_list declined parsing /ansible/local.inventory as it did not pass its verify_file() method
auto declined parsing /ansible/local.inventory as it did not pass its verify_file() method
yaml declined parsing /ansible/local.inventory as it did not pass its verify_file() method
Parsed /ansible/local.inventory inventory source with ini plugin
Loading collection paloaltonetworks.panos from /home/ansible/.ansible/collections/ansible_collections/paloaltonetworks/panos
statically imported: /ansible/roles/palo_configure/tasks/mgmt_config.yml
statically imported: /ansible/roles/palo_configure/tasks/email_profile.yml
statically imported: /ansible/roles/palo_configure/tasks/zones.yml
statically imported: /ansible/roles/palo_configure/tasks/interfaces.yml
statically imported: /ansible/roles/palo_configure/tasks/objects.yml
statically imported: /ansible/roles/palo_configure/tasks/security_profiles.yml
statically imported: /ansible/roles/palo_configure/tasks/security_rules.yml
statically imported: /ansible/roles/palo_configure/tasks/nat_rules.yml
statically imported: /ansible/roles/palo_configure/tasks/pbf_rules.yml
Loading callback plugin default of type stdout, v2.0 from /usr/local/lib/python3.10/dist-packages/ansible/plugins/callback/default.py
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: config_playbook.yml *************************************************************************************************************************************************************
Positional arguments: config_playbook.yml
verbosity: 4
connection: smart
timeout: 10
become_method: sudo
tags: ('security_rules',)
inventory: ('/ansible/local.inventory',)
forks: 5
1 plays in config_playbook.yml

PLAY [palo] *******************************************************************************************************************************************************************************

TASK [palo_configure : Create security rules] *********************************************************************************************************************************************
task path: /ansible/roles/palo_configure/tasks/security_rules.yml:1
EXEC /bin/sh -c 'echo ~ansible && sleep 0'
EXEC /bin/sh -c '( umask 77 && mkdir -p "echo /home/ansible/.ansible/tmp"&& mkdir "echo /home/ansible/.ansible/tmp/ansible-tmp-1728555898.4642293-732-217420695437818" && echo ansible-tmp-1728555898.4642293-732-217420695437818="echo /home/ansible/.ansible/tmp/ansible-tmp-1728555898.4642293-732-217420695437818" ) && sleep 0'
Using module file /home/ansible/.ansible/collections/ansible_collections/paloaltonetworks/panos/plugins/modules/panos_security_rule.py
PUT /home/ansible/.ansible/tmp/ansible-local-727vaemhhyn/tmpfb9sfkfk TO /home/ansible/.ansible/tmp/ansible-tmp-1728555898.4642293-732-217420695437818/AnsiballZ_panos_security_rule.py
EXEC /bin/sh -c 'chmod u+x /home/ansible/.ansible/tmp/ansible-tmp-1728555898.4642293-732-217420695437818/ /home/ansible/.ansible/tmp/ansible-tmp-1728555898.4642293-732-217420695437818/AnsiballZ_panos_security_rule.py && sleep 0'
EXEC /bin/sh -c '/usr/bin/python3 /home/ansible/.ansible/tmp/ansible-tmp-1728555898.4642293-732-217420695437818/AnsiballZ_panos_security_rule.py && sleep 0'
EXEC /bin/sh -c 'rm -f -r /home/ansible/.ansible/tmp/ansible-tmp-1728555898.4642293-732-217420695437818/ > /dev/null 2>&1 && sleep 0'

changed: [pa01] => (item={'rule_name': 'GP Connection', 'source_zone': ['External_LAN'], 'source_ip': ['any'], 'destination_zone': ['External_LAN'], 'destination_ip': ['any'], 'application': ['ipsec', 'panos-global-protect', 'ssl'], 'service': ['application-default'], 'action': 'allow', 'description': 'Permit GP Gateway Login', 'group_profile': 'Inbound Network Security', 'tag': ['Global Protect'], 'location': 'after', 'existing_rule': 'Dynamic Block List Outbound'}) => {
"after": {
"action": "allow",
"antivirus": null,
"application": [
"ipsec",
"panos-global-protect",
"ssl"
],
"category": [
"any"
],
"data_filtering": null,
"description": "Permit GP Gateway Login",
"destination_devices": [
"any"
],
"destination_ip": [
"any"
],
"destination_zone": [
"External_LAN"
],
"disable_server_response_inspection": false,
"disabled": false,
"file_blocking": null,
"group_profile": "Inbound Network Security",
"group_tag": null,
"hip_profiles": [
"any"
],
"icmp_unreachable": null,
"log_end": true,
"log_setting": null,
"log_start": false,
"negate_destination": false,
"negate_source": false,
"negate_target": null,
"rule_name": "GP Connection",
"rule_type": "universal",
"schedule": null,
"service": [
"application-default"
],
"source_devices": [
"any"
],
"source_ip": [
"any"
],
"source_user": [
"any"
],
"source_zone": [
"External_LAN"
],
"spyware": null,
"tag_name": null,
"target": null,
"url_filtering": null,
"uuid": "e74b64af-9806-4011-bf9a-23f2fc41fe28",
"vulnerability": null,
"wildfire_analysis": null
},
"ansible_loop_var": "item",
"before": {
"action": "allow",
"antivirus": null,
"application": [
"ipsec",
"panos-global-protect",
"ssl"
],
"category": [
"any"
],
"data_filtering": null,
"description": "Permit GP Gateway Login",
"destination_devices": [
"any"
],
"destination_ip": [
"any"
],
"destination_zone": [
"External_LAN"
],
"disable_server_response_inspection": false,
"disabled": false,
"file_blocking": null,
"group_profile": [
"Inbound Network Security"
],
"group_tag": null,
"hip_profiles": null,
"icmp_unreachable": null,
"log_end": true,
"log_setting": null,
"log_start": false,
"negate_destination": false,
"negate_source": false,
"negate_target": null,
"rule_name": "GP Connection",
"rule_type": "universal",
"schedule": null,
"service": [
"application-default"
],
"source_devices": [
"any"
],
"source_ip": [
"any"
],
"source_user": [
"any"
],
"source_zone": [
"External_LAN"
],
"spyware": null,
"tag_name": null,
"target": null,
"url_filtering": null,
"uuid": "e74b64af-9806-4011-bf9a-23f2fc41fe28",
"vulnerability": null,
"wildfire_analysis": null
},
"changed": true,
"diff": {
"after": "\n<entry name="GP Connection" uuid="e74b64af-9806-4011-bf9a-23f2fc41fe28">\n\t\n\t\tExternal_LAN\n\t\n\t\n\t\tExternal_LAN\n\t\n\t\n\t\tany\n\t\n\t\n\t\tany\n\t\n\t\n\t\tany\n\t\n\t\n\t\tipsec\n\t\tpanos-global-protect\n\t\tssl\n\t\n\t\n\t\tapplication-default\n\t\n\t\n\t\tany\n\t\n\tallow\n\tno\n\tyes\n\tPermit GP Gateway Login\n\tuniversal\n\tno\n\tno\n\tno\n\t\n\t\tno\n\t\n\t\n\t\t\n\t\t\tInbound Network Security\n\t\t\n\t\n\t\n\t\tany\n\t\n\t\n\t\tany\n\t\n\n",
"before": "\n<entry name="GP Connection" uuid="e74b64af-9806-4011-bf9a-23f2fc41fe28">\n\t\n\t\tExternal_LAN\n\t\n\t\n\t\tExternal_LAN\n\t\n\t\n\t\tany\n\t\n\t\n\t\tany\n\t\n\t\n\t\tany\n\t\n\t\n\t\tipsec\n\t\tpanos-global-protect\n\t\tssl\n\t\n\t\n\t\tapplication-default\n\t\n\t\n\t\tany\n\t\n\tallow\n\tno\n\tyes\n\tPermit GP Gateway Login\n\tuniversal\n\tno\n\tno\n\tno\n\t\n\t\tno\n\t\n\t\n\t\t\n\t\t\tInbound Network Security\n\t\t\n\t\n\t\n\t\tany\n\t\n\t\n\t\tany\n\t\n\n"
},
"invocation": {
"module_args": {
"action": "allow",
"antivirus": null,
"api_key": null,
"application": [
"ipsec",
"panos-global-protect",
"ssl"
],
"audit_comment": null,
"category": null,
"commit": false,
"data_filtering": null,
"description": "Permit GP Gateway Login",
"destination_ip": [
"any"
],
"destination_zone": [
"External_LAN"
],
"device_group": "shared",
"devicegroup": null,
"disable_server_response_inspection": null,
"disabled": null,
"existing_rule": "Dynamic Block List Outbound",
"file_blocking": null,
"gathered_filter": null,
"group_profile": "Inbound Network Security",
"group_tag": null,
"hip_profiles": null,
"icmp_unreachable": null,
"ip_address": null,
"location": "after",
"log_end": null,
"log_setting": null,
"log_start": null,
"negate_destination": null,
"negate_source": null,
"negate_target": null,
"password": null,
"port": 443,
"provider": {
},
"rule_name": "GP Connection",
"rule_type": null,
"rulebase": null,
"schedule": null,
"service": [
"application-default"
],
"source_ip": [
"any"
],
"source_user": null,
"source_zone": [
"External_LAN"
],
"spyware": null,
"state": "present",
"tag_name": null,
"target": null,
"url_filtering": null,
"username": "admin",
"uuid": null,
"vsys": "vsys1",
"vulnerability": null,
"wildfire_analysis": null
}
},
"item": {
"action": "allow",
"application": [
"ipsec",
"panos-global-protect",
"ssl"
],
"description": "Permit GP Gateway Login",
"destination_ip": [
"any"
],
"destination_zone": [
"External_LAN"
],
"existing_rule": "Dynamic Block List Outbound",
"group_profile": "Inbound Network Security",
"location": "after",
"rule_name": "GP Connection",
"service": [
"application-default"
],
"source_ip": [
"any"
],
"source_zone": [
"External_LAN"
],
"tag": [
"Global Protect"
]
}
}

[alperenkose]

@dmurarasu could you please create an issue for this to be tracked with the config_playbook.yml content you have run?