Delete VM workflow should attempt to remove external addresses in ipv4_nat_entry table

[askfongjojo]

Currently external IP address entries are deleted during instance-stop operations. There are atypical lifecycle situations when instances are deleted without being explicitly stopped, e.g. when an instance is left in running state prior to a system update. The instance in this case transitions to failed state when user attempts to reboot it. If the user deletes the instance, the snat and external IP addresses in the ip_nat_entry table will remain as is while they are marked as deleted in the external_ip table.

I've run into such a situation on rack2. The end result was that a newly created VM could acquire the external ip address released from a failed VM after it's been deleted but would fail to start up. Here is the 500 error I saw in the Nexus log:

01:04:00.050Z INFO 65a11c18-7f59-41ac-b9e7-680627f996e7 (dropshot_external): request completed
    error_message_external = Internal Server Error
    error_message_internal = saga ACTION error at node "dpd_ensure": unexpected database error: duplicate key value violates unique constraint "overlapping_ipv4_nat_entry"
    file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:841
    latency_us = 413890
    local_addr = 172.30.2.5:443
    method = POST
    remote_addr = 172.20.17.42:50510
    req_id = dff3c7b1-5fea-4597-b5c2-2763a0bdcfb5
    response_code = 500
    uri = https://oxide.sys.rack2.eng.oxide.computer/v1/instances/failed-vm/start?project=try

The ipv4_nat_entry will likely not have any active IP records for a stopped VM in most cases so the suggestion here is to attempt deletion only if such data exist, but do not complain if there isn't anything to delete.