Strange VPC diffs and failed updates after 'pulumi refresh': "Only 1 of gateway_id, egress_only_gateway_id, nat_gateway_id, instance_id, network_interface_id or vpc_peering_connection_id is allowed"

[ggilmore]

With https://github.com/pulumi/pulumi-awsx/releases/tag/v0.18.3 (Typescript) (and Pulumi CLI 0.17.10), I see some strange diffs to AWS VPC tags when I run pulumi up after a pulumi refresh.

Here is the code that I use to create the VPC:

const vpc = new awsx.ec2.Vpc(`${prefix}-vpc`, {
    // EKS requires at least two availability zones.
    numberOfAvailabilityZones: 3,

    // Ensure that there is a private AND public subnet in each AZ.
    // The Kubernetes worker nodes will be deployed to the private
    // subnet.
    //
    // The load balancers will be created in the public subnet.
    subnets: [{ type: 'public' }, { type: 'private' }],
})

Here is the pulumi up preview right after I run pulumi refresh:

 pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:ds-REDACTED-replica-release::sg-REDACTED-REDACTED-replica::pulumi:pulumi:Stack::sg-REDACTED-REDACTED-replica-ds-REDACTED-replica-release]
            ~ aws:ec2/route:Route: (update)
                [id=r-rtb-0981bff5ac5188cd91080289494]
                [urn=urn:pulumi:ds-REDACTED-replica-release::sg-REDACTED-REDACTED-replica::awsx:x:ec2:Vpc$awsx:x:ec2:Subnet$aws:ec2/route:Route::sourcegraph-REDACTED-REDACTED-replica-vpc-private-0-nat-0]
              + gatewayId: "nat-07f8b134c241f07f0"
            ~ aws:ec2/route:Route: (update)
                [id=r-rtb-0ffca6caaa8613ba91080289494]
                [urn=urn:pulumi:ds-REDACTED-replica-release::sg-REDACTED-REDACTED-replica::awsx:x:ec2:Vpc$awsx:x:ec2:Subnet$aws:ec2/route:Route::sourcegraph-REDACTED-REDACTED-replica-vpc-private-1-nat-1]
              + gatewayId: "nat-0b09091ded3d19792"
            ~ aws:ec2/route:Route: (update)
                [id=r-rtb-0a449cdf74c14f7601080289494]
                [urn=urn:pulumi:ds-REDACTED-replica-release::sg-REDACTED-REDACTED-replica::awsx:x:ec2:Vpc$awsx:x:ec2:Subnet$aws:ec2/route:Route::sourcegraph-REDACTED-REDACTED-replica-vpc-private-2-nat-2]
              + gatewayId: "nat-06e709f5d14313b1f"
        ~ aws:ec2/securityGroup:SecurityGroup: (update)
            [id=sg-0fba37d58cd28fb7b]
            [urn=urn:pulumi:ds-REDACTED-replica-release::sg-REDACTED-REDACTED-replica::eks:index:Cluster$aws:ec2/securityGroup:SecurityGroup::sourcegraph-REDACTED-REDACTED-replica-cluster-nodeSecurityGroup]
          ~ tags: {
              - kubernetes                                                                     : {
                  - io/cluster/sourcegraph-REDACTED-REDACTED-replica-cluster-eksCluster-3add07e: "owned"
                }
              + kubernetes.io/cluster/sourcegraph-REDACTED-REDACTED-replica-cluster-eksCluster-3add07e: "owned"
            }

Here is the pulumi up output after that refresh:

  aws:ec2:Route (sourcegraph-REDACTED-REDACTED-replica-vpc-private-0-nat-0):
    error: Plan apply failed: 1 error occurred:
    	* updating urn:pulumi:ds-REDACTED-replica-release::sg-REDACTED-REDACTED-replica::awsx:x:ec2:Vpc$awsx:x:ec2:Subnet$aws:ec2/route:Route::sourcegraph-REDACTED-REDACTED-replica-vpc-private-0-nat-0: Error: more than 1 target specified. Only 1 of gateway_id, egress_only_gateway_id, nat_gateway_id, instance_id, network_interface_id or vpc_peering_connection_id is allowed.

  aws:ec2:Route (sourcegraph-REDACTED-REDACTED-replica-vpc-private-1-nat-1):
    error: Plan apply failed: 1 error occurred:
    	* updating urn:pulumi:ds-REDACTED-replica-release::sg-REDACTED-REDACTED-replica::awsx:x:ec2:Vpc$awsx:x:ec2:Subnet$aws:ec2/route:Route::sourcegraph-REDACTED-REDACTED-replica-vpc-private-1-nat-1: Error: more than 1 target specified. Only 1 of gateway_id, egress_only_gateway_id, nat_gateway_id, instance_id, network_interface_id or vpc_peering_connection_id is allowed.

  aws:ec2:Route (sourcegraph-REDACTED-REDACTED-replica-vpc-private-2-nat-2):
    error: Plan apply failed: 1 error occurred:
    	* updating urn:pulumi:ds-REDACTED-replica-release::sg-REDACTED-REDACTED-replica::awsx:x:ec2:Vpc$awsx:x:ec2:Subnet$aws:ec2/route:Route::sourcegraph-REDACTED-REDACTED-replica-vpc-private-2-nat-2: Error: more than 1 target specified. Only 1 of gateway_id, egress_only_gateway_id, nat_gateway_id, instance_id, network_interface_id or vpc_peering_connection_id is allowed.

  pulumi:pulumi:Stack (sg-REDACTED-REDACTED-replica-ds-REDACTED-replica-release):
    (node:66606) ExperimentalWarning: queueMicrotask() is experimental.

    error: update failed

If I run pulumi up again (right after that last command), this is the pulumi preview output:

Do you want to perform this update? details
  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:ds-REDACTED-replica-release::sg-REDACTED-REDACTED-replica::pulumi:pulumi:Stack::sg-REDACTED-REDACTED-replica-ds-REDACTED-replica-release]
            ~ aws:ec2/route:Route: (update)
                [id=r-rtb-0981bff5ac5188cd91080289494]
                [urn=urn:pulumi:ds-REDACTED-replica-release::sg-REDACTED-REDACTED-replica::awsx:x:ec2:Vpc$awsx:x:ec2:Subnet$aws:ec2/route:Route::sourcegraph-REDACTED-REDACTED-replica-vpc-private-0-nat-0]
              - egressOnlyGatewayId   : ""
              + gatewayId             : "nat-07f8b134c241f07f0"
              - instanceId            : ""
              - natGatewayId          : "nat-07f8b134c241f07f0"
              - networkInterfaceId    : ""
              - transitGatewayId      : ""
              - vpcPeeringConnectionId: ""
            ~ aws:ec2/route:Route: (update)
                [id=r-rtb-0a449cdf74c14f7601080289494]
                [urn=urn:pulumi:ds-REDACTED-replica-release::sg-REDACTED-REDACTED-replica::awsx:x:ec2:Vpc$awsx:x:ec2:Subnet$aws:ec2/route:Route::sourcegraph-REDACTED-REDACTED-replica-vpc-private-2-nat-2]
              - egressOnlyGatewayId   : ""
              + gatewayId             : "nat-06e709f5d14313b1f"
              - instanceId            : ""
              - natGatewayId          : "nat-06e709f5d14313b1f"
              - networkInterfaceId    : ""
              - transitGatewayId      : ""
              - vpcPeeringConnectionId: ""
            ~ aws:ec2/route:Route: (update)
                [id=r-rtb-0ffca6caaa8613ba91080289494]
                [urn=urn:pulumi:ds-REDACTED-replica-release::sg-REDACTED-REDACTED-replica::awsx:x:ec2:Vpc$awsx:x:ec2:Subnet$aws:ec2/route:Route::sourcegraph-REDACTED-REDACTED-replica-vpc-private-1-nat-1]
              - egressOnlyGatewayId   : ""
              + gatewayId             : "nat-0b09091ded3d19792"
              - instanceId            : ""
              - natGatewayId          : "nat-0b09091ded3d19792"
              - networkInterfaceId    : ""
              - transitGatewayId      : ""
              - vpcPeeringConnectionId: ""
           

And that second pulumi update succeeds!

I have not made any changes to the AWS VPC configuration in between updates.

I'm not sure how to reproduce this, but I'm happy to hop on Slack and try to provide any more information.

[ggilmore]

Maybe hashicorp/terraform-provider-aws#684 is the underlying issue?

[lukehoban]

Reproducing this, I see the following diff during in the checkpoitn file before/after the refresh:

1146,1147c1354,1361
<                     "gatewayId": "nat-01070c24dbf3808ed",
<                     "routeTableId": "rtb-0e89d05739be1db54"
---
>                     "egressOnlyGatewayId": "",
>                     "gatewayId": "",
>                     "instanceId": "",
>                     "natGatewayId": "nat-01070c24dbf3808ed",
>                     "networkInterfaceId": "",
>                     "routeTableId": "rtb-0e89d05739be1db54",
>                     "transitGatewayId": "",
>                     "vpcPeeringConnectionId": ""

Notably, it looks like gatewayId is being replaced and placed in natGatewayId. This actually looks like it's a bug in the awsx package where we are using gatewayId instead of natGatewayId to provide this. I'm not even sure how that works in the first place, but I believe fixing it in awsx will address the core problem here.