Reorganize OPTE port management #1385

bnaecker · 2022-07-10T23:10:31Z

Simplify platform deps and module structure
Add new Port and PortManager types. The port manager is a centralized
object to manage all OPTE ports. This is currently required in order
to support correctly implementing the external IP workaround, which
requires keeping track of all current MAC addresses for guest
interfaces. This is all modeled after the instance / instance manager
relationship, where ports remove themselves from the manager on drop.
Add better logging
Add better handling of the overlay VNIC, currently also required for
OPTE to work with Viona.

bnaecker · 2022-07-10T23:11:44Z

This will resolve #1372, and prevent us hitting #1364 in the future. It's a bit more of a reorganization than I anticipated, but I think it helps in any case.

bnaecker · 2022-07-10T23:17:11Z

Also, lots of testing notes are forthcoming.

bnaecker · 2022-07-11T03:31:18Z

Here are some details about testing.

I started with a fresh reboot, created the virtual hardware, and installed the control plane. I populated the standard global images with tools/populate/populate-images.sh. Next I ran a little script to populate an IP Pool, create an org and project, create four disks from the Focal image, and then four instances with those disks attached. The whole script:

#!/bin/bash

set -x
set -e

# Create IP Pool with home network address range
oxide api /ip-pools --method POST --input - <<EOF
{"name": "p0", "description": "pool"}
EOF

oxide api /ip-pools/p0/ranges/add --method POST --input - <<EOF
{"first": "192.168.1.200", "last": "192.168.1.210"}
EOF

# Create project/org
oxide org create -D "my org" o
oxide project create -D "my project" -o o p

# Make a disk out of Focal image
for i in $(seq 0 3); do
    oxide api /organizations/o/projects/p/disks/ --method POST --input - <<EOF
    {
        "name": "focal$i",
        "description": "focal",
        "block_size": 512,
        "size": $((3*1024*1024*1024)),
        "disk_source": {
            "type": "global_image",
            "image_id": "$(oxide api /images/focal | jq -r .id)"
        }
    }
EOF
done

# Create and add SSH key
if ! [[ -f "demo.pub" ]]; then
    echo "Creating demo SSH key"
    ssh-keygen -t ecdsa -f demo -N ""
fi
oxide ssh-key add --file demo.pub --name "demo" --description "demo key"

for i in $(seq 0 3); do
        oxide api /organizations/o/projects/p/instances --method POST --input - <<EOF
        {
          "name": "i$i",
          "description": "i",
          "hostname": "myinst",
          "memory": 1073741824,
          "ncpus": 2,
          "disks": [
            {
              "type": "attach",
              "name": "focal$i"
            }
          ]
        }
EOF
done

And the output:

bnaecker@feldspar : ~/omicron $ ../cli/setup.sh
+ set -e
+ oxide api /ip-pools --method POST --input -
{
  "description": "pool",
  "id": "2b14afd3-ce03-4288-b366-d30e6bd86eb0",
  "name": "p0",
  "time_created": "2022-07-10T22:43:40.298547Z",
  "time_modified": "2022-07-10T22:43:40.298547Z"
}
+ oxide api /ip-pools/p0/ranges/add --method POST --input -
{
  "id": "72d5de2c-7118-4444-81e8-65b45fdfbb44",
  "range": {
    "first": "192.168.1.200",
    "last": "192.168.1.210"
  },
  "time_created": "2022-07-10T22:43:40.322665Z"
}
+ oxide org create -D 'my org' o
✔ Created organization o
+ oxide project create -D 'my project' -o o p
✔ Created project o/p
++ seq 0 3
+ for i in $(seq 0 3)
+ oxide api /organizations/o/projects/p/disks/ --method POST --input -
++ oxide api /images/focal
++ jq -r .id
{
  "block_size": 512,
  "description": "focal",
  "device_path": "/mnt/focal0",
  "id": "1186f337-a89d-4247-b9f8-5ae6ea71e65a",
  "image_id": "446b0a41-2122-4096-8c57-35816ea60fc7",
  "name": "focal0",
  "project_id": "78d68ad9-00c1-4e56-94a2-417a1fd31852",
  "size": 3221225472,
  "snapshot_id": null,
  "state": {
    "state": "creating"
  },
  "time_created": "2022-07-10T22:43:40.717960Z",
  "time_modified": "2022-07-10T22:43:40.717960Z"
}
+ for i in $(seq 0 3)
+ oxide api /organizations/o/projects/p/disks/ --method POST --input -
++ oxide api /images/focal
++ jq -r .id
{
  "block_size": 512,
  "description": "focal",
  "device_path": "/mnt/focal1",
  "id": "ca766a16-1bf6-4340-8fd6-17cac2fa8a7a",
  "image_id": "446b0a41-2122-4096-8c57-35816ea60fc7",
  "name": "focal1",
  "project_id": "78d68ad9-00c1-4e56-94a2-417a1fd31852",
  "size": 3221225472,
  "snapshot_id": null,
  "state": {
    "state": "creating"
  },
  "time_created": "2022-07-10T22:43:41.320107Z",
  "time_modified": "2022-07-10T22:43:41.320107Z"
}
+ for i in $(seq 0 3)
+ oxide api /organizations/o/projects/p/disks/ --method POST --input -
++ oxide api /images/focal
++ jq -r .id
{
  "block_size": 512,
  "description": "focal",
  "device_path": "/mnt/focal2",
  "id": "704475d9-f0f5-4480-8890-a6ea9a7c281f",
  "image_id": "446b0a41-2122-4096-8c57-35816ea60fc7",
  "name": "focal2",
  "project_id": "78d68ad9-00c1-4e56-94a2-417a1fd31852",
  "size": 3221225472,
  "snapshot_id": null,
  "state": {
    "state": "creating"
  },
  "time_created": "2022-07-10T22:43:41.787824Z",
  "time_modified": "2022-07-10T22:43:41.787824Z"
}
+ for i in $(seq 0 3)
+ oxide api /organizations/o/projects/p/disks/ --method POST --input -
++ oxide api /images/focal
++ jq -r .id
{
  "block_size": 512,
  "description": "focal",
  "device_path": "/mnt/focal3",
  "id": "a3d2c5de-eb32-4c22-b4b7-845f0f402351",
  "image_id": "446b0a41-2122-4096-8c57-35816ea60fc7",
  "name": "focal3",
  "project_id": "78d68ad9-00c1-4e56-94a2-417a1fd31852",
  "size": 3221225472,
  "snapshot_id": null,
  "state": {
    "state": "creating"
  },
  "time_created": "2022-07-10T22:43:42.314046Z",
  "time_modified": "2022-07-10T22:43:42.314046Z"
}
+ [[ -f demo.pub ]]
+ oxide ssh-key add --file demo.pub --name demo --description 'demo key'
Read SSH public key from demo.pub
✔ Added SSH public key demo: ecdsa-sha2-nistp256 SHA256:R4lafDXEN3m4N1xN/6oX/RFH9gfMk73U8oQsi3hANC4
++ seq 0 3
+ for i in $(seq 0 3)
+ oxide api /organizations/o/projects/p/instances --method POST --input -
{
  "description": "i",
  "hostname": "myinst",
  "id": "dc174481-6495-44d8-94b0-c5395d9e8665",
  "memory": 1073741824,
  "name": "i0",
  "ncpus": 2,
  "project_id": "78d68ad9-00c1-4e56-94a2-417a1fd31852",
  "run_state": "running",
  "time_created": "2022-07-10T22:43:42.799224Z",
  "time_modified": "2022-07-10T22:43:42.799224Z",
  "time_run_state_updated": "2022-07-10T22:43:54.961301Z"
}
+ for i in $(seq 0 3)
+ oxide api /organizations/o/projects/p/instances --method POST --input -
{
  "description": "i",
  "hostname": "myinst",
  "id": "6cdd1009-042b-4618-b4e9-74c81f83b4f1",
  "memory": 1073741824,
  "name": "i1",
  "ncpus": 2,
  "project_id": "78d68ad9-00c1-4e56-94a2-417a1fd31852",
  "run_state": "running",
  "time_created": "2022-07-10T22:43:55.102295Z",
  "time_modified": "2022-07-10T22:43:55.102295Z",
  "time_run_state_updated": "2022-07-10T22:44:09.116783Z"
}
+ for i in $(seq 0 3)
+ oxide api /organizations/o/projects/p/instances --method POST --input -
{
  "description": "i",
  "hostname": "myinst",
  "id": "0fe1d741-1a04-4330-a25e-1bd9ad6ee50d",
  "memory": 1073741824,
  "name": "i2",
  "ncpus": 2,
  "project_id": "78d68ad9-00c1-4e56-94a2-417a1fd31852",
  "run_state": "running",
  "time_created": "2022-07-10T22:44:09.225592Z",
  "time_modified": "2022-07-10T22:44:09.225592Z",
  "time_run_state_updated": "2022-07-10T22:44:20.844862Z"
}
+ for i in $(seq 0 3)
+ oxide api /organizations/o/projects/p/instances --method POST --input -
{
  "description": "i",
  "hostname": "myinst",
  "id": "921c8c01-61cb-4d60-a934-5743718df357",
  "memory": 1073741824,
  "name": "i3",
  "ncpus": 2,
  "project_id": "78d68ad9-00c1-4e56-94a2-417a1fd31852",
  "run_state": "running",
  "time_created": "2022-07-10T22:44:20.953090Z",
  "time_modified": "2022-07-10T22:44:20.953090Z",
  "time_run_state_updated": "2022-07-10T22:44:32.973610Z"
}
bnaecker@feldspar : ~/omicron $

At this point, we can verify that the OPTE ports are all as expected, and that the secondary-macs of the underlay VNIC net0 mirrors those MAC addresses of the guest:

bnaecker@feldspar : ~/omicron $ dladm | grep opte
opte0       xde       1500   up       --         --
vopte0      vnic      1500   up       --         opte0
opte1       xde       1500   up       --         --
vopte1      vnic      1500   up       --         opte1
opte2       xde       1500   up       --         --
vopte2      vnic      1500   up       --         opte2
opte3       xde       1500   up       --         --
vopte3      vnic      1500   up       --         opte3
bnaecker@feldspar : ~/omicron $ pfexec opteadm list-ports
LINK                             MAC ADDRESS              IPv4 ADDRESS     STATE
opte0                            A8:40:25:F4:11:D7        172.30.0.5       running
opte1                            A8:40:25:F2:A4:EA        172.30.0.6       running
opte2                            A8:40:25:F6:01:30        172.30.0.7       running
opte3                            A8:40:25:F1:89:78        172.30.0.8       running
bnaecker@feldspar : ~/omicron $
bnaecker@feldspar : ~/omicron $ dladm show-linkprop -p secondary-macs net0
LINK         PROPERTY        PERM VALUE          DEFAULT        POSSIBLE
net0         secondary-macs  rw   a8:40:25:f6:1:30, --          --
                                  a8:40:25:f2:a4:ea,
                                  a8:40:25:f1:89:78,
                                  a8:40:25:f4:11:d7

All three instances' networking stacks appear in order. We can ping them all, and SSH into each one. It's also important to note that we can ping out of each one as well.

bnaecker@feldspar : ~/omicron $ ping 192.168.1.200
192.168.1.200 is alive
bnaecker@feldspar : ~/omicron $ ping 192.168.1.201
192.168.1.201 is alive
bnaecker@feldspar : ~/omicron $ ping 192.168.1.202
192.168.1.202 is alive
bnaecker@feldspar : ~/omicron $ ssh -i demo [email protected]
The authenticity of host '192.168.1.200 (192.168.1.200)' can't be established.
ED25519 key fingerprint is SHA256:xJ5U2eM1kKFIhFUNcySxJcUn2Tth0l2JLFkg00h1VWM.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.1.200' (ED25519) to the list of known hosts.
Welcome to Ubuntu 20.04.3 LTS (GNU/Linux 5.4.0-99-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

 System information disabled due to load higher than 2.0

1 update can be applied immediately.
To see these additional updates run: apt list --upgradable


The list of available updates is more than a week old.
To check for new updates run: sudo apt update


The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@myinst:~$ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether a8:40:25:f4:11:d7 brd ff:ff:ff:ff:ff:ff
    inet 172.30.0.5/32 scope global dynamic enp0s8
       valid_lft 86159sec preferred_lft 86159sec
    inet6 fe80::aa40:25ff:fef4:11d7/64 scope link
       valid_lft forever preferred_lft forever
ubuntu@myinst:~$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=113 time=4.67 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=113 time=4.45 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=113 time=4.44 ms
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 4.436/4.518/4.670/0.107 ms
ubuntu@myinst:~$ logout
Connection to 192.168.1.200 closed.
bnaecker@feldspar : ~/omicron $ ssh -i demo [email protected]
The authenticity of host '192.168.1.201 (192.168.1.201)' can't be established.
ED25519 key fingerprint is SHA256:yo50JZT0a8pSvQ6CTPm4Q7OIh0SZWIyGQCZe+r1+1zc.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.1.201' (ED25519) to the list of known hosts.
Welcome to Ubuntu 20.04.3 LTS (GNU/Linux 5.4.0-99-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Sun Jul 10 22:55:15 UTC 2022

  System load:  0.62              Processes:               101
  Usage of /:   47.7% of 2.74GB   Users logged in:         0
  Memory usage: 19%               IPv4 address for enp0s8: 172.30.0.6
  Swap usage:   0%

1 update can be applied immediately.
To see these additional updates run: apt list --upgradable


The list of available updates is more than a week old.
To check for new updates run: sudo apt update


The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@myinst:~$ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether a8:40:25:f2:a4:ea brd ff:ff:ff:ff:ff:ff
    inet 172.30.0.6/32 scope global dynamic enp0s8
       valid_lft 86070sec preferred_lft 86070sec
    inet6 fe80::aa40:25ff:fef2:a4ea/64 scope link
       valid_lft forever preferred_lft forever
ubuntu@myinst:~$ ping -c 4 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=113 time=4.52 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=113 time=4.47 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=113 time=3.54 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=113 time=4.55 ms

--- 8.8.8.8 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 3.539/4.269/4.552/0.422 ms
ubuntu@myinst:~$ logout
Connection to 192.168.1.201 closed.
bnaecker@feldspar : ~/omicron $ ssh -i demo [email protected]
The authenticity of host '192.168.1.202 (192.168.1.202)' can't be established.
ED25519 key fingerprint is SHA256:WNmU53ReMZ7aamaYnT7ZaRSFLwtiB+znLBr5nz7Kk4s.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.1.202' (ED25519) to the list of known hosts.
Welcome to Ubuntu 20.04.3 LTS (GNU/Linux 5.4.0-99-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Sun Jul 10 22:56:48 UTC 2022

  System load:  0.45              Processes:               97
  Usage of /:   47.7% of 2.74GB   Users logged in:         0
  Memory usage: 19%               IPv4 address for enp0s8: 172.30.0.7
  Swap usage:   0%

1 update can be applied immediately.
To see these additional updates run: apt list --upgradable


The list of available updates is more than a week old.
To check for new updates run: sudo apt update


The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@myinst:~$ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether a8:40:25:f6:01:30 brd ff:ff:ff:ff:ff:ff
    inet 172.30.0.7/32 scope global dynamic enp0s8
       valid_lft 85992sec preferred_lft 85992sec
    inet6 fe80::aa40:25ff:fef6:130/64 scope link
       valid_lft forever preferred_lft forever
ubuntu@myinst:~$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=113 time=4.45 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=113 time=4.42 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=113 time=4.40 ms
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 4.400/4.423/4.453/0.022 ms
ubuntu@myinst:~$ logout
Connection to 192.168.1.202 closed.

Next, I verified that the OPTE ports, "overlay" VNICs (the guest side, sitting between OPTE and Viona), and the MAC addresses in the secondary-macs property are all cleaned up in order, as we stop (not delete) the instances.

bnaecker@feldspar : ~/omicron $ dladm | grep opte
opte0       xde       1500   up       --         --
vopte0      vnic      1500   up       --         opte0
opte1       xde       1500   up       --         --
vopte1      vnic      1500   up       --         opte1
opte2       xde       1500   up       --         --
vopte2      vnic      1500   up       --         opte2
opte3       xde       1500   up       --         --
vopte3      vnic      1500   up       --         opte3
bnaecker@feldspar : ~/omicron $ pfexec opteadm list-ports
LINK                             MAC ADDRESS              IPv4 ADDRESS     STATE
opte0                            A8:40:25:F4:11:D7        172.30.0.5       running
opte1                            A8:40:25:F2:A4:EA        172.30.0.6       running
opte2                            A8:40:25:F6:01:30        172.30.0.7       running
opte3                            A8:40:25:F1:89:78        172.30.0.8       running
bnaecker@feldspar : ~/omicron $ dladm show-linkprop -p secondary-macs net0
LINK         PROPERTY        PERM VALUE          DEFAULT        POSSIBLE
net0         secondary-macs  rw   a8:40:25:f6:1:30, --          --
                                  a8:40:25:f2:a4:ea,
                                  a8:40:25:f1:89:78,
                                  a8:40:25:f4:11:d7
bnaecker@feldspar : ~/omicron $ oxide instance list -o o -p p
                  id                  | name | description | hostname |   memory   | ncpus |              project_id              | run_state |  time_created  | time_modified  | time_run_state_updated
--------------------------------------+------+-------------+----------+------------+-------+--------------------------------------+-----------+----------------+----------------+------------------------
 dc174481-6495-44d8-94b0-c5395d9e8665 |  i0  |      i      |  myinst  | 1073741824 |   2   | 78d68ad9-00c1-4e56-94a2-417a1fd31852 |  running  | 14 minutes ago | 14 minutes ago |     14 minutes ago
 6cdd1009-042b-4618-b4e9-74c81f83b4f1 |  i1  |      i      |  myinst  | 1073741824 |   2   | 78d68ad9-00c1-4e56-94a2-417a1fd31852 |  running  | 14 minutes ago | 14 minutes ago |     14 minutes ago
 0fe1d741-1a04-4330-a25e-1bd9ad6ee50d |  i2  |      i      |  myinst  | 1073741824 |   2   | 78d68ad9-00c1-4e56-94a2-417a1fd31852 |  running  | 14 minutes ago | 14 minutes ago |     14 minutes ago
 921c8c01-61cb-4d60-a934-5743718df357 |  i3  |      i      |  myinst  | 1073741824 |   2   | 78d68ad9-00c1-4e56-94a2-417a1fd31852 |  running  | 14 minutes ago | 14 minutes ago |     14 minutes ago

bnaecker@feldspar : ~/omicron $ oxide instance stop -o o -p p i3
Type i3 to confirm stop:: i3
✔  Waiting for instance status to be `stopped`
✘ Stopped instance i3 in o/p
bnaecker@feldspar : ~/omicron $ dladm show-linkprop -p secondary-macs net0
LINK         PROPERTY        PERM VALUE          DEFAULT        POSSIBLE
net0         secondary-macs  rw   a8:40:25:f6:1:30, --          --
                                  a8:40:25:f2:a4:ea,
                                  a8:40:25:f4:11:d7
bnaecker@feldspar : ~/omicron $ dladm | grep opte
opte0       xde       1500   up       --         --
vopte0      vnic      1500   up       --         opte0
opte1       xde       1500   up       --         --
vopte1      vnic      1500   up       --         opte1
opte2       xde       1500   up       --         --
vopte2      vnic      1500   up       --         opte2
bnaecker@feldspar : ~/omicron $ pfexec opteadm list-ports
LINK                             MAC ADDRESS              IPv4 ADDRESS     STATE
opte0                            A8:40:25:F4:11:D7        172.30.0.5       running
opte1                            A8:40:25:F2:A4:EA        172.30.0.6       running
opte2                            A8:40:25:F6:01:30        172.30.0.7       running
bnaecker@feldspar : ~/omicron $ oxide instance stop -o o -p p i2
Type i2 to confirm stop:: i2
✔  Waiting for instance status to be `stopped`
✘ Stopped instance i2 in o/p
bnaecker@feldspar : ~/omicron $ dladm | grep opte
opte0       xde       1500   up       --         --
vopte0      vnic      1500   up       --         opte0
opte1       xde       1500   up       --         --
vopte1      vnic      1500   up       --         opte1
bnaecker@feldspar : ~/omicron $ dladm show-linkprop -p secondary-macs net0
LINK         PROPERTY        PERM VALUE          DEFAULT        POSSIBLE
net0         secondary-macs  rw   a8:40:25:f2:a4:ea, --         --
                                  a8:40:25:f4:11:d7
bnaecker@feldspar : ~/omicron $ pfexec opteadm list-ports
LINK                             MAC ADDRESS              IPv4 ADDRESS     STATE
opte0                            A8:40:25:F4:11:D7        172.30.0.5       running
opte1                            A8:40:25:F2:A4:EA        172.30.0.6       running
bnaecker@feldspar : ~/omicron $ oxide instance stop -o o -p p i1
Type i1 to confirm stop:: i1
✔  Waiting for instance status to be `stopped`
✘ Stopped instance i1 in o/p
bnaecker@feldspar : ~/omicron $ dladm | grep opte
opte0       xde       1500   up       --         --
vopte0      vnic      1500   up       --         opte0
bnaecker@feldspar : ~/omicron $ dladm show-linkprop -p secondary-macs net0
LINK         PROPERTY        PERM VALUE          DEFAULT        POSSIBLE
net0         secondary-macs  rw   a8:40:25:f4:11:d7 --          --
bnaecker@feldspar : ~/omicron $ pfexec opteadm list-ports
LINK                             MAC ADDRESS              IPv4 ADDRESS     STATE
opte0                            A8:40:25:F4:11:D7        172.30.0.5       running
bnaecker@feldspar : ~/omicron $ oxide instance stop -o o -p p i0
Type i0 to confirm stop:: i0
✔  Waiting for instance status to be `stopped`
✘ Stopped instance i0 in o/p
bnaecker@feldspar : ~/omicron $ dladm | grep opte
bnaecker@feldspar : ~/omicron $ dladm show-linkprop -p secondary-macs net0
LINK         PROPERTY        PERM VALUE          DEFAULT        POSSIBLE
net0         secondary-macs  rw   --             --             --
bnaecker@feldspar : ~/omicron $ pfexec opteadm list-ports
LINK                             MAC ADDRESS              IPv4 ADDRESS     STATE
bnaecker@feldspar : ~/omicron $

- Simplify platform deps and module structure - Add new Port and PortManager types. The port manager is a centralized object to manage all OPTE ports. This is currently required in order to support correctly implementing the external IP workaround, which requires keeping track of all current MAC addresses for guest interfaces. This is all modeled after the instance / instance manager relationship, where ports remove themselves from the manager on drop. - Add better logging - Add better handling of the overlay VNIC, currently also required for OPTE to work with Viona.

sled-agent/src/illumos/running_zone.rs

sled-agent/src/opte/port_manager.rs

sled-agent/src/instance.rs

sled-agent/src/opte/mod.rs

sled-agent/src/opte/port.rs

- Adds some better documentation around the semantics of dropping a port ticket - Adds links to issues in some todos - Splits illumos/other platform implementations at module level.

- Store ports keyed on the combination of instance ID and port name, allowing each port to be stored as a separate kv-pair - Collect a ticket for each port in the running zone, and release each port explicitly and separately

bnaecker requested a review from smklein July 10, 2022 23:10

bnaecker force-pushed the append-secondary-macs branch from c46f934 to a9636e6 Compare July 11, 2022 03:31

bnaecker force-pushed the append-secondary-macs branch from a9636e6 to 637dbde Compare July 11, 2022 04:10

smklein self-assigned this Jul 11, 2022

smklein approved these changes Jul 11, 2022

View reviewed changes

smklein removed their assignment Jul 11, 2022

bnaecker and others added 5 commits July 12, 2022 03:32

Review feedback

15668b1

- Adds some better documentation around the semantics of dropping a port ticket - Adds links to issues in some todos - Splits illumos/other platform implementations at module level.

Fixup non-illumos OPTE module

4edec92

Merge branch 'main' into append-secondary-macs

1b4626a

Store OPTE ports individually in the manager

a98be98

- Store ports keyed on the combination of instance ID and port name, allowing each port to be stored as a separate kv-pair - Collect a ticket for each port in the running zone, and release each port explicitly and separately

fixup non-illumos

189efc0

bnaecker enabled auto-merge (squash) July 12, 2022 15:55

bnaecker added 2 commits July 12, 2022 09:00

more non-illumos fixup

61c5fad

more non-illumos patching

8a4f641

bnaecker merged commit 08befc0 into main Jul 12, 2022

bnaecker deleted the append-secondary-macs branch July 12, 2022 17:42

bnaecker mentioned this pull request Jul 12, 2022

Need to append new instances to VNIC's secondary-macs link property #1372

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reorganize OPTE port management #1385

Reorganize OPTE port management #1385

bnaecker commented Jul 10, 2022

bnaecker commented Jul 10, 2022

bnaecker commented Jul 10, 2022

bnaecker commented Jul 11, 2022

Reorganize OPTE port management #1385

Reorganize OPTE port management #1385

Conversation

bnaecker commented Jul 10, 2022

bnaecker commented Jul 10, 2022

bnaecker commented Jul 10, 2022

bnaecker commented Jul 11, 2022