Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds basic per-sled sequential IP address allocation #891

Merged
merged 2 commits into from
Apr 8, 2022

Conversation

bnaecker
Copy link
Collaborator

@bnaecker bnaecker commented Apr 7, 2022

  • Adds the last_used_address column to the omicron.sled table, which
    tracks the last IP address within the sled's prefix allocated to a
    service running on the sled
  • Adds method for selecting the next IP address from the sled table,
    with a few basic tests for it
  • Uses a static address when launching guest instances, providing it to
    the propolis server managing them.

- Adds the `last_used_address` column to the `omicron.sled` table, which
  tracks the last IP address within the sled's prefix allocated to a
  service running on the sled
- Adds method for selecting the next IP address from the `sled` table,
  with a few basic tests for it
- Uses a static address when launching guest instances, providing it to
  the propolis server managing them.
@bnaecker bnaecker requested review from jmpesp and smklein April 7, 2022 23:00
@bnaecker
Copy link
Collaborator Author

bnaecker commented Apr 7, 2022

In addition to the small unit test I added, I've tested this manually launching a real VM managed by Propolis. Here's the CLI invocation to create a disk and launch an instance with it:

bnaecker@feldspar : ~/cli $ ./target/debug/oxide disk create mydisk -D "my disk" -o o -p p --size $((1024 * 1024 * 1024))
bnaecker@feldspar : ~/cli $ ./target/debug/oxide disk list -o o -p p
                  id                  |  name  | description | device_path |              project_id              |    size    | snapshot_id |   state   | time_created  | time_modified
--------------------------------------+--------+-------------+-------------+--------------------------------------+------------+-------------+-----------+---------------+---------------
 9e3baa49-4e2e-4bef-af31-81ca6b7c6cfe | mydisk |   my disk   | /mnt/mydisk | 5e08adf7-2cc4-412d-862d-9daaf8269059 | 1073741824 |             | detached= | 2 minutes ago | 2 minutes ago
bnaecker@feldspar : ~/cli $ ./target/debug/oxide api /organizations/o/projects/p/instances --method POST --input - <<EOF
> {"name": "vm0", "description": "a vm", "hostname": "vm0", "memory": 1073741824, "ncpus": 2, "disks": [{"type": "attach", "disk": "mydisk"}]}
> EOF
{
  "description": "a vm",
  "hostname": "vm0",
  "id": "7ed287e3-d8cb-4dd1-ad1a-ffac2336150a",
  "memory": 1073741824,
  "name": "vm0",
  "ncpus": 2,
  "project_id": "5e08adf7-2cc4-412d-862d-9daaf8269059",
  "run_state": "running",
  "time_created": "2022-04-07T20:34:38.570301Z",
  "time_modified": "2022-04-07T20:34:38.570301Z",
  "time_run_state_updated": "2022-04-07T20:34:47.878695Z"
}

Here's a snippet of the sled agent log, showing that a static IPv6 address is provided to the launched Propolis server:

Apr 07 20:34:44.213 INFO Adding address: Static(V6(Ipv6Network { addr: fd00:1de::66, prefix: 64 })), zone: oxz_propolis-server_14fbb4fb-723c-4573-9fea-9e9b678e47de, instance id: 7ed287e3-d8cb-4dd1-ad1a-ffac2336150a, instance: 7ed287e3-d8cb-4dd1-ad1a-ffac2336150a, server: fb0f7546-4d46-40ca-9d56-cbb810684ca7, component: SledAgent
Apr 07 20:34:44.759 INFO Created address fd00:1de::66/64 for zone: oxz_propolis-server_14fbb4fb-723c-4573-9fea-9e9b678e47de, instance id: 7ed287e3-d8cb-4dd1-ad1a-ffac2336150a, instance: 7ed287e3-d8cb-4dd1-ad1a-ffac2336150a, server: fb0f7546-4d46-40ca-9d56-cbb810684ca7, component: SledAgent
Apr 07 20:34:45.210 INFO Started propolis in zone: oxz_propolis-server_14fbb4fb-723c-4573-9fea-9e9b678e47de, instance id: 7ed287e3-d8cb-4dd1-ad1a-ffac2336150a, instance: 7ed287e3-d8cb-4dd1-ad1a-ffac2336150a, server: fb0f7546-4d46-40ca-9d56-cbb810684ca7, component: SledAgent
Apr 07 20:34:45.431 INFO GET request to http://[fd00:1de::66]:12400/instances/00000000-0000-0000-0000-000000000000, propolis_client address: [fd00:1de::66]:12400, component: propolis-client, instance id: 7ed287e3-d8cb-4dd1-ad1a-ffac2336150a, instance: 7ed287e3-d8cb-4dd1-ad1a-ffac2336150a, server: fb0f7546-4d46-40ca-9d56-cbb810684ca7, component: SledAgent
Apr 07 20:34:45.432 INFO Sending ensure request to propolis: InstanceEnsureRequest { properties: InstanceProperties { id: 7ed287e3-d8cb-4dd1-ad1a-ffac2336150a, name: "vm0", description: "Test description", image_id: 00000000-0000-0000-0000-000000000000, bootrom_id: 00000000-0000-0000-0000-000000000000, memory: 1024, vcpus: 2 }, nics: [NetworkInterfaceRequest { name: "oxGuestInstance0", slot: Slot(0) }], disks: [DiskRequest { name: "mydisk", slot: Slot(0), read_only: false, device: "nvme", gen: 3, volume_construction_request: Volume { block_size: 4096, sub_volumes: [Region { block_size: 4096, opts: CrucibleOpts { target: [[fd00:1de::9]:19000, [fd00:1de::11]:19000, [fd00:1de::10]:19000], lossy: false, key: Some("EJpbE1KTCRzLrtEUN9z52vH12p9oOdXsIiTO9A1Xifk="), cert_pem: None, key_pem: None, root_cert_pem: None, control: None }, gen: 0 }], read_only_parent: None } }], migrate: None, cloud_init_bytes: None }, instance id: 7ed287e3-d8cb-4dd1-ad1a-ffac2336150a, instance: 7ed287e3-d8cb-4dd1-ad1a-ffac2336150a, server: fb0f7546-4d46-40ca-9d56-cbb810684ca7, component: SledAgent

Here's the log from one of the Crucible downstairs, showing the negotiation, and including the assigned address of the Propolis server:

root@oxz_crucible_oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b:~# tail -f $(svcs -L downstairs)
UUID: 3927a194-aec1-4622-b123-0374dc92c9c5
Blocks per extent:256 Total Extents: 1024
Using address: [fd00:1de::9]:19000
No SSL acceptor configured
listening on [fd00:1de::9]:19000
accepted connection from [fd00:1de::66]:48465
upstairs 44e39647-5685-4c74-a2f8-540a5dd11455 connected
44e39647-5685-4c74-a2f8-540a5dd11455 is now active
Current flush_numbers [0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Downstairs has completed Negotiation

And lastly, here's the disk actually showing up in the guest, with the right size:

localhost:~# blockdev --getsize64 /dev/nvme0n1
1073741824

That's after getting the guest console with propolis-cli serial, using the provided address that shows up in the sled-agent logs.

In addition to y'all's eyes, I would love some more test-driving. I spoke to @jmpesp about this. I'm not very familiar with Crucible, so some more sanity checks that the behavior here looks correct would be much appreciated.

Copy link
Contributor

@jmpesp jmpesp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Will test this shortly

nexus/src/db/model.rs Show resolved Hide resolved
Copy link
Collaborator

@smklein smklein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great, thanks for putting it together.

The subnets are becoming real!

common/src/sql/dbinit.sql Show resolved Hide resolved
nexus/src/db/datastore.rs Show resolved Hide resolved
nexus/src/db/model.rs Show resolved Hide resolved
nexus/src/db/model.rs Show resolved Hide resolved
nexus/src/sagas.rs Outdated Show resolved Hide resolved
nexus/src/sagas.rs Show resolved Hide resolved
- Adds some comments and issue links
- Make allocation of IP addresses a separate saga action, to ensure
  idempotency. Also adds a generic helper, since this will likely be a
  common saga node.
@bnaecker
Copy link
Collaborator Author

bnaecker commented Apr 8, 2022

@smklein @jmpesp Made some updates, mostly around Sean's idempotency comments. Please take another pass when you get a chance!

Copy link
Collaborator

@smklein smklein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

@jmpesp
Copy link
Contributor

jmpesp commented Apr 8, 2022

Looks good! Will test this shortly

Works perfectly! :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants