Integrate softnpu from ry-working-softer-npu branch

.gitignore

@@ -10,3 +10,5 @@ tools/cockroach*
 /cockroachdb/
 smf/nexus/root.json
 core
+*.vdev
+debug.out

docs/boundary-services-a-to-z.adoc

@@ -0,0 +1,213 @@
+= Boundary Services A-Z
+
+This document describes how to run a RAP environment with boundary services.
+It's a quick rundown and assumes knowledge of the basic setup described in the
+Running Omicron (Non-Simulated) document.
+
+== 0. Install softnpu ASIC emulator machinery
+
+----
+./tools/install_softnpu_machinery.sh
+----
+
+== 1. Setup virtual hardware
+
+----
+pfexec ./tools/create_virtual_hardware.sh <wan interface>
+----
+
+The virtual hardware is a bit different than what's currently being used. What
+we'll eventually wind up with looks like this.
+
+image::plumbing.png[]
+
+== 2. Build and install the control plane.
+
+----
+./tools/create_self_signed_cert.sh
+cargo run --release --bin omicron-package -- package
+pfexec cargo run --release --bin omicron-package -- install
+----
+
+The control plane is now starting, reference the Running Omicron (Non-Simulated)
+doc for more details on determining when things are ready to go.
+
+
+== 3. Launch and configure the softnpu zone
+
+Launch the zone.
+
+----
+pfexec ./tools/scrimlet/create-softnpu-zone.sh
+----
+
+Configure the softnpu zone. The following will drop you into a zone shell.
+
+----
+pfexec zlogin softnpu
+----
+
+Now run softnpu.
+
+----
+root@scrimlet:~# cd /stuff/
+root@scrimlet:/stuff# ./softnpu softnpu.toml
+Config {
+    p4_program: "/stuff/libsidecar_lite.so",
+    ports: [
+        Port {
+            sidecar: "sc0_0",
+            scrimlet: "sr0_0",
+            mtu: 1600,
+        },
+        Port {
+            sidecar: "sc0_1",
+            scrimlet: "sr0_1",
+            mtu: 1500,
+        },
+    ],
+}
+----
+
+Back in the global zone, softnpu can be configured.
+
+----
+ry@korgano: cd /opt/softnpu/stuff
+ry@korgano: pfexec ./softnpu-init.sh
+[00:00:01] ######################################## 14.31 MiB/14.31 MiB done
+local v6:
+fe80::aae1:deff:fe01:701c
+fe80::aae1:deff:fe01:701d
+fd00:99::1
+local v4:
+router v6:
+fd00:1122:3344:101::/64 -> fe80::aae1:deff:fe00:1 (1)
+router v4:
+0.0.0.0/0 -> 10.100.0.1 (2)
+resolver v4:
+10.100.0.1 -> 90:ec:77:2e:70:27
+resolver v6:
+fe80::aae1:deff:fe00:1 -> a8:e1:de:00:00:01
+nat_v4:
+10.100.0.6 1024/65535 -> fd00:1122:3344:101:: 8717766/a8:40:25:f0:51:75
+nat_v6:
+port_mac:
+1: a8:e1:de:01:70:1c
+2: a8:e1:de:01:70:1d
+icmp_v6:
+icmp_v4:
+----
+
+== 4. Populating the system
+
+Follow the
+https://github.com/oxidecomputer/meta/blob/master/engineering/remote-access-preview-demo-setup.adoc#setting-up-the-cli[RAP document]
+to set up IPs, images, disks, instances etc. Things to pay particular attention
+to here are the following.
+
+- The address range in the IP pool should be on a subnet in your local network that
+  can NAT out to the Internet.
+- Be sure to set up an external IP for the instance you create.
+
+== 5. Configuring scrimlet/sidecar
+
+A this point we have an instance up and running. At the time of writing there is
+not control plane driven boundary services automation so we're going to
+configure the scrimlet it by hand.
+
+First we need to collect some information. In particular we need to know about
+the virtual network our instance is sitting on. We can get that info from
+`opteadm`.
+
+....
+pfexec /opt/oxide/opte/bin/opteadm dump-v2p
+
+Virtual to Physical Mappings
+======================================================================
+
+VPC 15103089
+----------------------------------------------------------------------
+
+IPv4 mappings
+----------------------------------------------------------------------
+VPC IP                   VPC MAC ADDR      UNDERLAY IP
+172.30.0.5               A8:40:25:F2:84:3F fd00:1122:3344:101::1
+
+IPv6 mappings
+----------------------------------------------------------------------
+VPC IP                   VPC MAC ADDR      UNDERLAY IP
+....
+
+The key pieces of information we need here are the following. Please note these
+will vary from instance to instance.
+
+1. The VNI = `15103089`
+2. The MAC = `A8:40:25:F2:84:3F`
+3. The underlay = `fd00:1122:3344:101::1`
+
+We also need to know what external IP has been assigned to our instance. The
+only way I know how to do this right now is by looking at the web console. Go to
+the instances page and navigate to the network tab. You'll see an External IP
+field in the table. Let's assume that is `10.100.0.6` for this example.
+
+Now we need to go tell boundary services about this information.
+
+Log back into the scrimlet VM
+
+----
+./out/propolis/propolis-cli --server 127.0.0.1 serial
+----
+
+Go back to the `/opt/cargo-bay` and open up `softnpu-init.sh` in an editor.
+There are a few things we need to edit here. Locate the line with the following
+content.
+
+----
+./softnpuadm add-nat4 10.100.0.6 1024 65535 fd00:1122:3344:0101:: 8717766 a8:40:25:f0:51:75
+----
+
+Edit this line to use the information we gathered above. For the specific
+information I have for this run, this look like:
+
+----
+./softnpuadm add-nat4 10.100.0.6 1024 65535 fd00:1122:3344:101::1 15103089 A8:40:25:F2:84:3F
+----
+
+While editing this file, also note the comments guiding you to change the
+upstream gateway IP and MAC addresses. The MAC address is the same one you would
+use for the OPTE hack. The IP address honestly does not matter a whole lot for
+this setup since it's a default route. Just make sure the address used for the
+gateway IP is the same in both places.
+
+Now run
+
+----
+./softnpu-init.sh
+----
+
+This will reconfigure the ASIC (you could also run just 
+`./softnpuadm remove-<x> ...` and `./softnpuadm add-<x>` if you feel like
+being more surgical) with a boundary services config that will give your
+instance access to the Internet.
+
+----
+ry@korgano:~/omicron$ ~/propolis/target/release/propolis-cli --server fd00:1122:3344:101::c serial
+
+debian login: root
+Linux debian 5.10.0-9-amd64 #1 SMP Debian 5.10.70-1 (2021-09-30) x86_64
+
+The programs included with the Debian GNU/Linux system are free software;
+the exact distribution terms for each program are described in the
+individual files in /usr/share/doc/*/copyright.
+
+Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
+permitted by applicable law.
+root@debian:~# host oxide.computer
+oxide.computer has address 76.76.21.61
+oxide.computer has address 76.76.21.22
+oxide.computer mail is handled by 5 alt2.aspmx.l.google.com.
+oxide.computer mail is handled by 1 aspmx.l.google.com.
+oxide.computer mail is handled by 10 aspmx3.googlemail.com.
+oxide.computer mail is handled by 5 alt1.aspmx.l.google.com.
+oxide.computer mail is handled by 10 aspmx2.googlemail.com.
+----

docs/how-to-run.adoc

@@ -22,7 +22,8 @@ Any additional prerequisite software may be installed with the following script:
 
 [source,text]
 ----
-$ ./tools/install_prerequisites.sh
+$ source tools/setup_path.sh
+$ pfexec ./tools/install_prerequisites.sh
 ----
 
 This script expects that you are both attempting to compile code and execute
@@ -42,17 +43,27 @@ $ ./tools/install_runner_prerequisites.sh
 The sled agent expects to manage a real Gimlet. However, until those are built,
 developers generally make do with something else, usually a commodity machine.
 To make your machine "look" like a Gimlet, the
-`./tools/create_virtual_hardware.sh` script can be used. This creates a few
+`pfexec ./tools/create_virtual_hardware.sh` script can be used. This creates a few
 file-based ZFS vdevs and ZFS zpools on top of those, and a couple of VNICs. The
 vdevs model the actual U.2s that will be in a Gimlet, and the VNICs model the
 two Chelsio NIC ports.
 
-You can clean up these resources with `./tools/destroy_virtual_hardware.sh`.
+You can clean up these resources with `pfexec ./tools/destroy_virtual_hardware.sh`.
 This script requires Omicron be uninstalled, e.g., with `pfexec
 ./target/release/omicron-package uninstall`, and a warning will be printed if
 that is not the case. The script will then remove the file-based vdevs and the
 VNICs created by `create_virtual_hardware.sh`.
 
+=== Set up a scrimlet/sidecar device.
+
+To access the outside world or other gimlets, we'll need to set up a sidecar
+device for those network paths.
+
+----
+./tools/install_scrimlet_machinery.sh
+./tools/launch_scrimlet.sh
+----
+
 === Make me a certificate!
 
 Nexus's external interface will typically be served using public-facing x.509

package/src/bin/omicron-package.rs

@@ -578,7 +578,7 @@ async fn do_clean(
         artifact_dir.to_string_lossy()
     );
     const ARTIFACTS_TO_KEEP: &[&str] =
-        &["clickhouse", "cockroachdb", "xde", "console-assets", "downloads"];
+        &["clickhouse", "cockroachdb", "xde", "console-assets", "downloads", "softnpu"];
     remove_all_except(artifact_dir, ARTIFACTS_TO_KEEP, &config.log)?;
     info!(
         config.log,

sled-agent/src/opte/illumos/mod.rs

@@ -8,7 +8,6 @@ use crate::common::underlay;
 use crate::illumos::dladm;
 use opte_ioctl::OpteHdl;
 use slog::Logger;
-use std::fs;
 use std::path::Path;
 
 mod firewall_rules;
@@ -79,17 +78,6 @@ pub fn initialize_xde_driver(log: &Logger) -> Result<(), Error> {
         return Err(Error::NoXdeConf);
     }
 
-    // TODO-remove
-    //
-    // See https://github.com/oxidecomputer/omicron/issues/1337
-    //
-    // An additional part of the workaround to connect into instances. This is
-    // required to tell OPTE to actually act as a 1-1 NAT when an instance is
-    // provided with an external IP address, rather than do its normal job of
-    // encapsulating the traffic onto the underlay (such as for delivery to
-    // boundary services).
-    use_external_ip_workaround(&log, &xde_conf);
-
     let underlay_nics = underlay::find_nics()?;
     info!(log, "using '{:?}' as data links for xde driver", underlay_nics);
     if underlay_nics.len() < 2 {
@@ -132,36 +120,3 @@ pub fn initialize_xde_driver(log: &Logger) -> Result<(), Error> {
         Err(e) => Err(e.into()),
     }
 }
-
-fn use_external_ip_workaround(log: &Logger, xde_conf: &Path) {
-    const NEEDLE: &str = "ext_ip_hack = 0;";
-    const NEW_NEEDLE: &str = "ext_ip_hack = 1;";
-
-    // NOTE: This only works in the real sled agent, which is run as root. The
-    // file is not world-readable.
-    let contents = fs::read_to_string(xde_conf)
-        .expect("Failed to read xde configuration file");
-    let new = contents.replace(NEEDLE, NEW_NEEDLE);
-    if contents == new {
-        info!(
-            log,
-            "xde driver configuration file appears to already use external IP workaround";
-            "conf_file" => ?xde_conf,
-        );
-    } else {
-        info!(
-            log,
-            "updating xde driver configuration file for external IP workaround";
-            "conf_file" => ?xde_conf,
-        );
-        fs::write(xde_conf, &new)
-            .expect("Failed to modify xde configuration file");
-    }
-
-    // Ensure the driver picks up the updated configuration file, if it's been
-    // loaded previously without the workaround.
-    std::process::Command::new(crate::illumos::PFEXEC)
-        .args(&["update_drv", "xde"])
-        .output()
-        .expect("Failed to reload xde driver configuration file");
-}