diff --git a/README.adoc b/README.adoc index c691666f07..123d016493 100644 --- a/README.adoc +++ b/README.adoc @@ -33,400 +33,26 @@ You can **format the code** using `cargo fmt`. Make sure to run this before pus You can **run the https://github.com/rust-lang/rust-clippy[Clippy linter]** using `cargo clippy \-- -D warnings -A clippy::style`. Make sure to run this before pushing changes. The CI checks that the code is clippy-clean. -=== Installing Prerequisite Software +=== Running Omicron -The following software may be installed automatically with the following script: +Omicron has two modes of operation: "simulated" and "non-simulated". ----- -$ ./tools/install_prerequisites.sh ----- - -=== Running (Simulated) Omicron - -NOTE: If you'd like to run on Helios, refer to <> below. - -To **run Omicron** you need to run four programs: - -* a CockroachDB cluster. For development, you can use the `omicron-dev` tool in this repository to start a single-node CockroachDB cluster **that will delete the database when you shut it down.** -* a ClickHouse server. You should use the `omicron-dev` tool for this as well, see below, and as with CockroachDB, -the database files will be deleted when you stop the program. -* `nexus`: the guts of the control plane -* `sled-agent-sim`: a simulator for the component that manages a single sled - -The easiest way to start the required databases is to use the built-in `omicron-dev` tool. This tool assumes that the `cockroach` and `clickhouse` executables are on your PATH, and match the versions above. - -. Start CockroachDB using `omicron-dev db-run`: -+ -[source,text] ----- -$ cargo run --bin=omicron-dev -- db-run - Finished dev [unoptimized + debuginfo] target(s) in 0.15s - Running `target/debug/omicron-dev db-run` -omicron-dev: using temporary directory for database store (cleaned up on clean exit) -omicron-dev: will run this to start CockroachDB: -cockroach start-single-node --insecure --http-addr=:0 --store /var/tmp/omicron_tmp/.tmpM8KpTf/data --listen-addr 127.0.0.1:32221 --listening-url-file /var/tmp/omicron_tmp/.tmpM8KpTf/listen-url -omicron-dev: temporary directory: /var/tmp/omicron_tmp/.tmpM8KpTf -* -* WARNING: ALL SECURITY CONTROLS HAVE BEEN DISABLED! -* -* This mode is intended for non-production testing only. -* -* In this mode: -* - Your cluster is open to any client that can access 127.0.0.1. -* - Intruders with access to your machine or network can observe client-server traffic. -* - Intruders can log in without password and read or write any data in the cluster. -* - Intruders can consume all your server's resources and cause unavailability. -* -* -* INFO: To start a secure server without mandating TLS for clients, -* consider --accept-sql-without-tls instead. For other options, see: -* -* - https://go.crdb.dev/issue-v/53404/v20.2 -* - https://www.cockroachlabs.com/docs/v20.2/secure-a-cluster.html -* - -omicron-dev: child process: pid 3815 -omicron-dev: CockroachDB listening at: postgresql://root@127.0.0.1:32221/omicron?sslmode=disable -omicron-dev: populating database -* -* INFO: Replication was disabled for this cluster. -* When/if adding nodes in the future, update zone configurations to increase the replication factor. -* -CockroachDB node starting at 2021-04-13 15:58:59.680359279 +0000 UTC (took 0.4s) -build: OSS v20.2.5 @ 2021/03/17 21:00:51 (go1.16.2) -webui: http://127.0.0.1:41618 -sql: postgresql://root@127.0.0.1:32221?sslmode=disable -RPC client flags: cockroach --host=127.0.0.1:32221 --insecure -logs: /var/tmp/omicron_tmp/.tmpM8KpTf/data/logs -temp dir: /var/tmp/omicron_tmp/.tmpM8KpTf/data/cockroach-temp022560209 -external I/O path: /var/tmp/omicron_tmp/.tmpM8KpTf/data/extern -store[0]: path=/var/tmp/omicron_tmp/.tmpM8KpTf/data -storage engine: pebble -status: initialized new cluster -clusterID: 8ab646f0-67f0-484d-8010-e4444fb86336 -nodeID: 1 -omicron-dev: populated database ----- -+ -Note that as the output indicates, this cluster will be available to anybody that can reach 127.0.0.1. - -. Start the ClickHouse database server: -+ -[source,text] ----- -$ cargo run --bin omicron-dev -- ch-run - Finished dev [unoptimized + debuginfo] target(s) in 0.47s - Running `target/debug/omicron-dev ch-run` -omicron-dev: running ClickHouse (PID: 2463), full command is "clickhouse server --log-file /var/folders/67/2tlym22x1r3d2kwbh84j298w0000gn/T/.tmpJ5nhot/clickhouse-server.log --errorlog-file /var/folders/67/2tlym22x1r3d2kwbh84j298w0000gn/T/.tmpJ5nhot/clickhouse-server.errlog -- --http_port 8123 --path /var/folders/67/2tlym22x1r3d2kwbh84j298w0000gn/T/.tmpJ5nhot" -omicron-dev: using /var/folders/67/2tlym22x1r3d2kwbh84j298w0000gn/T/.tmpJ5nhot for ClickHouse data storage ----- - -. `nexus` requires a configuration file to run. You can use `nexus/examples/config.toml` to start with. Build and run it like this: -+ -[source,text] ----- -$ cargo run --bin=nexus -- nexus/examples/config.toml -... -listening: http://127.0.0.1:12220 ----- -Nexus can also serve the web console. Instructions for generating the static assets and pointing Nexus to them are https://github.com/oxidecomputer/console/blob/main/docs/serve-from-nexus.md[here]. Without console assets, Nexus will still start and run normally as an API. A few link:./nexus/src/external_api/console_api.rs[console-specific routes] will 404. +The simulated version of Omicron allows the high-level control plane logic to run without +actually managing any sled-local resources. This version can be executed on Linux, Mac, and illumos. +This mode of operation is provided for development and testing only. -. `sled-agent-sim` only accepts configuration on the command line. Run it with a uuid identifying itself (this would be a uuid for the sled it's managing), an IP:port for itself, and the IP:port of `nexus`'s _internal_ interface. Using default config, this might look like this: -+ -[source,text] ----- -$ cargo run --bin=sled-agent-sim -- $(uuidgen) 127.0.0.1:12345 127.0.0.1:12221 -... -Jun 02 12:21:50.989 INFO listening, local_addr: 127.0.0.1:12345, component: dropshot ----- +See: xref:docs/how-to-run-simulated.adoc[]. -. `oximeter` is similar to `nexus`, requiring a configuration file. You can use `oximeter/collector/config.toml`, and the whole thing can be run with: -+ -[source,text] ----- -$ cargo run --bin=oximeter -- oximeter/collector/config.toml -Dec 02 18:00:01.062 INFO starting oximeter server -Dec 02 18:00:01.062 DEBG creating ClickHouse client -Dec 02 18:00:01.068 DEBG initializing ClickHouse database, component: clickhouse-client, collector_id: 1da65e5b-210c-4859-a7d7-200c1e659972, component: oximeter-agent -Dec 02 18:00:01.093 DEBG registered endpoint, path: /producers, method: POST, local_addr: [::1]:12223, component: dropshot -... ----- +The non-simulated version of Omicron actually manages sled-local resources, and may only +be executed on hosts running Helios. +This mode of operation will be used in production. -Once everything is up and running, you can use `curl` directly to hit either of -the servers. But it's easier to use the [`oxide` CLI](https://docs.oxide.computer/cli/manual) (see below). +See: xref:docs/how-to-run.adoc[]. == Docker image This repo includes a Dockerfile that builds an image containing the Nexus and sled agent. There's a GitHub Actions workflow that builds and publishes the Docker image. This is used by the https://github.com/oxidecomputer/console/[console] project for prototyping, demoing, and testing. This is **not** the way Omicron will be deployed on production systems, but it's a useful vehicle for working with it. -== Quick demo - -The `oxide` CLI can be installed from the [latest -release](https://github.com/oxidecomputer/cli/releases). - -Run `oxide auth login` to authenticate with your Oxide account. Alternatively, -`oxide` will respect the `OXIDE_TOKEN` and `OXIDE_HOST` environment variables. - -If you find bugs or have feedback, leave it on the [cli -repo](https://github.com/oxidecomputer/cli/issues). - -Here's a small demo that creates a project, creates an instance, and attaches a disk to it: - -[source,text] ----- -$ oxide org create myorg \ - --description "My organization" -✔ Created organization myorg - -$ oxide project create myproject \ - --description "My project" \ - --organization myorg -✔ Created project myorg/myproject - -$ oxide instance create myinstance \ - --description "My instance" \ - --project myproject \ - --org myorg \ - --hostname "myinstance.maze-war.com" \ - --ncpus 1 \ - --memory 8 -✔ Created instance myinstance in myorg/myproject - -$ oxide instance view myinstance \ - --org myorg \ - --project myproject \ - --format json -{ - "id": "99ad2514-050c-4493-9cb9-d9ceba980a98", - "name": "myinstance", - "description": "My instance", - "timeCreated": "2021-11-17T01:45:07.606749Z", - "timeModified": "2021-11-17T01:45:07.606749Z", - "projectId": "c197b9d2-285c-4e9f-9461-1815ef093c8d", - "ncpus": 1, - "memory": 8, - "hostname": "myinstance.maze-war.com", - "runState": "running", - "timeRunStateUpdated": "2021-11-17T01:45:09.120652Z" -} - -$ oxide disk create nginx \ - -D "The nginx disk." \ - -o myorg \ - -p myproject \ - --size 10 -✔ Created disk nginx in myorg/myproject - - -$ oxide disk view nginx \ - --org myorg \ - --project myproject \ - --format json -{ - "id": "551bbe67-3640-41c9-b968-249a136e5e31", - "name": "nginx", - "description": "The nginx disk.", - "timeCreated": "2021-11-17T01:47:36.524136Z", - "timeModified": "2021-11-17T01:47:36.524136Z", - "projectId": "c197b9d2-285c-4e9f-9461-1815ef093c8d", - "snapshotId": null, - "size": 1024, - "state": { - "state": "detached" - }, - "devicePath": "/mnt/nginx" -} - -$ oxide disk attach nginx myinstance \ - -o maze-war \ - -p prod-online -✔ Attached disk nginx to instance myinstance in myorg/myproject - -$ oxide instance disks myinstance \ - -o maze-war \ - -p prod-online \ - --format json -{ - "instanceId": "99ad2514-050c-4493-9cb9-d9ceba980a98", - "diskId": "551bbe67-3640-41c9-b968-249a136e5e31", - "diskName": "nginx", - "diskState": { - "state": "attached", - "instance": "99ad2514-050c-4493-9cb9-d9ceba980a98" - } -} - -# Alternatively, you can use the API command to run any endpoint. -# This operates like a fancy, authenticated curl. - -$ oxide api --help -Makes an authenticated HTTP request to the Oxide API and prints the response. - -The endpoint argument should be a path of a Oxide API endpoint. - -The default HTTP request method is "GET" normally and "POST" if any parameters -were added. Override the method with `--method`. - -Pass one or more `-f/--raw-field` values in "key=value" format to add static string -parameters to the request payload. To add non-string or otherwise dynamic values, see -`--field` below. Note that adding request parameters will automatically switch the -request method to POST. To send the parameters as a GET query string instead, use -`--method GET`. - -The `-F/--field` flag has magic type conversion based on the format of the value: - -- literal values "true", "false", "null", and integer/float numbers get converted to - appropriate JSON types; -- if the value starts with "@", the rest of the value is interpreted as a - filename to read the value from. Pass "-" to read from standard input. - -Raw request body may be passed from the outside via a file specified by `--input`. -Pass "-" to read from standard input. In this mode, parameters specified via -`--field` flags are serialized into URL query parameters. - -In `--paginate` mode, all pages of results will sequentially be requested until -there are no more pages of results. - -USAGE: - oxide api [OPTIONS] - -ARGS: - - The endpoint to request - -OPTIONS: - -d, --debug - Print debug info - - [env: DEBUG=] - - -f, --raw-field - Add a string parameter in key=value format - - -F, --field - Add a typed parameter in key=value format - - -h, --help - Print help information - - -H, --header
- Add a HTTP request header in `key:value` format - - -i, --include - Include HTTP response headers in the output - - --input - The file to use as body for the HTTP request (use "-" to read from standard input) - - [default: ] - - --paginate - Make additional HTTP requests to fetch all pages of results - - -X, --method - The HTTP method for the request - -$ oxide api /session/me -{ - "id": "99ad2514-050c-4493-9cb9-d9ceba980a98" -} - - ----- - -== Deploying Omicron - - -Prerequisite: Have a machine already running Helios. An easy way to -do this is by using a https://github.com/oxidecomputer/helios-engvm[Helios VM]. -ISOs are also available for download https://pkg.oxide.computer/install[here]. - -The control plane repository contains a packaging tool which bundles binaries -and SMF manifests. After building the expected binaries, they can be packaged -in a format which lets them be transferred to a Helios machine. - -This tool acts on a `package-manifest.toml` file which describes the packages to be -bundled in the build. - -Configuration files are used to select IP addresses, and to manage Zpools -utilized by the Sled Agent. These configuration files are located within -`smf/`, and likely need to be modified to use addresses and zpools which match -your hardware. Much of this configuration will be automated in the future -(e.g., IP addresses will be inferred and posted to a DNS system, Zpools will -automatically be detected on discovered disks), but for now it remains -hard-coded. - -[source,text] ----- -$ cargo build -$ ./target/debug/omicron-package package ----- - -The aforementioned package command fills a target directory of choice -(by default, `out/` within the omicron repository) with tarballs ready -to be unpacked as services. - -To install the services on a target machine, the following command -may be executed: - -[source,text] ----- -# Note that "sudo" is required to install SMF services; an appropriate pfexec -# profile may also be used. -$ sudo ./target/debug/omicron-package install ----- - -This service installs a bootstrap service, which itself loads other -requested services. The bootstrap service is currently the only -service which is "persistent" across reboots - although it will -initialize other service as part of its setup sequence anyway. - -[source,text] ----- -# List all services: -$ svcs -# View zones managed by Omicron (prefixed with "oxz_"): -$ zoneadm list -cv -# View logs for a service: -$ pfexec tail -f $(pfexec svcs -z oxz_nexus -L nexus) ----- - -To uninstall all Omicron services from a machine, the following may be -executed: - -[source,text] ----- -$ sudo ./target/debug/omicron-package uninstall ----- - -=== Test Environment - -When we deploy, we're effectively creating a number of different zones -for all the components that make up Omicron (Nexus, Clickhouse, Crucible, etc). -Since all these services run in different zones they cannot communicate with -each other (and Sled Agent in the global zone) via `localhost`. In practice, -we'll assign addresses as per RFD 63 as well as incorporating DNS based -service discovery. - -For the purposes of local development today, we specify some hardcoded IPv6 -unique local addresses in `fd00:1de::/16`: - -[options="header"] -|=================================================================================================== -| Service | Endpoint -| Sled Agent: Bootstrap | `[::]:12346` -| Sled Agent: Dropshot API | `[fd00:1de::1]:12345` -| Cockroach DB | `[fd00:1de::5]:32221` -| Oximeter | `[fd00:1de::6]:12223` -| Nexus: External API | `[fd00:1de::7]:12220` -| Nexus: Internal API | `[fd00:1de::7]:12221` -| Clickhouse | `[fd00:1de::8]:8123` -| Crucible Downstairs | `[fd00:1de::9]:32345`, `[fd00:1de::10]:32345`, `[fd00:1de::11]:32345` -|=================================================================================================== - -Note that Sled Agent runs in the global zone and is the one responsible for bringing up all the other -other services and allocating them with vNICs and IPv6 addresses. - == Configuration reference `nexus` requires a TOML configuration file. There's an example in diff --git a/docs/cli.adoc b/docs/cli.adoc new file mode 100644 index 0000000000..c1cdfe2a0d --- /dev/null +++ b/docs/cli.adoc @@ -0,0 +1,184 @@ +:showtitle: +:toc: left +:icons: font + += Oxide CLI + +The `oxide` CLI is used to access Nexus' external API, which is +the public interface to Omicron. + +For more detail, refer to https://docs.oxide.computer/cli/manual[oxide's CLI Manual]. + +== Quick demo + +The `oxide` CLI can be installed from the https://github.com/oxidecomputer/cli/releases[latest release]. + +Run `oxide auth login` to authenticate with your Oxide account. Alternatively, +`oxide` will respect the `OXIDE_TOKEN` and `OXIDE_HOST` environment variables. + +If you find bugs or have feedback, leave it on the https://github.com/oxidecomputer/cli/issues[cli repo]. + +Here's a small demo that creates a project, creates an instance, and attaches a disk to it: + +[source,text] +---- +$ oxide org create myorg \ + --description "My organization" +✔ Created organization myorg + +$ oxide project create myproject \ + --description "My project" \ + --organization myorg +✔ Created project myorg/myproject + +$ oxide instance create myinstance \ + --description "My instance" \ + --project myproject \ + --organization myorg \ + --hostname "myinstance.maze-war.com" \ + --ncpus 1 \ + --memory 8 +✔ Created instance myinstance in myorg/myproject + +$ oxide instance view myinstance \ + --organization myorg \ + --project myproject \ + --format json +{ + "id": "99ad2514-050c-4493-9cb9-d9ceba980a98", + "name": "myinstance", + "description": "My instance", + "timeCreated": "2021-11-17T01:45:07.606749Z", + "timeModified": "2021-11-17T01:45:07.606749Z", + "projectId": "c197b9d2-285c-4e9f-9461-1815ef093c8d", + "ncpus": 1, + "memory": 8, + "hostname": "myinstance.maze-war.com", + "runState": "running", + "timeRunStateUpdated": "2021-11-17T01:45:09.120652Z" + + +$ oxide disk create nginx \ + -D "The nginx disk." \ + -o myorg \ + -p myproject \ + --size 10 +✔ Created disk nginx in myorg/myproject + + +$ oxide disk view nginx \ + --organization myorg \ + --project myproject \ + --format json +{ + "id": "551bbe67-3640-41c9-b968-249a136e5e31", + "name": "nginx", + "description": "The nginx disk.", + "timeCreated": "2021-11-17T01:47:36.524136Z", + "timeModified": "2021-11-17T01:47:36.524136Z", + "projectId": "c197b9d2-285c-4e9f-9461-1815ef093c8d", + "snapshotId": null, + "size": 1024, + "state": { + "state": "detached" + }, + "devicePath": "/mnt/nginx" +} + +$ oxide disk attach nginx myinstance \ + -o maze-war \ + -p prod-online +✔ Attached disk nginx to instance myinstance in myorg/myproject + +$ oxide instance disks myinstance \ + -o maze-war \ + -p prod-online \ + --format json +{ + "instanceId": "99ad2514-050c-4493-9cb9-d9ceba980a98", + "diskId": "551bbe67-3640-41c9-b968-249a136e5e31", + "diskName": "nginx", + "diskState": { + "state": "attached", + "instance": "99ad2514-050c-4493-9cb9-d9ceba980a98" + } +} +---- + +Alternatively, you can use the API command to run any endpoint. +This operates like a fancy, authenticated curl. + +[source,text] +---- +$ oxide api --help +Makes an authenticated HTTP request to the Oxide API and prints the response. + +The endpoint argument should be a path of a Oxide API endpoint. + +The default HTTP request method is "GET" normally and "POST" if any parameters +were added. Override the method with `--method`. + +Pass one or more `-f/--raw-field` values in "key=value" format to add static string +parameters to the request payload. To add non-string or otherwise dynamic values, see +`--field` below. Note that adding request parameters will automatically switch the +request method to POST. To send the parameters as a GET query string instead, use +`--method GET`. + +The `-F/--field` flag has magic type conversion based on the format of the value: + +- literal values "true", "false", "null", and integer/float numbers get converted to + appropriate JSON types; +- if the value starts with "@", the rest of the value is interpreted as a + filename to read the value from. Pass "-" to read from standard input. + +Raw request body may be passed from the outside via a file specified by `--input`. +Pass "-" to read from standard input. In this mode, parameters specified via +`--field` flags are serialized into URL query parameters. + +In `--paginate` mode, all pages of results will sequentially be requested until +there are no more pages of results. + +USAGE: + oxide api [OPTIONS] + +ARGS: + + The endpoint to request + +OPTIONS: + -d, --debug + Print debug info + + [env: DEBUG=] + + -f, --raw-field + Add a string parameter in key=value format + + -F, --field + Add a typed parameter in key=value format + + -h, --help + Print help information + + -H, --header
+ Add a HTTP request header in `key:value` format + + -i, --include + Include HTTP response headers in the output + + --input + The file to use as body for the HTTP request (use "-" to read from standard input) + + [default: ] + + --paginate + Make additional HTTP requests to fetch all pages of results + + -X, --method + The HTTP method for the request + +$ oxide api /session/me +{ + "id": "99ad2514-050c-4493-9cb9-d9ceba980a98" +} +---- diff --git a/docs/how-to-run-simulated.adoc b/docs/how-to-run-simulated.adoc new file mode 100644 index 0000000000..039cc0d662 --- /dev/null +++ b/docs/how-to-run-simulated.adoc @@ -0,0 +1,143 @@ +:showtitle: +:toc: left +:icons: font + += Running Omicron (Simulated) + +== What is "Simulated Omicron"? + +The "Sled-local" component of the control plane - which expects to manage local +resources - has tight coupling with the illumos Operating System. However, a +good portion of the control plane (interactions with the database, metrics +collection, and the console, for example) executes within programs that are +decoupled from the underlying Sled. + +To enable more flexible testing of this software, a "simulated" sled agent is +provided, capable of running across many platforms (Linux, Mac, illumos). This +allows developers to test the control plane flows without actually having any +resources to manage. + +If you are interested in running the "real" control plane (which is necessary +for managing instances, storage, and networking) refer to the non-simulated +guide at xref:how-to-run.adoc[]. + +== Installing Prerequisites + +Prerequisite software may be installed with the following script: + +[source,text] +---- +$ ./tools/install_prerequisites.sh +---- + +== Running + +To **run Omicron** you need to run four programs: + +* a CockroachDB cluster. For development, you can use the `omicron-dev` tool in this repository to start a single-node CockroachDB cluster **that will delete the database when you shut it down.** +* a ClickHouse server. You should use the `omicron-dev` tool for this as well, see below, and as with CockroachDB, +the database files will be deleted when you stop the program. +* `nexus`: the guts of the control plane +* `sled-agent-sim`: a simulator for the component that manages a single sled + +The easiest way to start the required databases is to use the built-in `omicron-dev` tool. This tool assumes that the `cockroach` and `clickhouse` executables are on your PATH, and match the versions above. + +. Start CockroachDB using `omicron-dev db-run`: ++ +[source,text] +---- +$ cargo run --bin=omicron-dev -- db-run + Finished dev [unoptimized + debuginfo] target(s) in 0.15s + Running `target/debug/omicron-dev db-run` +omicron-dev: using temporary directory for database store (cleaned up on clean exit) +omicron-dev: will run this to start CockroachDB: +cockroach start-single-node --insecure --http-addr=:0 --store /var/tmp/omicron_tmp/.tmpM8KpTf/data --listen-addr 127.0.0.1:32221 --listening-url-file /var/tmp/omicron_tmp/.tmpM8KpTf/listen-url +omicron-dev: temporary directory: /var/tmp/omicron_tmp/.tmpM8KpTf +* +* WARNING: ALL SECURITY CONTROLS HAVE BEEN DISABLED! +* +* This mode is intended for non-production testing only. +* +* In this mode: +* - Your cluster is open to any client that can access 127.0.0.1. +* - Intruders with access to your machine or network can observe client-server traffic. +* - Intruders can log in without password and read or write any data in the cluster. +* - Intruders can consume all your server's resources and cause unavailability. +* +* +* INFO: To start a secure server without mandating TLS for clients, +* consider --accept-sql-without-tls instead. For other options, see: +* +* - https://go.crdb.dev/issue-v/53404/v20.2 +* - https://www.cockroachlabs.com/docs/v20.2/secure-a-cluster.html +* + +omicron-dev: child process: pid 3815 +omicron-dev: CockroachDB listening at: postgresql://root@127.0.0.1:32221/omicron?sslmode=disable +omicron-dev: populating database +* +* INFO: Replication was disabled for this cluster. +* When/if adding nodes in the future, update zone configurations to increase the replication factor. +* +CockroachDB node starting at 2021-04-13 15:58:59.680359279 +0000 UTC (took 0.4s) +build: OSS v20.2.5 @ 2021/03/17 21:00:51 (go1.16.2) +webui: http://127.0.0.1:41618 +sql: postgresql://root@127.0.0.1:32221?sslmode=disable +RPC client flags: cockroach --host=127.0.0.1:32221 --insecure +logs: /var/tmp/omicron_tmp/.tmpM8KpTf/data/logs +temp dir: /var/tmp/omicron_tmp/.tmpM8KpTf/data/cockroach-temp022560209 +external I/O path: /var/tmp/omicron_tmp/.tmpM8KpTf/data/extern +store[0]: path=/var/tmp/omicron_tmp/.tmpM8KpTf/data +storage engine: pebble +status: initialized new cluster +clusterID: 8ab646f0-67f0-484d-8010-e4444fb86336 +nodeID: 1 +omicron-dev: populated database +---- ++ +Note that as the output indicates, this cluster will be available to anybody that can reach 127.0.0.1. + +. Start the ClickHouse database server: ++ +[source,text] +---- +$ cargo run --bin omicron-dev -- ch-run + Finished dev [unoptimized + debuginfo] target(s) in 0.47s + Running `target/debug/omicron-dev ch-run` +omicron-dev: running ClickHouse (PID: 2463), full command is "clickhouse server --log-file /var/folders/67/2tlym22x1r3d2kwbh84j298w0000gn/T/.tmpJ5nhot/clickhouse-server.log --errorlog-file /var/folders/67/2tlym22x1r3d2kwbh84j298w0000gn/T/.tmpJ5nhot/clickhouse-server.errlog -- --http_port 8123 --path /var/folders/67/2tlym22x1r3d2kwbh84j298w0000gn/T/.tmpJ5nhot" +omicron-dev: using /var/folders/67/2tlym22x1r3d2kwbh84j298w0000gn/T/.tmpJ5nhot for ClickHouse data storage +---- + +. `nexus` requires a configuration file to run. You can use `nexus/examples/config.toml` to start with. Build and run it like this: ++ +[source,text] +---- +$ cargo run --bin=nexus -- nexus/examples/config.toml +... +listening: http://127.0.0.1:12220 +---- +Nexus can also serve the web console. Instructions for generating the static assets and pointing Nexus to them are https://github.com/oxidecomputer/console/blob/main/docs/serve-from-nexus.md[here]. Without console assets, Nexus will still start and run normally as an API. A few link:./nexus/src/external_api/console_api.rs[console-specific routes] will 404. + +. `sled-agent-sim` only accepts configuration on the command line. Run it with a uuid identifying itself (this would be a uuid for the sled it's managing), an IP:port for itself, and the IP:port of `nexus`'s _internal_ interface. Using default config, this might look like this: ++ +[source,text] +---- +$ cargo run --bin=sled-agent-sim -- $(uuidgen) 127.0.0.1:12345 127.0.0.1:12221 +... +Jun 02 12:21:50.989 INFO listening, local_addr: 127.0.0.1:12345, component: dropshot +---- + +. `oximeter` is similar to `nexus`, requiring a configuration file. You can use `oximeter/collector/config.toml`, and the whole thing can be run with: ++ +[source,text] +---- +$ cargo run --bin=oximeter -- oximeter/collector/config.toml +Dec 02 18:00:01.062 INFO starting oximeter server +Dec 02 18:00:01.062 DEBG creating ClickHouse client +Dec 02 18:00:01.068 DEBG initializing ClickHouse database, component: clickhouse-client, collector_id: 1da65e5b-210c-4859-a7d7-200c1e659972, component: oximeter-agent +Dec 02 18:00:01.093 DEBG registered endpoint, path: /producers, method: POST, local_addr: [::1]:12223, component: dropshot +... +---- + +Once everything is up and running, you can use `curl` directly to hit either of +the servers. But it's easier to use the xref:cli.adoc[`oxide` CLI]. diff --git a/docs/how-to-run.adoc b/docs/how-to-run.adoc new file mode 100644 index 0000000000..11505406a8 --- /dev/null +++ b/docs/how-to-run.adoc @@ -0,0 +1,114 @@ +:showtitle: +:toc: left +:icons: font + += Running Omicron (Non-Simulated) + +Omicron is the control plane for an Oxide rack. It expects to execute +on Helios systems, and Sleds use Helios-specific interfaces to manage +resources. + +If you're interested in running the control plane on other platforms, including +Linux and Mac, refer to the guide on xref:how-to-run-simulated.adoc[running +simulated Omicron]. + +== Installing Prerequisite Software + +A major prerequisite is to have a machine already running Helios. An easy way to +do this is by using a https://github.com/oxidecomputer/helios-engvm[Helios VM]. +ISOs are also available for download https://pkg.oxide.computer/install[here]. + +Any additional prerequisite software may be installed with the following script: + +[source,text] +---- +$ ./tools/install_prerequisites.sh +---- + +== Deploying Omicron + +The control plane repository contains a packaging tool which bundles binaries +and SMF manifests. After building the expected binaries, they can be packaged +in a format which lets them be transferred to a Helios machine. + +This tool acts on a `package-manifest.toml` file which describes the packages to be +bundled in the build. + +Configuration files are used to select IP addresses, and to manage Zpools +utilized by the Sled Agent. These configuration files are located within +`smf/`, and likely need to be modified to use addresses and zpools which match +your hardware. Much of this configuration will be automated in the future +(e.g., IP addresses will be inferred and posted to a DNS system, Zpools will +automatically be detected on discovered disks), but for now it remains +hard-coded. + +[source,text] +---- +$ cargo run --release --bin omicron-package -- package +---- + +NOTE: Running in `release` mode isn't strictly required, but improves +the performance of the packaging tools significantly. + +The aforementioned package command fills a target directory of choice +(by default, `out/` within the omicron repository) with tarballs ready +to be unpacked as services. + +To install the services on a target machine, the following command +may be executed: + +[source,text] +---- +$ pfexec cargo run --release --bin omicron-package -- install +---- + +This service installs a bootstrap service, which itself loads other +requested services. The bootstrap service is currently the only +service which is "persistent" across reboots - although it will +initialize other service as part of its setup sequence anyway. + +[source,text] +---- +# List all services: +$ svcs +# View zones managed by Omicron (prefixed with "oxz_"): +$ zoneadm list -cv +# View logs for a service: +$ pfexec tail -f $(pfexec svcs -z oxz_nexus -L nexus) +---- + +To uninstall all Omicron services from a machine, the following may be +executed: + +[source,text] +---- +$ pfexec cargo run --release --bin omicron-package -- uninstall +---- + +=== Test Environment + +When we deploy, we're effectively creating a number of different zones +for all the components that make up Omicron (Nexus, Clickhouse, Crucible, etc). +Since all these services run in different zones they cannot communicate with +each other (and Sled Agent in the global zone) via `localhost`. In practice, +we'll assign addresses as per RFD 63 as well as incorporating DNS based +service discovery. + +For the purposes of local development today, we specify some hardcoded IPv6 +unique local addresses in `fd00:1de::/16`: + +[options="header"] +|=================================================================================================== +| Service | Endpoint +| Sled Agent: Bootstrap | `[::]:12346` +| Sled Agent: Dropshot API | `[fd00:1de::]:12345` +| Cockroach DB | `[fd00:1de::5]:32221` +| Oximeter | `[fd00:1de::6]:12223` +| Nexus: External API | `[fd00:1de::7]:12220` +| Nexus: Internal API | `[fd00:1de::7]:12221` +| Clickhouse | `[fd00:1de::8]:8123` +| Crucible Downstairs | `[fd00:1de::9]:32345`, `[fd00:1de::10]:32345`, `[fd00:1de::11]:32345` +|=================================================================================================== + +Note that Sled Agent runs in the global zone and is the one responsible for bringing up all the other +other services and allocating them with vNICs and IPv6 addresses.