obsservice: initial commit

These are the humble beginnings of an Observability Service for CRDB. In
time, this is meant to take over a lot of the observability
functionality, extracting it from the database itself. For now, all
there is here is an HTTP reverse proxy, able to route HTTP requests to a
CRDB node (or cluster, through a load balancer). At least for the
transitioning period, if not forever, the Obs Service will route HTTP
paths that it doesn't understand to the Cockroach cluster being
monitored.

The HTTP proxy accepts its own certificates and can serve HTTPS
independently of CRDB (which might be running in --insecure mode and not
serving HTTPS). However, if CRDB is serving HTTPS, then the Obs Service
must also be configured with a certificate so it can also serve HTTPS.

The code of the Obs Service is split in between a binary (minimal
shell), and a library. The idea is to keep the door open for other repos
to extend the Obs Service with extra functionality (e.g. the CC managed
service might want to add cloud-specific observability features). For
such purposes, I've considered making a dedicated Go module for the Obs
Service so that source-level imports of the Obs Service would be as
ergonomic as Go wants them to be. I've put a decent amount of work in
playing with this but, ultimately, I've decided against it, at least for
now. The problem is that the Obs Service wants to take minor code
dependencies on random CRDB libraries (syncutil, log, etc.) and the
cockroach Go module is fairly broken (our git tags look like we do
semantic versioning, but we don't actually do it). The go tooling has a
terrible time with the cockroach module. Also, our code is generally not
`go build`-able. If the obs service was a dedicated module, it would
need to take a dependency on the cockroach module, which would negate
the win for people who want to import it (in fact, it'd make matters
worse for importers because the nasty cockroach dependency would be more
hidden). An alternative I've considered was to start creating modules
for all of the dependencies of the Obs Service. But, if CRDB would then
need to depend on these modules, there's all sorts of problems.

Release note: None

.github/CODEOWNERS

@@ -291,6 +291,7 @@
 /pkg/util/stop               @cockroachdb/server-prs
 /pkg/util/tracing            @cockroachdb/obs-inf-prs
 /pkg/workload/               @cockroachdb/sql-experience-noreview
+/pkg/obsservice/             @cockroachdb/obs-inf-prs
 
 # Allow the security team to have insight into changes to
 # authn/authz code.

.gitignore

@@ -25,6 +25,8 @@ work-Fuzz*
 *-fuzz.zip
 # vendoring by `go mod vendor` may produce this file temporarily
 /.vendor.tmp.*
+# The Observability Service binary.
+/obsservice
 
 # Custom or private env vars (e.g. internal keys, access tokens, etc).
 customenv.mk

pkg/obsservice/README.md

@@ -0,0 +1,68 @@
+# CockroachDB Observability Service
+
+This directory contains the source code of the CRDB Observability Service - a
+service that collects monitoring and observability functionality from CRDB and
+serves a web console that exposes the data.
+
+The Obs Service is developed as a library (in the `obslib` package) and a binary
+(the `cmd\obsservice` package). The idea is for other binaries to be able to
+embed and extend the library (for example we imagine CockroachCloud doing so in
+the future).
+
+## Building the Obs Service
+
+Build with
+
+```shell
+go build ./pkg/obsservice/cmd/obsservice
+```
+
+or
+
+```shell
+./dev build pkg/obsservice/cmd/obsservice
+```
+
+## Running
+
+```shell
+obsservice --http-addr=localhost:8081 --crdb-http-url=http://localhost:8080 --ui-cert=certs/cert.pem --ui-cert-key=certs/key.pem --ca-cert=certs/ca.crt
+```
+
+- `--crdb-http-url` is CRDB's HTTP address. For a multi-node CRDB cluster, this
+  can point to a load-balancer. It can be either a HTTP or an HTTPS address,
+  depending on whether the CRDB cluster is running as `--insecure`.
+- `--ui-cert` and `--ui-cert-key` are the paths to the certificate
+  presented by the Obs Service to its HTTPS clients, and the private key
+  corresponding to the certificate. If no certificates are configured, the Obs
+  Service will not use TLS. Certificates need to be specified if the CRDB
+  cluster is not running in `--insecure` mode: i.e. the Obs Service will refuse
+  to forward HTTP requests over HTTPS. The reverse is allowed, though: the Obs
+  Service can be configured with certificates even if CRDB is running in
+  `--insecure`. In this case, the Obs Service will terminate TLS connections and
+  forward HTTPS requests over HTTP.
+
+  If configured with certificates, HTTP requests will be redirected to HTTPS.  
+
+  For testing, self-signed certificates can be generated, for example, with the
+  [`generate_cert.go`](https://go.dev/src/crypto/tls/generate_cert.go) utility in
+  the Golang standard library: `go run ./crypto/tls/generate_cert.go
+  --host=localhost`.
+- `--ca-cert` is the path to a certificate authority certificate file (perhaps
+  one created with `cockroach cert create-ca`). If specified, HTTP requests are
+  only proxied to CRDB nodes that present certificates signed by this CA. If not
+  specified, the system's CA list is used.
+
+## Functionality
+
+_This is the very beginning of the Obs Service; it doesn't have any
+functionality yet. Hopefully we'll keep this up to date as it evolves._
+
+The Obs Service will reverse-proxy all HTTP routes it doesn't handle itself to
+CRDB. Currently the Obs Service doesn't handle any routes (other than
+`/debug/pprof/*`, which exposes the Obs Service's own pprof endpoints), so all
+requests are forwarded.
+
+## Licensing
+
+The Observability Service is licensed as Apache 2.0.

pkg/obsservice/cmd/obsservice/BUILD.bazel

@@ -0,0 +1,19 @@
+load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library")
+
+go_library(
+    name = "obsservice_lib",
+    srcs = ["main.go"],
+    importpath = "github.com/cockroachdb/cockroach/pkg/obsservice/cmd/obsservice",
+    visibility = ["//visibility:private"],
+    deps = [
+        "//pkg/cli/exit",
+        "//pkg/obsservice/obslib",
+        "@com_github_spf13_cobra//:cobra",
+    ],
+)
+
+go_binary(
+    name = "obsservice",
+    embed = [":obsservice_lib"],
+    visibility = ["//visibility:public"],
+)

pkg/obsservice/cmd/obsservice/main.go

@@ -0,0 +1,83 @@
+// Copyright 2022 The Cockroach Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+
+package main
+
+import (
+	"context"
+	"fmt"
+
+	"github.com/cockroachdb/cockroach/pkg/cli/exit"
+	"github.com/cockroachdb/cockroach/pkg/obsservice/obslib"
+	"github.com/spf13/cobra"
+)
+
+// RootCmd represents the base command when called without any subcommands
+var RootCmd = &cobra.Command{
+	Use:   "obsservice",
+	Short: "An observability service for CockroachDB",
+	Long: `The Observability Service ingests monitoring and observability data 
+from one or more CockroachDB clusters.`,
+	Run: func(cmd *cobra.Command, args []string) {
+		ctx := context.Background()
+		cfg := obslib.ReverseHTTPProxyConfig{
+			HTTPAddr:      httpAddr,
+			TargetURL:     targetURL,
+			CACertPath:    caCertPath,
+			UICertPath:    uiCertPath,
+			UICertKeyPath: uiCertKeyPath,
+		}
+
+		// Block forever running the proxy.
+		<-obslib.NewReverseHTTPProxy(ctx, cfg).RunAsync(ctx)
+	},
+}
+
+// Flags.
+var (
+	httpAddr                  string
+	targetURL                 string
+	caCertPath                string
+	uiCertPath, uiCertKeyPath string
+)
+
+func main() {
+	RootCmd.PersistentFlags().StringVar(
+		&httpAddr,
+		"http-addr",
+		"localhost:8081",
+		"The address on which to listen for HTTP requests.")
+	RootCmd.PersistentFlags().StringVar(
+		&targetURL,
+		"crdb-http-url",
+		"http://localhost:8080",
+		"The base URL to which HTTP requests are proxied.")
+	RootCmd.PersistentFlags().StringVar(
+		&caCertPath,
+		"ca-cert",
+		"",
+		"Path to the certificate authority certificate file. If specified,"+
+			" HTTP requests are only proxied to CRDB nodes that present certificates signed by this CA."+
+			" If not specified, the system's CA list is used.")
+	RootCmd.PersistentFlags().StringVar(
+		&uiCertPath,
+		"ui-cert",
+		"",
+		"Path to the certificate used used by the Observability Service.")
+	RootCmd.PersistentFlags().StringVar(
+		&uiCertKeyPath,
+		"ui-cert-key",
+		"",
+		"Path to the private key used by the Observability Service. "+
+			"This is the key corresponding to the --ui-cert certificate.")
+
+	if err := RootCmd.Execute(); err != nil {
+		fmt.Println(err)
+		exit.WithCode(exit.UnspecifiedError())
+	}
+}

pkg/obsservice/obslib/BUILD.bazel

@@ -0,0 +1,14 @@
+load("@io_bazel_rules_go//go:def.bzl", "go_library")
+
+go_library(
+    name = "obslib",
+    srcs = ["lib.go"],
+    importpath = "github.com/cockroachdb/cockroach/pkg/obsservice/obslib",
+    visibility = ["//visibility:public"],
+    deps = [
+        "//pkg/util/log",
+        "//pkg/util/syncutil",
+        "@com_github_cockroachdb_cmux//:cmux",
+        "@com_github_cockroachdb_errors//:errors",
+    ],
+)