Add developer docs to README

README.md

@@ -220,3 +220,331 @@ If a page number is out of bounds, an empty list is returned.
 `page` and `pagesize` can be used alongside the `limit` and `continue`
 parameters supported by Kubernetes. `limit` and `continue` are typically used
 for server-side chunking and do not guarantee results in any order.
+
+Running the Steve server
+------------------------
+
+Steve is typically imported as a library. The calling code starts the server:
+
+```
+import (
+    "fmt"
+    "context"
+
+    "github.com/rancher/steve/pkg/server"
+    "github.com/rancher/wrangler/pkg/kubeconfig"
+)
+
+func steve() error {
+    restConfig, err := kubeconfig.GetNonInteractiveClientConfigWithContext("", "").ClientConfig()
+    if err != nil {
+        return err
+    }
+    ctx := context.Background()
+    s, err := server.New(ctx, restConfig, nil)
+    if err != nil {
+        return err
+    }
+    fmt.Println(s.ListenAndServe(ctx, 9443, 9080, nil))
+    return nil
+}
+```
+
+steve can be run directly as a binary for testing. By default it runs on ports 9080 and 9443:
+
+```
+export KUBECONFIG=your.cluster
+go run main.go
+```
+
+The API can be accessed by navigating to https://localhost:9443/v1.
+
+Steve Features
+--------------
+
+Steve's main use is as an opinionated consumer of [rancher/apiserver](), which
+it uses to dynamically register every Kubernetes API as its own. It implements
+apiserver [Stores]() to use Kubernetes as its data store.
+
+### Stores
+
+Steve uses apiserver Stores to transform and store data, mainly in Kubernetes.
+The main mechanism it uses is the proxy store, which is actually a series of
+four nested stores and a "partitioner". It can be instantiated by calling
+[NewProxyStore](). This gives you:
+
+* [`proxy.errorStore`]() - translates any returned errors into HTTP errors
+* [`proxy.WatchRefresh`]() - wraps the nested store's Watch method, canceling the
+  watch if access to the watched resource changes
+* [`partition.Store`]() - the partition store wraps the nested store's List method
+  and parallelizes the request according to the given partitioner, and
+  additionally implements filtering, sorting, and pagination on the
+  unstructured data from the nested store
+* [`proxy.rbacPartitioner`]() - the partitioner fed to the `partition.Store`
+  which
+  allows it to parallelize requests based on the user's access to certain
+  namespaces or resources
+* [`proxy.Store`]() - the Kubernetes proxy store which performs the actual
+  connection to Kubernetes for all operations
+
+The default schema additionally wraps this proxy store in [`metrics.Store`](),
+which records request metrics to Prometheus, by calling
+[`metrics.NewNetricsStore()`]() on it.
+
+Steve provides two additional exported stores that are mainly used by Rancher's catalogv2 package:
+
+* [`selector.Store`]() - wraps the list and watch commands with a label
+* [`switchschema.Store`]() - transforms the object's schema
+
+### Schemas
+
+Steve watches all Kuberentes API resources, including builtins, CRDs, and
+APIServices, and registers them under its own /v1 endpoint. Schemas can be
+queried from the /v1/schemas endpoint. Steve also registers a few of its own
+schemas not from Kubernetes to facilitate certain use cases.
+
+#### Cluster
+
+Steve creates a fake local cluster to use in standalone scenarios when there is
+not a real management.cattle.io Cluster resource available. Rancher overrides
+this and sets its own customizations on the cluster resource.
+
+#### UserPreferences
+
+Userpreferences in steve provides a way to configure dashboard preferences
+through a configfile named prefs.json. Rancher overrides this and uses the
+preferences.management.cattle.io resource for preference storage instead.
+
+#### Counts
+
+Counts keeps track of the number of resources and updates the count in a
+buffered stream that the dashboard can subscribe to.
+
+#### Subscribe
+
+Subscription to events is a feature in rancher/apiserver that is exposed in steve.
+
+TODO
+
+### Schema Templates
+
+Existing schemas can be customized using schema templates. You can customize individual schemas or apply customizations to all schemas.
+
+For example, if you wanted to customize the store for secrets so that secret data is always redacted, you could implement a store like this:
+
+```go
+import (
+	"github.com/rancher/apiserver/pkg/store/empty"
+	"github.com/rancher/apiserver/pkg/types"
+)
+
+type redactStore struct {
+	empty.Store
+}
+
+func (r *redactStore) ByID(_ *types.APIRequest, _ *types.APISchema, _ string) (types.APIObject, error) {
+	return types.APIObject{
+		Object: map[string]string{
+			"value": "[redacted]",
+		},
+	}, nil
+}
+
+func (r *redactStore) List(_ *types.APIRequest, _ *types.APISchema) (types.APIObjectList, error) {
+	return types.APIObjectList{
+		Objects: []types.APIObject{
+			{
+				Object: map[string]string{
+					"value": "[redacted]",
+				},
+			},
+		},
+	}, nil
+}
+```
+
+and then create a schema template for the schema with ID "secrets" that uses
+that store:
+
+```go
+import (
+	"github.com/rancher/steve/pkg/schema"
+)
+
+template := schema.Template{
+	ID: "secret",
+	Store: &redactStore{},
+}
+```
+
+You could specify the same by providing the group and kind:
+
+```go
+template := schema.Template{
+	Group: "", // core resources have an empty group
+	Kind: "secret",
+	Store: &redactStore{},
+}
+```
+
+then add the template to the schema factory:
+
+```go
+schemaFactory.AddTemplate(template)
+```
+
+As another example, if you wanted to add custom field to all objects in a
+collection response, you can add a schema template with a collection formatter
+omit the ID or the group and kind:
+
+```
+template := schema.Template{
+	Customize: func(schema *types.APISchema) {
+		schema.CollectionFormatter = func(apiOp *types.APIRequest, collection *types.GenericCollection) {
+			schema.CollectionFormatter = func(apiOp *types.APIRequest, collection *types.GenericCollection) {
+				for i, d := range collection.Data {
+					fmt.Println(i, d)
+					obj := d.APIObject.Object.(*unstructured.Unstructured)
+					obj.Object["tag"] = "custom"
+				}
+			}
+		}
+	}
+}
+```
+
+### Schema Access Control
+
+Steve implements access control on schemas based on the user's RBAC in
+Kubernetes.
+
+The apiserver [`Server`]() object exposes an AccessControl field which is used
+to customize how access control is performed on server requests.
+
+An [`accesscontrol.AccessStore`]() is stored on the schema factory. When a user
+makes any request, the request handler first finds all the schemas that are
+available to the user. To do this, it first retrieves an
+[`accesscontrol.AccessSet`]() for the user object by calling [`AccessFor`]() on
+the user. The AccessSet contains a map of resources and the verbs the can use
+on them. The AccessSet is calculated by looking up all of the user's role
+bindings and cluster role bindings for the user's name and group. The result is
+cached, and the cached result is used until the user's role assignments change.
+Once the AccessSet is retrieved, each registered schema is checked for
+existence in the AccessSet, and filtered out if it is not available.
+
+This final set of schemas is inserted into the [`types.APIRequest`]() object
+and passed to the apiserver handler.
+
+### Authentication
+
+Steve authenticates incoming requests using a customizeable authentication
+middleware. The default authenticator in standalone steve is the
+[AlwaysAdmin]() middleware, which accepts all incoming requests and sets admin
+attributes on the user. The authenticator can be overridden by passing a custom
+middleware to the steve server:
+
+```go
+import (
+    "context"
+    "github.com/rancher/steve/pkg/server"
+    "github.com/rancher/steve/pkg/auth"
+    "k8s.io/apiserver/pkg/authentication/user"
+)
+
+func run() {
+    restConfig := getRestConfig()
+    authenticator := func (req *http.Request) (user.Info, bool, error) {
+        username, password, ok := req.BasicAuth()
+        if !ok {
+            return nil, false, nil
+        }
+        if username == "hello" && password == "world" {
+            return &user.DefaultInfo{
+                Name: username,
+                UID: username,
+                Groups: []string{
+                    "system:authenticated",
+                },
+            }, true, nil
+        }
+        return nil, false, nil
+    }
+    server := server.New(context.TODO(), restConfig, &server.Options{
+        AuthMiddleware: auth.ToMiddlware(auth.AuthenticatorFunc(authenticator)),
+    }
+    server.ListenAndServe(context.TODO(), 9443, 9080, nil)
+}
+```
+
+Once the user is authenticated, if the request is for a Kubernetes resource
+then steve must proxy the request to Kubernetes, so it needs to transform the
+request. Steve passes the user Info object from the authenticator to a proxy
+handler, either a generic handler or an impersonating handler. The generic
+[Handler]() mainly sets transport options and cleans up the headers on the
+request in preparation for forwarding it to Kubernetes. The
+[ImpersonatingHandler]() uses the user Info object to set Impersonate-* headers
+on the request, which Kubernetes uses to decide access.
+
+### Dashboard
+
+Steve is designed to be consumed by a graphical user inferface and therefore
+serves one by default, even in the test server. The default UI is the Rancher
+Vue UI hosted on releases.rancher.com. It can be viewed by visiting the running
+steve instance on port 9443 in a browser.
+
+The UI can be enabled and customized by passing options to [NewUIHandler()]().
+For example, to create a route that serves an alternative index.html, add the
+index.html to a directory called `./ui`, then create a route that serves a
+custom UI handler:
+
+```go
+import (
+	"net/http"
+    "github.com/rancher/steve/pkg/ui"
+	"github.com/gorilla/mux"
+)
+
+func routes() http.Handler {
+	custom := ui.NewUIHandler(&Options{
+		Index: func() string {
+			return "./ui/index.html"
+		},
+	}
+	router := mux.NewRouter()
+	router.Handle("/hello", custom.IndexFile())
+	return router
+```
+
+If no options are set, the UI handler will serve the latest index.html file from the Rancher Vue UI.
+
+### Misc
+
+#### Cluster Cache
+
+The cluster cache keeps watches of all resources with registered schemas. This
+is mainly used to update the summary cache and resource counts, but any module
+could add a handler to react to any resource change or get cached cluster data.
+For example, if we wanted a handler to log all "add" events for newly created
+secrets:
+
+```go
+import (
+    "context"
+    "github.com/rancher/steve/pkg/server"
+    "k8s.io/apimachinery/pkg/runtime"
+    "github.com/sirupsen/logrus"
+    "k8s.io/apimachinery/pkg/runtime/schema"
+)
+func logSecretEvents(server *server.Server) {
+    server.ClusterCache.OnAdd(context.TODO(), func(gvk schema.GroupVersionKind, key string, obj runtime.Object) error {
+        if gvk.Kind == "Secret" {
+            logrus.Infof("[event] add: %s", key)
+        }
+        return nil
+    })
+}
+```
+
+#### Aggregation
+
+TODO