diff --git a/gaps/gap-8_basic_auth_runtime/gap-8_basic_auth_runtime.md b/gaps/gap-8_basic_auth_runtime/gap-8_basic_auth_runtime.md deleted file mode 100644 index 8482b738..00000000 --- a/gaps/gap-8_basic_auth_runtime/gap-8_basic_auth_runtime.md +++ /dev/null @@ -1,41 +0,0 @@ ---- -gap: -title: -description: -author: , FirstName (@GitHubUsername) and GitHubUsername (@GitHubUsername)> -status: Draft -type: -requires (*optional): -replaces (*optional): ---- - -This is the template to be used for new GAP submissions. Note this has been heavily inspired by Ethereum [EIP template](https://github.com/ethereum/EIPs/blob/master/eip-template.md). - -The GAP number will be assigned by an editor. A GAP summary document shall be submitted by opening a pull request with GAP-related files placed in `./gaps/gap-draft_title/` directory. To submit your GAP, please use an abbreviated title in the filename, `gap-draft_title.md`. - -## Abstract -Abstract is a multi-sentence (short paragraph) technical summary. This should be a very terse and human-readable version of the specification section. Someone should be able to read only the abstract to get the gist of what this specification does. - -## Motivation -The motivation section should describe the "why" of this GAP. What problem does it solve? What benefit does it provide to the Golem ecosystem? What use cases does this GAP address? - -## Specification -The technical specification should describe the syntax and semantics of any new feature. - -## Rationale -The rationale fleshes out the specification by describing what motivated the design and why particular design decisions were made. It should describe alternate designs that were considered and related work. - -## Backwards Compatibility -All GAPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. The GAP **must** explain how the author proposes to deal with these incompatibilities. - -## Test Cases -Test cases are very useful in summarizing the scope and specifics of a GAP. If the test suite is too large to reasonably be included inline, then consider adding it as one or more files in `./gaps/gap-draft_title/` directory. - -## [Optional] Reference Implementation -An optional section that contains a reference/example implementation that people can use to assist in understanding or implementing this specification. If the implementation is too large to reasonably be included inline, then consider adding it as one or more files in `./gaps/gap-draft_title/`. - -## Security Considerations -All GAPs must contain a section that discusses the security implications/considerations relevant to the proposed change. Include information that might be important for security discussions, surfaces risks and can be used throughout the life cycle of the proposal. E.g. include security-relevant design decisions, concerns, important discussions, implementation-specific guidance and pitfalls, an outline of threats and risks and how they are being addressed. - -## Copyright -Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). \ No newline at end of file diff --git a/gaps/gap-8_http_auth_runtime/gap-8_http_auth_runtime.md b/gaps/gap-8_http_auth_runtime/gap-8_http_auth_runtime.md new file mode 100644 index 00000000..d4bc69e9 --- /dev/null +++ b/gaps/gap-8_http_auth_runtime/gap-8_http_auth_runtime.md @@ -0,0 +1,433 @@ +--- +gap: 8 +title: HTTP authentication runtime +description: A gateway runtime for already-running HTTP-based services. +author: mf (@mfranciszkiewicz) +status: Draft +type: Feature +--- + +## Abstract + +This document describes a runtime gateway between the Golem Marketplace and an HTTP-based service, to be made publicly +accessible over the Internet. Lifecycle and configuration management of that service lies with the authority of +a Provider Agent administrator. + +Inbound HTTP requests from the Internet to the service are routed via a custom reverse HTTP proxy implementation. +The proxy authorizes the requests and collects per-user, per-endpoint usage statistics +for the purpose of billing Requestors on the Golem Marketplace. + +The runtime is responsible for adding and removing users authorized to use the service on Requestor's demand. The former +is achieved via Proxy's Management API, available only on a private network address. + +## Motivation + +The specification has been derived from the previous work on Erigon / Lighthouse service runtimes. +It was concluded that a generic HTTP authentication runtime could be developed instead. + +## Specification + +Key assumptions made while designing the runtime: + +- Provider Agent administrator can add / remove services by adding / removing config files from a designated directory +- The runtime controls access to services + - Enforces HTTP basic authentication by default; the runtime is extensible in this manner + - There is a separate access list for every service + - Requestor can pass in the access credentials (username + password); these should **not** be generated by the runtime. Username collision should be handled. +- Runtime should expose number of requests as a custom usage metric. + - Breakdown of request count into particular service endpoints (i.e. by URI) + +Key responsibilities of the reverse HTTP proxy: + +- securing external HTTP communication with TLS + - plain HTTP mode can be explicitly enabled via a configuration file / command line arguments +- forwarding authorized HTTP requests to services + +- exposing a private Management API, utilized the Runtime, for: + - exposing / hiding a service + - listing registered services + - adding / removing user credentials per service + - listing registered users + - reading usage statistics per service user + +### Runtime implementation + +The implementation will be based on [ya-runtime-sdk](https://github.com/golemfactory/ya-runtime-sdk). + +#### Command line arguments + +Command line should accept a service name argument, according to the agreement. + +An example can be found [here](https://github.com/golemfactory/ya-runtime-sdk/blob/mf/dynamic-rt-name/examples/environment/src/main.rs). + +#### Configuration + +The runtime configuration file may contain the following values: +- proxy Management API network address +- proxy log directory path +- additional service configuration lookup directories + +#### Background jobs + +Implement the following in order to publish the custom runtime counter values: + +1. Request counter fetching + + Periodically calls the statistics endpoints of proxy's Management API regarding all users registered + by the current Requestor. + +2. Publishing counter values to the ExeUnit Supervisor + + Periodically updates the custom counter value (total requests made by a user) via the interface provided by Runtime SDK. + An example of a custom counter can be found [here](https://github.com/golemfactory/ya-test-runtime-counters/blob/main/src/main.rs). + + +#### `Runtime` trait implementation + +1. `deploy` + +- Read the variant of the requested service from command line arguments +- Perform a service configuration file lookup and locate the requested service +- Save deployment metadata to the current working directory + +2. `start` + +- If not running, **in background**, start an instance of the proxy server; the proxy binary is expected to reside in the + same directory as the runtime binary. The proxy binary is guarded via a lock file when starting up. + - loop + - if no lock file is present, create and lock the file + - start the process and use the via Management API to check whether it's responding properly + - break + - if the file is present, wait for acquiring the lock + - if the Management API is responding, break + - else, create the process and await until it's responsive + +3. `run` + + - `user add ` + + Adds a new user via Management API and within local memory. If a username exists, returns an error + (with a proper message outputted to `stderr`); the runtime is terminated. + + - `user remove ` + + Removes a user via Management API. Performs a lookup in a local memory store firstt. + + - `user stats ` + + Reads the specific counter value for the given user. The foreseen counter + names will be set to the requested URIs (locations) configured in nginx. + + The commands above support an `-a,--auth` for setting the authentication method: + + - `basic` for HTTP basic authentication (default) + - other forms of authentication are planned for future releases + +5. `transfer` + + - none + +6. `offer-template` + +- extra Offer properties defined in service configuration +- supported proxy authentication methods as runtime capabilities +- (optional) partial service configuration as Offer properties (e.g. cpu threads, timeout values, user auth method) +- (optional) in case of self-signed certificates, add a `cert` string property in the runtime property namespace, + containing the certificate checksum, in the following format: `sha3:0badbeef` + +6. `test` + +- perform a service configuration file lookup +- assert that the proxy binary is runnable +- assert that the service HTTP interface is available +- (optional) perform a public IP assertion check for the Provider + +#### Shutdown procedure + +Implement a process termination handler using `ya-runtime-sdk`: + +1. Fetch last available counter values via Management API +2. Publish the latest known counter values via `ya-runtime-sdk` +3. De-register the user (if any) via Management API + +### Deployment + +Each service should be assigned its own runtime name and use the common `ya-runtime-basic-auth` binary, e.g.: + +```json +[ + { + "name": "ya-runtime-erigon", + "version": "0.1.0", + "supervisor-path": "exe-unit", + "runtime-path": "ya-runtime-http-auth/ya-runtime-http-auth", + "description": "The Basic-Auth runtime for the Erigon service", + "extra-args": [ + "--runtime-managed-image", + "--runtime-arg", "ya-runtime-erigon" + ] + } +] +``` + +depending on the installation method, located at: +- `/usr/lib/yagna/plugins/ya-runtime-erigon.json` (manual or `deb` installation) +- `~/.local/lib/yagna/plugins/ya-runtime-erigon.json` (installer script) + +### HTTP proxy + +Multi-threaded TCP server supporting HTTP 1.1 and HTTP 2.0 protocols. + +By default, the proxy binds to a single local address: +- private (`localhost:6668`) for the Management API + +For each service, a separate HTTPS (TLS 1.2, 1.3) server can be spawned. Each service **can** be configured with +a custom listening socket address and certificate / key pair, otherwise it is expected to use the defaults. +Domain configuration lies in Provider Agent administrator's responsibility. + +The proxy supports a plain HTTP mode for development purposes, enabled explicitly via command line arguments. + +#### Management API (private address space) + +Management API implementation keeps track of the number of registered users and de-registers services when the user counter reaches 0 +(i.e. at least one user needs to be registered first). + +1. Service management + +- `GET /services -> List[Service]` ? [`pageSize`, `offset`, `count`] + + Retrieves a list of registered services + +- `POST /services` w/ `CreateService` + + Registers a service; returns 204 if the service already exists and all service parameters match. + + Tries to spawn an additional HTTP server thread when any of these conditions are met: + - the listening address is not covered by the default configuration + - service port is not covered by the default configuration + + In other cases, the default server thread will be used, but the service configuration must match the default + proxy configuration (listening address, certificates, etc.). + +- `GET /services/{service_name} -> Service` + + Retrieves service details + +- `DELETE /services/{service_name}` + + De-registers a service and stops the extra HTTP server (if any) + +2. User management per service + +- `GET /services/{service_name}/users -> List[User]` ? [`pageSize`, `offset`, `count`] + + Lists users registered within the service + +- `POST /services/{service_name}/users` w/ `CreateUser` + + Registers a new user within the service; returns 400 if a username already exists + +- `GET /services/{service_name}/users/{user_name} -> User` + + Retrieves single registered user information + +- `DELETE /services/{service_name}/users/{user_name}` + + De-registers a user from the service. If the user count drops to 0, de-registers the service and stops the + extra HTTP server (if any) + +3. User statistics + +- `GET /services/{service_name}/users/{user_name}/stats -> UserStats` + + Per-service user statistics + +- `GET /services/{service_name}/users/{user_name}/endpoints/stats -> UserEndpointStats` + + Per-endpoint, per-service user statistics + +- `GET /stats -> GlobalStats` (optional) + + Proxy-wide (global) statistics + +Runtime developers are expected to create an OpenAPI specification of the Management API. + +#### Proxy (public address space) + +The public TCP connection handler inspects and mangles each incoming HTTP request only to forward that request +to the destination service server and tunnel the response. + +The inspection and mangling module operates as follows: + +1. Resolve the registered service basing on the requested URI +2. Authorize credentials included in the header +3. Rewrite the location in request's header to a relative service URI (the extra part of the proxied service endpoint) +4. Add / replace a `X-Forwarded-Host` header, if a domain name was set for the service +5. Add / replace a `X-Forwarded-For` header containing the connecting client's IP address +6. (optional, TBD) Add / replace a `Forward` header instead `X-Forwarded-For` +7. Forward the requested packet to the destination service server + +The proxy server should update statistics for each unauthorized (global), failed (user / global) and successful request (user / global). + +#### API models + +- `CreateService` + + ```json + { + "name": "service_name", + "bind": "1.120.111.43:443", + "domain": "example.com", + "user": { + "auth": { + "method": "basic" + }, + "requestTimeout": 10000, + "responseTimeout": 2000 + }, + "auth": { + "method": "basic" + }, + "cert": { + "path": "/etc/certs/cert.crt", + "keyPath": "/etc/certs/cert.key" + }, + "from": "/service", + "to": "http://127.0.0.1:12345/api/v1/service", + "requestTimeout": 10000, + "responseTimeout": 2000, + "cpuThreads": 2 + } + ``` + +- `Service` + + `CreateService` extended with a `createdAt` property (RFC3339 timestamp as string). + +- `CreateUser` + + ```json + { + "name": "username", + "password": "dXNlcm5hbWU=" + } + ``` + + - `password` - base64 encoded password + +- `User` + + ```json + { + "name": "username", + "createdAt": "2022-01-12T23:20:50.52Z" + } + ``` + +- `UserStats` + + ```json + { + "total": 12, + "failures": 0 + } + ``` + +- `UserEndpointStats` + + ```json + { + "/service/run": 1, + "/service/build": 12 + } + ``` + +- `GlobalStats` (optional) + + ```json + { + "users": 43, + "services": 3, + "requests": { + "total": 11314, + "unauthorized": 44 + } + } + ``` + +#### CLI + +The proxy server can be parameterized via the following command line arguments: + +- (optional) Management API addresses +- (optional) log directory path +- (optional) default proxy API binding address (defaults to `0.0.0.0:443`) +- (optional) TLS certificate path +- (optional) TLS certificate key path + +### Service configuration + +Configuration files should contain the following service information in a JSON / YAML / TOML format: + +- name (identifier) +- root proxy REST endpoint (public) +- root service URL (local) +- (optional) user request timeout +- (optional) user response timeout +- (optional) service request timeout +- (optional) service response timeout +- (optional) server binding addresses +- (optional) dedicated certificate entry (e.g. when binding to a separate network interface, resolvable under a different domain name) + - certificate path + - certificate key path +- (optional) number of designated CPU threads (defaults to all available logical cores) +- (optional) extra static properties to be included in the Offer +- (unused) user authentication method (currently basic auth only) +- (unused) service authentication method (currently none) + +## Rationale + +### Custom reverse HTTP proxy + +Per-user, per-endpoint statistics in (e.g.) `nginx` OSS are not available out of the box. These statistics would need to be derived +from server logs, which follow a specific (and often user-defined) format, so a custom parser would have to be implemented. +While it's valid to spawn multiple instances of the runtime binary, there would be multiple instances of the parser +running in parallel; this solution does not scale well enough for a large number of runtimes. + +Although most web servers are pluggable, the plugin development process would require setting up additional CD infrastructure +for building with multiple versions of that software. Additionally, if the server running on Provider's machine is used +for other purposes, interoperability with other plugins will increase the difficulty and time required for development. +Third-party server configuration would become a prerequisite that can be difficult to validate, becoming a source of issues for +the end users. + +With a custom proxy implementation, a full flexibility in implementing usage counters is gained. The proxy will be packaged as +a portable binary, so that it does not interfere with system configuration. + +### Self-signed certificates + +The proxy will allow self-signed certificates and provide the certificate hash for clients to verify the public key used; +they need not verify the domain name and the CA signature. + +## Backwards Compatibility + +No backward-compatibility issues were discovered in the design / development process. + +## Test Cases + +N/A + +## Reference Implementation + +- [HTTP proxy](https://github.com/golemfactory/ya-runtime-basic-auth/tree/mf/http-proxy/crates/ya-http-proxy) + +## Security Considerations + +- the provider needs to properly configure TLS certificates so that the authentication credentials + cannot be sniffed (i.e. they won't be sent as plain text) +- self-signed certificate hashes need to be included in the offer in order to prevent Man In The Middle attacks +- a proxy server can operate in a plain HTTP mode, which is only recommended for development. Still, it's possible + for the server itself to be proxied by e.g. `nginx` taking over the responsibility of managing certificates + and encrypting connections. + +## Copyright +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).