Merge branch 'main' into ci-refactor

docs/API/V1/README.md

@@ -8,6 +8,14 @@ Once you know the hostname of a Pbench Server, you can ask for the API
 configuration using the [endpoints](endpoints.md) API. This will report the
 server's version and a list of all API end points supported by the server.
 
+## Pbench Server configuration settings
+
+Some aspects of Pbench Server operation can be controlled by a user holding the
+`ADMIN` [role](../access_model.md) through the
+[server configuration](./server_config.md) API, including disabling the Pbench
+Server API while the server is undergoing maintenance and setting a banner
+message accessible to clients.
+
 ## Pbench Server resources
 
 ### Datasets

docs/API/V1/contents.md

@@ -47,6 +47,12 @@ The authenticated client does not have READ access to the specified dataset.
 Either the `<dataset>` or the relative `<path>` within the dataset does not
 exist.
 
+`503`   **SERVICE UNAVAILABLE** \
+The server has been disabled using the `server-state` server configuration
+setting in the [server configuration](./server_config.md) API. The response
+body is an `application/json` document describing the current server state,
+a message, and optional JSON data provided by the system administrator.
+
 ## Response body
 
 The `application/json` response body consists of a JSON object describing the

docs/API/V1/inventory.md

@@ -49,6 +49,12 @@ The `<path>` refers to a directory. Use
 `/api/v1/dataset/contents/<dataset><path>` to request a JSON response document
 describing the directory contents.
 
+`503`   **SERVICE UNAVAILABLE** \
+The server has been disabled using the `server-state` server configuration
+setting in the [server configuration](./server_config.md) API. The response
+body is an `application/json` document describing the current server state,
+a message, and optional JSON data provided by the system administrator.
+
 ## Response body
 
 The `application/octet-stream` response body is the raw byte stream contents of

docs/API/V1/list.md

@@ -87,6 +87,12 @@ by `owner` or `access=private`.
 The client asked to filter `access=private` datasets for an `owner` for which
 the client does not have READ access.
 
+`503`   **SERVICE UNAVAILABLE** \
+The server has been disabled using the `server-state` server configuration
+setting in the [server configuration](./server_config.md) API. The response
+body is an `application/json` document describing the current server state,
+a message, and optional JSON data provided by the system administrator.
+
 ## Response body
 
 The `application/json` response body contains a list of objects which describe

docs/API/V1/server_config.md

@@ -0,0 +1,268 @@
+# `GET /api/v1/server/configuration[/][{key}]`
+
+This API returns an `application/json` document describing the Pbench Server
+configuration settings. When the `{key}` parameter is specified, the API will
+return the specific named server configuration setting. When `{key}` is omitted,
+all server configuration settings will be returned.
+
+## Query parameters
+
+None.
+
+## Request headers
+
+`authorization: bearer` token [_optional_] \
+*Bearer* schema authorization may be specified, but is not required to `GET`
+server configuration settings.
+
+## Response headers
+
+`content-type: application/json` \
+The return is a serialized JSON object with the requested server configuration
+settings.
+
+## Response status
+
+`400`   **BAD REQUEST** \
+The specified `{key}` value (see [settings](#server-configuration-settings))
+is unknown.
+
+## Response body
+
+The `application/json` response body is a JSON document containing the requested
+server configuration setting key and value or, if no `{key}` was specified, all
+supported server configuration settings.
+
+### Examples
+
+```
+GET /api/v1/server/configuration/dataset-lifetime
+{
+    "dataset-lifetime": "4"
+}
+
+GET /api/v1/server/configuration/
+{
+    "dataset-lifetime": "4",
+    "server-banner": {
+        "message": "Server will be down for maintenance on Tuesday!",
+        "contact": "[email protected]"
+    }
+    "server-state": {
+        "status": "readonly",
+        "message": "rebuilding index ... back to normal soon"
+    }
+}
+```
+
+# `PUT /api/v1/server/configuration[/][{key}]`
+
+This API allows a user holding the `ADMIN` role to modify server configuration
+settings. When the `{key}` parameter is specified, the API will modify a single
+named configuration setting. When `{key}` is omitted, the `application/json`
+request body can be used to modify multiple configuration settings at once.
+
+## Query parameters
+
+`value`    string \
+When a single configuration setting is specified with `{key}` in the URI, you
+can specify a string value for the parameter using this query parameter without
+an `application/json` request body. For example,
+`PUT /api/v1/server/configuration/key?value=1`.
+
+You cannot specify complex JSON configuration settings this way. Instead, use
+the `value` field in the `application/json` request body.
+
+## Request body
+
+When specifying a complex JSON value for a server configuration setting, or when
+specifying multiple configuration settings, the data to be set is specified in
+an `application/json` request body.
+
+You can specify a single `{key}` in the URI and then specify the value
+using a `value` field in the `application/json` request body instead of using
+the `value` query parameter. You can do this even if the value is a simple
+string, although it's more useful when you need to specify a JSON object value.
+For example,
+
+```
+PUT /api/v1/server/configuration/server-state
+{
+    "value": {"status": "enabled"}
+}
+```
+
+If you omit the `{key}` value from the URI, specify all configuration settings
+you wish to change in the `application/json` request body. You can specify a
+single server configuration setting, or any group of server configuration
+settings at once. For example,
+
+```
+PUT /api/v1/server/configuration/
+{
+    "server-state": {"status": "disabled", "message": "down for maintenance"},
+    "server-banner": {"message": "Days of 100% uptime: 0"}
+}
+```
+
+## Request headers
+
+`authorization: bearer` token \
+*Bearer* schema authorization is required to change any server configuration
+settings. The authenticated user must have `ADMIN` role.
+
+## Response headers
+
+`content-type: application/json` \
+The response body is a serialized JSON object with the selected server
+configuration settings.
+
+## Response status
+
+`401`   **UNAUTHORIZED** \
+The client did not provide an authentication token.
+
+`403`   **FORBIDDEN** \
+The client's authentication token does not correspond to a user with `ADMIN`
+role.
+
+## Response body
+
+The `application/json` response body for `PUT` is exactly the same as for
+[`GET`](#response-body) when the same server configuration settings are
+requested, showing only the server configuration settings that were changed
+in the `PUT`.
+
+_This request:_
+```
+PUT /api/v1/server/configuration/dataset-lifetime?value=4
+```
+
+_returns this response:_
+```
+{
+    "dataset-lifetime": "4"
+}
+```
+
+
+_And this request:_
+```
+PUT /api/v1/server/configuration
+{
+    "dataset-lifetime": "4 days",
+    "server-state": {"status": "enabled"}
+}
+```
+
+_returns this response:_
+```
+{
+    "dataset-lifetime": "4",
+    "server-state": {"status": "enabled"}
+}
+```
+
+## Server configuration settings
+
+### dataset-lifetime
+
+The value for the `dataset-lifetime` server configuration setting is the *maximum*
+number of days a dataset can be retained on the server. When each dataset is
+uploaded to the server, a "deletion date" represented by the dataset
+[metadata](../metadata.md) key `server.deletion` is calculated based on this
+value and user preferences (which may specify a shorter lifetime, but not a
+longer lifetime). When a dataset has remained on the server past the
+`server.deletion` date, it may be removed automatically by the server to
+conserve space.
+
+The number of days is specified as an string representing an integer, optionally
+followed by a space and `day` or `days`. For example, "4" or "4 days" or "4 day"
+are equivalent.
+
+```
+{
+    "dataset-lifetime": "4"
+}
+```
+
+### server-banner
+
+This server configuration setting allows a server administrator to set an
+informational message that can be retrieved and displayed by any client, for
+example as a banner on a UI. The value is a JSON object, containing at least
+a `message` field.
+
+Any additional JSON data may be provided. The server will store the entire
+JSON object and return it when a client requests it with
+`GET /api/v1/server/configuration/server-banner`. The server will not interpret
+any information in this JSON object.
+
+For example, the following are examples of valid banners:
+
+```
+{
+    "server-banner": {
+        "message": "Have a Happy Pbench Day"
+    }
+}
+```
+
+```
+{
+    "server-banner": {
+        "message": "The server will be down for 2 hours on Monday, July 31",
+        "contact": {
+            "email": "[email protected]",
+            "phone": "(555) 555-5555",
+            "hours": ["8:00 EST", "17:00 EST"]
+        },
+        "statistics": {
+            "datasets": 50000,
+            "hours-up": 2.3,
+            "users": 26
+        }
+    }
+}
+```
+
+### server-state
+
+This server configuration setting allows a server administrator to control
+the operating state of the server remotely. As for
+[`server-banner`](#server-banner), the value is a JSON object, and any JSON
+fields passed in to the server will be returned to a client. The
+following fields have special meaning:
+
+**`status`** \
+The operating status of the server.
+
+* `enabled`: Normal operation.
+* `disabled`: Most server API endpoints will fail with the **503** (service
+unavailable) HTTP status. However a few endpoints are always allowed:
+  * [endpoints](./endpoints.md) for server configuration information;
+  * [login](./login.md) because only an authenticated user with `ADMIN` role
+    can modify server configuration settings;
+  * [logout](./logout.md) for consistency;
+  * [server_configuration](./server_config.md) to allow re-enabling the server
+    or modifying the banner.
+* `readonly`: The server will respond to `GET` requests for information, but will return **503** (service unavailable) for any attempt to modify server state. (With the same exceptions as listed above for `disabled`.)
+
+**`message`** \
+A message to explain downtime. This is required when the `status` is `disabled`
+or `readonly` and optional otherwise.
+
+When the server status is `disabled`, or when it's `readonly` and a client
+tries to modify some data through the API, the server will fail the request
+with a `503` (service unavailable) error. It will return an `application/json`
+error response payload containing the full `server-state` JSON object. The
+`message` key in an error response is a standard convention, and many clients
+will display this as an explanation for the failure. The client will also have
+access to any additional information provided in the `server-state` JSON object.
+
+Note that you can set a `message` when the `status` is `enabled` but it won't
+be reported to a client unless a client asks for the `server-state`
+configuration setting. The `server-state` message is intended to explain server
+downtime when an API call fails. The `server-banner` configuration setting
+is generally more appropriate to provide client information under normal
+operating conditions.

exec-unittests

@@ -12,17 +12,17 @@ fi
 shift
 
 # The first argument will be which major sub-system to run tests for: the
-# agent side or the server side of the source tree.
+# agent code, the functional test client code, or the server code.
 
 major="${1}"
 shift
-major_list="agent server"
+major_list="agent client server"
 if [[ -n "${major}" ]]; then
-    if [[ "${major}" != "agent" && "${major}" != "server" && "${major}" != "both" ]]; then
-        printf -- "Expected major sub-system to be either 'agent', 'server', or 'both', got '%s'\n" "${major}" >&2
+    if [[ "${major}" != "agent" && "${major}" != "server" && "${major}" != "client" && "${major}" != "all" ]]; then
+        printf -- "Expected major sub-system to be 'agent', 'client', 'server', or 'all', got '%s'\n" "${major}" >&2
         exit 2
     fi
-    if [[ "${major}" != "both" ]]; then
+    if [[ "${major}" != "all" ]]; then
         major_list="${major}"
     fi
 fi
@@ -131,7 +131,10 @@ if [[ -z "${subtst}" || "${subtst}" == "python" ]]; then
     for _major in ${major_list}; do
         _pytest_majors="${_pytest_majors} pbench.test.unit.${_major}"
         if [[ "${_major}" == "agent" ]]; then
-            _pytest_majors="${_pytest_majors} pbench.test.functional"
+            # TODO: real functional tests require a deployed instance. Current
+            # agent "functional" tests are mocked Click tests rather than true
+            # "functional" tests.
+            _pytest_majors="${_pytest_majors} pbench.test.functional.agent"
         fi
     done
 
@@ -162,7 +165,9 @@ trap "rm -f ${_para_jobs_file}" EXIT INT TERM
 let count=0
 for _major in ${major_list}; do
     # Verify the Agent or Server Makefile functions correctly.
-    verify_make_source_tree ${_major} || rc=1
+    if [[ "${_major}" == "agent" || "${_major}" == "server" ]]; then
+        verify_make_source_tree ${_major} || rc=1
+    fi
 
     if [[ "${_major}" == "agent" ]]; then
         # The parallel program is really cool.  The usage of `parallel` is
@@ -186,7 +191,7 @@ for _major in ${major_list}; do
             (( count++ ))
             run_legacy ${_major} bin ${posargs} || rc=1
         fi
-    else
+    elif [[ "${_major}" != "client" ]]; then
         printf -- "Error - unrecognized major test sub-set, '%s'\n" "${_major}" >&2
         rc=1
     fi

jenkins/Makefile

@@ -14,6 +14,7 @@
 #  push-rpmbuild-all:  pushes all the default RPM build containers to the registry
 #
 
+
 # Include definition of _DISTROS
 include ../utils/utils.mk