From 711247a36e793db072566cc3585a9a84d379cbca Mon Sep 17 00:00:00 2001 From: David Butenhof Date: Wed, 31 May 2023 12:23:57 -0400 Subject: [PATCH] Review comments --- docs/Server/API/README.md | 14 +++++++---- docs/Server/API/V1/README.md | 17 -------------- docs/Server/API/V1/relay.md | 45 +++++++++++++++++++----------------- docs/Server/API/V1/upload.md | 6 +++-- 4 files changed, 37 insertions(+), 45 deletions(-) diff --git a/docs/Server/API/README.md b/docs/Server/API/README.md index 814a2ebdbf..50714c11a1 100644 --- a/docs/Server/API/README.md +++ b/docs/Server/API/README.md @@ -9,8 +9,12 @@ or just "datasets". The [V1 API](V1/README.md) provides a REST-like functional interface. -The Pbench Server APIs use a combination of parameters embedded in the URI path -along with query parameters and serialized JSON parameters (mimetype -`application/json`) for requests. A few exceptions use raw byte streams -(`application/octet-stream`) to allow uploading new datasets and to access -individual files from a dataset. +The Pbench Server APIs accept parameters from a variety of sources. See the +individual API documentation for details. +1. Some parameters, especially "resource ids", are embedded in the URI, such as +`/api/v1/datasets/`; +2. Some parameters are passed as query parameters, such as +`/api/v1/datasets?name:fio`; +3. For `PUT` and `POST` APIs, parameters may also be passed as a JSON +(`application/json` content type) request payload, such as +`{"metadata": {"dataset.name": "new name"}}` diff --git a/docs/Server/API/V1/README.md b/docs/Server/API/V1/README.md index 23f461734c..9d677bf04d 100644 --- a/docs/Server/API/V1/README.md +++ b/docs/Server/API/V1/README.md @@ -132,20 +132,3 @@ through the `directories` list of each The [inventory](inventory.md) API returns the raw byte stream of any regular file within the directory hierarchy, including log files, postprocessed JSON files, and benchmark result text files. - -### Example - -``` - def directory(request, url: str, name: str = "/", level: int = 0): - ls = request.get(url).get_json() - print(f"{' '*level}{name}") - for d in ls.directories: - directory(request, level + 1, d.name, d.url) - for f in ls.files: - print(f"{' '*(level+1)}{f.name}) - bytes = request.get(f.url) - # display byte stream: - # inline on terminal doesn't really make sense - - directory(request, "http://host.example.com/api/v1/datasets//contents/") -``` diff --git a/docs/Server/API/V1/relay.md b/docs/Server/API/V1/relay.md index bed54aa1db..0fa4d87593 100644 --- a/docs/Server/API/V1/relay.md +++ b/docs/Server/API/V1/relay.md @@ -1,37 +1,39 @@ # `POST /api/v1/relay/` -This API creates a dataset resource by reading data from a Relay server in two -steps. The provided URI represents a "manifest file" stored on the Relay server, -a JSON file (`application/json` MIME format) defining the original tarball name, -the MD5 hash value, optional metadata key/value pairs for the dataset, and a -URI from which the Pbench Server expects to `GET` an `application/octet-stream` -of a file created by `tar` and compressed with the `xz` program. - -Primarily this is expected to be a native Pbench Agent tarball with a specific -structure; however with the `server.archiveonly` metadata key the Pbench Server -can be used to archive and manage metadata for any tarball passed through a -Relay server. +This API creates a dataset resource by reading data from a Relay server. There +are two distinct steps involved: + +1. A `GET` on the provided URI returns a "manifest file" stored on the Relay +server. This is a JSON file (`application/json` MIME format) defining the +original tarball filename, the tarball's MD5 hash value, the URI of the +performance data tarball file, and optionally metadata key/value pairs to be +applied to the new dataset. +2. A `GET` on the Relay manifest file's `uri` field value returns an +`application/octet-stream` payload for a file created by `tar` and compressed +with the `xz` program, which will be stored by the Pbench Server as a +dataset. ## URI parameters `` string \ The Relay server URI of the tarball's manifest `application/json` file. This -must provide the following JSON keys: +JSON object must provide a set of parameter keys as defined below in +[Manifest file keys](#manifest-file-keys). + +## Manifest file keys For example, ```json { "uri": "https://relay.example.com/52adfdd3dbf2a87ed6c1c41a1ce278290064b0455f585149b3dadbe5a0b62f44", - "md5": "22a4bc5748b920c6ce271eb68f08d91c". - "name": "fio_rw_2018.02.01T22.40.57.tar.xz". + "md5": "22a4bc5748b920c6ce271eb68f08d91c", + "name": "fio_rw_2018.02.01T22.40.57.tar.xz", "access": "private", "metadata": ["server.origin:myrelay", "global.agent:cloud1"] } ``` -## Manifest file keys - `access`: [ `private` | `public` ] \ The desired initial access scope of the dataset. Select `public` to make the dataset accessible to all clients, or `private` to make the dataset accessible @@ -56,8 +58,6 @@ In particular the client can set any of: * `server.archiveonly`: [suppress indexing](../metadata.md#serverarchiveonly) * `server.deletion`: [default dataset expiration time](../metadata.md#serverdeletion). -For example, `?metadata=server.archiveonly:true,global.project:oidc` - `name`: The original tarball file name \ The string value must represent a legal filename with the compound type of `.tar.xz` representing a `tar` archive compressed with the `xz` program. @@ -86,7 +86,8 @@ The return is a serialized JSON object with status information. `200` **OK** \ Successful request. The dataset MD5 hash is identical to that of a dataset previously uploaded to the Pbench Server. This is assumed to be an identical -tarball. +tarball, and the secondary URI (the `uri` field in the Relay manifest file) +has not been consumed. `201` **CREATED** \ The tarball was successfully uploaded and the dataset has been created. @@ -112,8 +113,10 @@ a message, and optional JSON data provided by the system administrator. ## Response body -The `application/json` response body consists of a JSON object giving a detailed -message on success or failure: +The `application/json` response body consists of a JSON object containing a +`message` field. On failure this will describe the nature of the problem and +in some cases an `errors` array will provide details for cases where multiple +problems can occur. ```json { diff --git a/docs/Server/API/V1/upload.md b/docs/Server/API/V1/upload.md index 221c5dd9a4..f047f90cae 100644 --- a/docs/Server/API/V1/upload.md +++ b/docs/Server/API/V1/upload.md @@ -82,8 +82,10 @@ a message, and optional JSON data provided by the system administrator. ## Response body -The `application/json` response body consists of a JSON object giving a detailed -message on success or failure: +The `application/json` response body consists of a JSON object containing a +`message` field. On failure this will describe the nature of the problem and +in some cases an `errors` array will provide details for cases where multiple +problems can occur. ```json {