Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add page about Studio REST API #4681

Merged
merged 27 commits into from
Oct 13, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
5260c54
add page about REST API
aguschin Jul 6, 2023
c031684
link to adding credentials to studio
aguschin Jul 10, 2023
d10fb04
make link relative
aguschin Jul 10, 2023
9d7f82b
fix api handler name
aguschin Jul 10, 2023
0c7c289
add page about studio python api
aguschin Jul 12, 2023
7d1fb17
fix format of request description
aguschin Jul 12, 2023
832efae
Merge branch 'main' into studio/api
aguschin Jul 12, 2023
ba1fe76
note about latest version
aguschin Jul 12, 2023
5059499
Merge branch 'studio/api' of github.com:iterative/dvc.org into studio…
aguschin Jul 12, 2023
3e22bfc
explain monorepo case
aguschin Jul 12, 2023
4c971c6
add link to apis
aguschin Jul 13, 2023
d777c0a
Update content/docs/studio/rest-api.md
aguschin Jul 13, 2023
2f46406
link from dvc api; add with to download_model
aguschin Jul 13, 2023
ef0c38c
Merge branch 'studio/api' of github.com:iterative/dvc.org into studio…
aguschin Jul 13, 2023
0855839
drop studio python api
Jul 24, 2023
7134202
Merge branch 'main' into studio/api
Jul 24, 2023
a4f66a2
minor edits to studio api
Jul 24, 2023
b291eee
add stage
Jul 25, 2023
f68bf78
studio api: add expiration time
Aug 1, 2023
d0139df
Merge branch 'main' into studio/api
Aug 31, 2023
a6493f4
studio api: add python example
Aug 31, 2023
87f9402
Merge branch 'main' into studio/api
Sep 26, 2023
bb54bda
use model registry abbr
Sep 26, 2023
e2aa45a
Merge branch 'main' into studio/api
Sep 29, 2023
b87e2f7
reference studio rest api in artifacts get and reframe around model r…
Sep 29, 2023
166599d
reference studio rest api in api.artifacts_show
Sep 29, 2023
ef27fc3
reword artifacts get fallback
Oct 2, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 16 additions & 2 deletions content/docs/api-reference/artifacts_show.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# dvc.api.artifacts_show()

Get the path and Git revision for an <abbr>artifact</abbr> tracked in a
<abbr>DVC repository</abbr>.
Get the path and Git revision for an <abbr>artifact</abbr> tracked in the
<abbr>model registry</abbr>.

```py
def artifacts_show(
Expand All @@ -12,6 +12,15 @@ def artifacts_show(
) -> Dict[str, str]:
```

<admon type="tip">

If you have a Studio project configured with your [remote storage credentials],
you may also use the Studio [REST API] to programmatically access artifacts. It
does not require the client to have any credentials other than the Studio
[access token] and does not require DVC to be installed.

</admon>

## Usage:

```py:
Expand Down Expand Up @@ -104,3 +113,8 @@ fs.get_file(artifact['path'], os.path.basename(artifact['path']))
This example uses the returned path and Git revision in conjunction with
`dvc.api.DVCFileSystem` to download the artifact to the current working
directory.

[rest api]: /doc/studio/rest-api
[access token]: /doc/studio/user-guide/account-management#studio-access-token
[remote storage credentials]:
/doc/studio/user-guide/experiments/configure-a-project#data-remotes--cloud-storage-credentials
43 changes: 17 additions & 26 deletions content/docs/command-reference/artifacts/get.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
## artifacts get

Download an <abbr>artifact</abbr> tracked in a DVC project into the current
working directory.
Download an <abbr>artifact</abbr> tracked in the <abbr>model registry</abbr>
into the current working directory (with optional Studio support).

## Synopsis

Expand All @@ -20,22 +20,10 @@ positional arguments:

## Description

Provides a way to download artifacts tracked in a DVC project. Unlike `dvc get`,
`dvc artifacts get` supports downloading an artifact by name, rather than by
path. Likewise, `dvc artifacts get` supports downloading a registered artifact
version or stage, instead of requiring a specified Git revision.

`dvc artifacts get` also supports downloading artifacts both from the
<abbr>model registry</abbr> and from DVC remotes.

<admon type="tip">

Downloading an artifact from the <abbr>model registry</abbr> only requires a
valid Studio
[access token](/doc/studio/user-guide/account-management#studio-access-token).
It does not require the client to have DVC remote credentials.

</admon>
Provides a way to download artifacts tracked in the model registry. Unlike
`dvc get`, `dvc artifacts get` supports downloading an artifact by name, rather
than by path. Likewise, `dvc artifacts get` supports downloading a registered
artifact version or stage, instead of requiring a specified Git revision.

The `url` argument specifies the address of the DVC or Git repository containing
the artifact. Both HTTP and SSH protocols are supported (e.g.
Expand All @@ -48,14 +36,12 @@ of the DVC repository. Artifacts declared in other `dvc.yaml` files should be
addressed in the form `path/to/dvc.yaml:artifact_name` or
`path/to:artifact_name` (where `dvc.yaml` is omitted).

<admon type="info">

`dvc artifacts get` will first try to download artifacts via the <abbr>model
registry</abbr>. If you do not have a valid Studio token, or the artifact is not
tracked in the model registry, DVC will fall back to downloading the artifact
from the project's default DVC remote.

</admon>
`dvc artifacts get` will first try to download artifacts via the Studio [REST
API]. Downloading an artifact using the Studio REST API only requires a valid
Studio [access token] and a Studio project configured with your [remote storage
credentials]. It does not require the client to have those credentials. If you
do not have a valid Studio token, or the artifact is not tracked in the model
registry, DVC will fall back to its typical method to get files (see `dvc get`).

## Options

Expand Down Expand Up @@ -132,3 +118,8 @@ case, the artifact is defined in `results/dvc.yaml` so we must include the path
to the `dvc.yaml` file when addressing the artifact. Since we do not specify
`--rev` or `--stage`, `dvc artifacts get` will download the latest version of
the artifact by default.

[rest api]: /doc/studio/rest-api
[access token]: /doc/studio/user-guide/account-management#studio-access-token
[remote storage credentials]:
/doc/studio/user-guide/experiments/configure-a-project#data-remotes--cloud-storage-credentials
3 changes: 2 additions & 1 deletion content/docs/sidebar.json
Original file line number Diff line number Diff line change
Expand Up @@ -755,7 +755,8 @@
]
}
]
}
},
"rest-api"
]
},
{
Expand Down
115 changes: 115 additions & 0 deletions content/docs/studio/rest-api.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
# REST API

The purpose of Studio REST API is to give programmatic access to information in
Studio and executing actions in it.

The API is hosted under the `/api` route on the Studio server:
https://studio.iterative.ai/api or https://your-domain/api in case of
[self-hosted Studio](/doc/studio/self-hosting/installation).

To use API, you need to generate
[Studio access token](/doc/studio/user-guide/account-management#studio-access-token).

## Download model

Get signed URL to download the model binaries for a model from the <abbr>model
registry</abbr>. Requires the model to be stored with DVC with S3 or Azure
[remote]. Note, that you need to
[set up remote cloud credentials](/doc/studio/user-guide/account-management#cloud-credentials)
for Studio to have rights to the signed URLs. The signed URLs expire after one
hour.

```yaml
Endpoint: api/model-registry/get-download-uris
HTTP Method: GET
```

### Request

| param | desc | type | required | example value |
| ------- | ------------- | ------ | -------- | ---------------------------------- |
| repo | Git repo URL | string | true | iterative/demo-bank-customer-churn |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
| repo | Git repo URL | string | true | iterative/demo-bank-customer-churn |
| repo | Git repo URL | string | true | [email protected]:iterative/demo-bank-customer-churn |

Does this also support a Studio project ID per https://github.com/iterative/studio/pull/6338#issuecomment-1596468450? My feeling is it still might be useful since users have to do a lookup either way and this requires less escaping than a full repo url.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not rn. But we can add it in some future. Adding this as an item to "out of scope of phase I" https://github.com/iterative/studio/issues/5177

| name | Model name | string | true | randomforest-model |
| version | Model version | string | false | v2.0.0 |
dberenbaum marked this conversation as resolved.
Show resolved Hide resolved
| stage | Model stage | string | false | prod |

Only one of stage or version is allowed. If no version or stage is specified,
the latest version is returned.

When your model is annotated in non-root [`dvc.yaml`] file (typical for monorepo
case), model name will be constructed from two parts separated by colon:
`path/to/dvc/yaml:model_name`. For example, take a look at this
[model from example-get-started-experiments repo](https://studio.iterative.ai/user/aguschin/models/VtQdva13kMSPsN_N8004aQ==/pool-segmentation/v1.0.1).
Its full name that you need to use in API is `results/train:pool-segmentation`.

| header | desc | example value |
| ------------- | --------------- | ------------- |
| Authorization | Header for auth | token abc123 |

### Response

Response is a JSON-encoded dict. If the request was successful, keys will be
paths to files inside the repo, and values will be signed URLs you can query to
actually download the model.

### Example

First, you need your [Studio access token]. For this example, we set it in the
`DVC_STUDIO_TOKEN` environment variable:

```sh
export DVC_STUDIO_TOKEN=<TOKEN>
```

<toggle>

<tab title="Python">

```python
import json
import os
import requests


url = "https://studio.iterative.ai/api/model-registry/get-download-uris"
token = os.environ["DVC_STUDIO_TOKEN"]
headers = {"Authorization": f"token {token}"}
params = {
"repo": "[email protected]:iterative/demo-bank-customer-churn.git",
"name": "randomforest-model",
"version": "v2.0.0"
}

response = requests.get(url, headers=headers, params=params)
for rel_path, obj_url in json.loads(response.content).items():
obj = requests.get(obj_url)
...
```

</tab>

<tab title="CLI">

```sh
$ curl "https://studio.iterative.ai/api/model-registry/[email protected]:iterative/demo-bank-customer-churn.git&name=randomforest-model&version=v2.0.0" --header "Authorization:token ${DVC_STUDIO_TOKEN}"

{
dberenbaum marked this conversation as resolved.
Show resolved Hide resolved
".mlem/model/clf-model": "https://sandbox-datasets-iterative.s3.amazonaws.com/bank-customer-churn/86/bd02376ac675568ba2fac566169ef9?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAU7UXIWDIQFPCO76Q%2F20230706%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230706T134619Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=6807259ddd1f4448ed1e3c5d4503039884f7779381ee556175096b0a884ba1a6"
}
```

</tab>

<admon type="warn">

Running this example will fail because it requires that you have [set up remote
cloud credentials] in Studio to where the model is stored.

</admon>

[remote]: /doc/user-guide/data-management/remote-storage
[`dvc.yaml`]: /doc/user-guide/project-structure/dvcyaml-files
[Studio access token]:
/doc/studio/user-guide/account-management#studio-access-token
[set up remote cloud credentials]:
/doc/studio/user-guide/account-management#cloud-credentials
3 changes: 2 additions & 1 deletion content/docs/studio/user-guide/account-and-billing/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,8 @@ page.
### Studio access token

Iterative Studio uses access tokens to authorize [DVC] and [DVCLive] to send
experiment updates.
experiment updates, and to authenticate you in Studio
[REST API](/doc/studio/rest-api).

In the `Studio access token` section of your [Profile] page, you can generate a
new token as well as regenerate (replace) or delete your access token.
Expand Down
20 changes: 17 additions & 3 deletions content/docs/studio/user-guide/model-registry/use-models.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,16 +7,30 @@ capabilities.
## Download models

If your model file is DVC-tracked, you can download any of its registered
versions using `dvc get`.
versions using Studio [REST API], `dvc get`, or DVC [Python API].

### Looking up the right command in Studio
### Using Studio API

Using Studio API will require you to add your remote storage secrets to
respective projects but downloading models can be easier than DVC since you only
need to have the Studio API token. You do not need direct access to your remote
storage, and you do not need to install DVC. Studio APIs rely on signing urls,
which limits the number of remotes they're working with. You can learn more on
[REST API] pages.

[REST API]: /doc/studio/rest-api
[Python API]: /doc/api-reference

### Using `dvc get`

#### Looking up the right command in Studio

The `Get the model file` section of a model's details page contains a `dvc get`
command ready to copy and use.

![](/img/mr-dvc-get.png)

### Constructing the command manually
#### Constructing the command manually

You can also construct this command manually using this template:

Expand Down
Loading