Skip to content

Commit

Permalink
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs: api-reference: add config to open/read/dvcfs
Browse files Browse the repository at this point in the history
Fixes #4627
efiop committed Jun 15, 2023
1 parent c9c5c02 commit d8c99b1
Showing 3 changed files with 59 additions and 2 deletions.
2 changes: 2 additions & 0 deletions content/docs/api-reference/dvcfilesystem.md
Original file line number Diff line number Diff line change
@@ -29,6 +29,8 @@ The optional `rev` argument can be passed to open a filesystem from a certain
Git commit (any [revision](https://git-scm.com/docs/revisions) such as a branch
or a tag name, a commit hash, or an [experiment name]).

The optional `config` argument can be passed through to the DVC project.

[experiment name]: /doc/command-reference/exp/run#-n

## Opening a file
30 changes: 29 additions & 1 deletion content/docs/api-reference/open.md
Original file line number Diff line number Diff line change
@@ -8,7 +8,8 @@ def open(path: str,
rev: str = None,
remote: str = None,
mode: str = "r",
encoding: str = None)
encoding: str = None,
config: dict = None)
```

## Usage
@@ -84,11 +85,16 @@ call – no _context manager_ involved. Neither function utilizes disc space.
only be used in text mode. Defaults to `"utf-8"`. Mirrors the namesake
parameter in builtin `open()`.

- `config` - [config] dictionary to pass to the DVC project. This is merged with
the existing project config and can be used to, for example, provide
credentials to the `remote`.

[revision]: https://git-scm.com/docs/revisions
[experiment name]: /doc/command-reference/exp/run#-n
[dvc remote]: /doc/user-guide/data-management/remote-storage
[default remote]: /doc/command-reference/remote/default
[codec]: https://docs.python.org/3/library/codecs.html#standard-encodings
[config]: /doc/command-reference/config

## Exceptions

@@ -199,3 +205,25 @@ import dvc.api
with dvc.api.open('data/nlp/words_ru.txt', encoding='koi8_r') as f:
# ... Process Russian words
```
## Example: Specify credentials for your remote
See [remote modify](/doc/command-reference/remote/modify) for full list of
remote-specific config options.
```py
import dvc.api
config = {
'remote': {
'myremote': {
'access_key_id': 'mykey',
'secret_access_key': 'mysecretkey',
'session_token': 'mytoken',
},
},
}
with dvc.api.open('data', config=config) as f:
# ... Process data
```
29 changes: 28 additions & 1 deletion content/docs/api-reference/read.md
Original file line number Diff line number Diff line change
@@ -14,7 +14,8 @@ def read(path: str,
rev: str = None,
remote: str = None,
mode: str = "r",
encoding: str = None)
encoding: str = None,
config: dict = None)
```

## Usage
@@ -75,11 +76,16 @@ Python's [`open()`] built-in, which is used under the hood.
only be used in text mode. Defaults to `"utf-8"`. Mirrors the namesake
parameter in builtin `open()`.

- `config` - [config] dictionary to pass to the DVC project. This is merged with
the existing project config and can be used to, for example, provide
credentials to the `remote`.

[revision]: https://git-scm.com/docs/revisions
[experiment name]: /doc/command-reference/exp/run#-n
[dvc remote]: /doc/user-guide/data-management/remote-storage
[default remote]: /doc/command-reference/remote/default
[codec]: https://docs.python.org/3/library/codecs.html#standard-encodings
[config]: /doc/command-reference/config

## Exceptions

@@ -109,3 +115,24 @@ model = pickle.loads(data)
```

> We're using `'rb'` mode here for compatibility with `pickle.loads()`.
## Example: Specify credentials for your remote

See [remote modify](/doc/command-reference/remote/modify) for full list of
remote-specific config options.

```py
import dvc.api

config = {
'remote': {
'myremote': {
'access_key_id': 'mykey',
'secret_access_key': 'mysecretkey',
'session_token': 'mytoken',
},
},
}

modelpkl = dvc.api.read('model.pkl', config=config)
```

0 comments on commit d8c99b1

Please sign in to comment.