Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python config API docs #22951

Open
wants to merge 2 commits into
base: gh-pages
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 72 additions & 1 deletion docs/genai/api/python.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,65 @@ pip install onnxruntime-genai
import onnxruntime_genai
```

## Config class

If the config needs to be modified at runtime to change providers/set provider options, this object can be first created and used, then a model can be created from this config object. If no runtime config changes are needed, a model can be directly created from the model_folder.

### Load a config

```python
onnxruntime_genai.Config(model_folder: str) -> onnxruntime_genai.Config
```
#### Parameters

- `model_folder`: Location of model and configuration on disk

#### Returns

`onnxruntime_genai.Config`

### Add a provider to the list

If the provider isn't already in the list of providers, this adds it to the end of the onnxruntime provider priority list.

```python
onnxruntime_genai.Config.append_provider(provider_name: str)
```

#### Parameters

- `provider_name`: (Required) The provider to set the option on (for example, "dml" or "cuda")

### Clear list of providers

Clears the list of providers. This is the only way to remove existing providers to enforce using only a specific provider.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once the list is cleared, what provider is used? CPU? Can you state that?


```python
onnxruntime_genai.Config.clear_providers()
```

Example to override the config and only use the dml provider:
```python
config=onnxruntime_genai.Config("model_path")
config.clear_providers()
config.append_provider("dml")
model=onnxruntime_genai.Model(config)
```

### Set an option for a provider

Set an option for a provider. If the provider is not already in the list, it will be added at the end (automatically calls append_provider internally).

```python
onnxruntime_genai.Config.set_provider_option(provider_name: str, option_name: str, option_value: str)
```

#### Parameters
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add an example of setting a new provider, as that is the scenario we are addressing with this change


- `provider_name`: (Required) The provider to set the option on (for example, "dml" or "cuda")
- `option_name`: (Required) Name of the option
- `option_value`: (Required) Value of the option

## Model class

### Load a model
Expand All @@ -46,6 +105,18 @@ onnxruntime_genai.Model(model_folder: str) -> onnxruntime_genai.Model

`onnxruntime_genai.Model`

### Load a model from a config

Loads the ONNX model(s) from a config object

```python
onnxruntime_genai.Model(config: config) -> onnxruntime_genai.Model
```

#### Parameters

- `config`: Config object previously created through onnxruntime_genai.Config

### Generate method

```python
Expand Down Expand Up @@ -369,4 +440,4 @@ onnxruntime_genai.Generator(adapters: Generators::Adapters, adapter: str) -> Non

#### Return value

None
None
Loading