Azure OpenAI crashes without non-Azure API key

[mtakemoto]

• I have read and agree to the contributing guidelines.

Describe the bug
After configuring the AzureOpenAiChatPromptDriver, it won't start without also setting the "OPENAI_API_KEY" environment variable. This is a bit deceptive since (in theory) Azure OpenAI should support everything that the 1st party offering does.

To Reproduce
Steps to reproduce the behavior:

1. Initialize an AzureOpenAiChatPromptDriver, setting the api_key, model, azure_deployment and azure_endpoint
2. Initialize the rest according to the docs: https://docs.griptape.ai/latest/griptape-framework/structures/prompt-drivers/
3. Start the program
4. Observe a crash and the following stack trace:

Traceback (most recent call last):
  File "/workspaces/base-tooling/examples/griptape_demo.py", line 38, in <module>
    agent = Agent(
  File "<attrs generated init griptape.structures.agent.Agent>", line 15, in __init__
    _setattr('embedding_driver', __attr_factory_embedding_driver())
  File "/home/vscode/.cache/pypoetry/virtualenvs/examples-79_hlSy2-py3.10/lib/python3.10/site-packages/griptape/structures/structure.py", line 42, in <lambda>
    embedding_driver: BaseEmbeddingDriver = field(default=Factory(lambda: OpenAiEmbeddingDriver()), kw_only=True)
  File "<attrs generated init griptape.drivers.embedding.openai_embedding_driver.OpenAiEmbeddingDriver>", line 17, in __init__
    self.client = __attr_factory_client(self)
  File "/home/vscode/.cache/pypoetry/virtualenvs/examples-79_hlSy2-py3.10/lib/python3.10/site-packages/griptape/drivers/embedding/openai_embedding_driver.py", line 34, in <lambda>
    lambda self: openai.OpenAI(api_key=self.api_key, base_url=self.base_url, organization=self.organization),
  File "/home/vscode/.cache/pypoetry/virtualenvs/examples-79_hlSy2-py3.10/lib/python3.10/site-packages/openai/_client.py", line 92, in __init__
    raise OpenAIError(
openai.OpenAIError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable

Expected behavior
The embedding driver should use the Azure endpoint for embeddings and not have a dependency on environment variable

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: Windows 11
• Version 22621.2861
• Poetry: 1.7.1
• Container: mcr.microsoft.com/vscode/devcontainers/python:0-3.10-bullseye
• Docker for Windows: v24.0.6
• WSL: 2.0.9.0

Additional context
The WebScraper also has a hard dependency on the OpenAI tokenizer. Falls into the same category of making the library more flexible but probably worth opening a separate isue

[collindutter]

Hey @mtakemoto! This error is from the embedding_driver, not the prompt_driver that you have overridden.

See the note here regarding this behavior. In your case, you can provide the AzureOpenAiEmbeddingDriver like this:

import os
from griptape.structures import Agent
from griptape.drivers import AzureOpenAiChatPromptDriver, AzureOpenAiEmbeddingDriver

agent = Agent(
    prompt_driver=AzureOpenAiChatPromptDriver(
        api_key=os.environ["AZURE_OPENAI_API_KEY_1"],
        model="gpt-3.5-turbo-16k",
        azure_deployment=os.environ["AZURE_OPENAI_35_16k_DEPLOYMENT_ID"],
        azure_endpoint=os.environ["AZURE_OPENAI_API_BASE_1"],
    ),
    embedding_driver=AzureOpenAiEmbeddingDriver(
        api_key=os.environ["AZURE_OPENAI_API_KEY_1"],
        model="text-embedding-ada-002",
        azure_deployment=os.environ["AZURE_OPENAI_35_16k_DEPLOYMENT_ID"],
        azure_endpoint=os.environ["AZURE_OPENAI_API_BASE_1"],
    ),
)

[collindutter]

Hey @mtakemoto just checking in to see whether this has resolved your issue. Thanks!

[mtakemoto]

Hey @collindutter thanks for the reply here showing me how to override the embedding driver! However, I'd like to propose that the AzureOpenAiChatPromptDriver automatically initialize the associated Azure embedding driver for the following reasons:

1. Easier to get started (no override needed)
2. Getting an "API key missing" error for a service I haven't specified yet is confusing
3. Makes Griptape more vendor agnostic (by removing a hardcoded dependency on OpenAI)

More broadly, I'd like to propose that any cloud-based AI provider automatically initialize with their own embedding endpoint unless explicitly overridden (e.g. wanting to mix HuggingFace models with an OpenAI embedding provider).

Thanks again and please let me know what your thoughts are!

[collindutter]

Thank you for the well thought out suggestion, and totally agree with your points!

The topic of defaults is something we are actively working on in #242; we should have a proper solution in the coming weeks. The Prompt Driver will likely still remain decoupled from the Embedding Driver, but there will be a new construct to tie them together.

[collindutter]

We've gotten this issue a handful of times over the last few weeks, I am going to keep this issue open as the source of discussion.

[derekpai]

closed by #242