Exception - Structured Output - Vision Models (Llama 3.2)

[luisquintanilla]

Description

Using structured output with vision models like gpt-4o-mini works. I'd like to do the same for Llama-3.2-11B-Vision-Instruct from GitHub models. Currently it throws an exception.

Reproduction Steps

1. Clone this repo: https://github.com/lqdev/AIBookmarks/
2. Set modelName in Program.cs to Llama-3.2-11B-Vision-Instruct
3. Run the application

Working Azure AI Inference SDK Code

Replacing the inference loop with this code works

foreach(var path in filePaths)
{
    var file = await File.ReadAllBytesAsync(path);
    var messages = new List<ChatRequestMessage>
    {
        new ChatRequestSystemMessage(systemPrompt),
        new ChatRequestUserMessage(new ChatMessageImageContentItem(BinaryData.FromBytes(file), "image/jpeg")),
        new ChatRequestUserMessage("Extract the marked passages from the image")
    };

    var response = await aoaiClient.CompleteAsync(new ChatCompletionsOptions{
        Messages = messages,
        Model = modelName
    });

    Console.WriteLine(response.Value.Content);
}

Expected Output

The marked passages are:

* If we try ten experiments and none of them work, we have a choice. We can take it personally, and think of ourselves as a failure and question our ability to solve the problems that come our way.
* We are required to believe in something that doesn't exist in order to allow it to come into being.
* Faith is rewarded, perhaps even more than talent or ability.

These passages are found on pages 278 and 279.

Similarly, using MEAI without structured output

foreach(var path in filePaths)
{
    var file = await File.ReadAllBytesAsync(path);
    var messages = new List<ChatMessage>
    {
        new ChatMessage(ChatRole.System, systemPrompt),
        new ChatMessage(ChatRole.User, new AIContent[] {
            new ImageContent(file, "image/jpeg"),
            new TextContent("Extract the marked passages from the image"),
        })
    };

    var response = await meaiClient.CompleteAsync(messages, options: new ChatOptions {Temperature = 0.1f});

    Console.WriteLine(response.Message);
}

Expected behavior

Response returns LLM response as specified type parameter, even if it's not supported by the model.

See generic completion API for reference.

Actual behavior

Exception is thrown.

Unhandled exception. Azure.RequestFailedException: An error occurred during the request processing: All connection attempts failed | Type: ConnectError | Traceback (most recent call last):||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions||    yield||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_transports/default.py", line 366, in handle_async_request||    resp = await self._pool.handle_async_request(req)||           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 216, in handle_async_request||    raise exc from None||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 196, in handle_async_request||    response = await connection.handle_async_request(||               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_async/connection.py", line 99, in handle_async_request||    raise exc||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_async/connection.py", line 76, in handle_async_request||    stream = await self._connect(request)||             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_async/connection.py", line 122, in _connect||    stream = await self._network_backend.connect_tcp(**kwargs)||             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_backends/auto.py", line 30, in connect_tcp||    return await self._backend.connect_tcp(||           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 115, in connect_tcp||    with map_exceptions(exc_map):||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/contextlib.py", line 158, in __exit__||    self.gen.throw(typ, value, traceback)||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions||    raise to_exc(exc) from exc||httpcore.ConnectError: All connection attempts failed||||The above exception was the direct cause of the following exception:||||Traceback (most recent call last):||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__||    await self.app(scope, receive, _send)||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/billingproxy/main.py", line 411, in __call__||    return await handler_task||           ^^^^^^^^^^^^^^^^^^||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py", line 189, in __call__||    with collapse_excgroups():||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/contextlib.py", line 158, in __exit__||    self.gen.throw(typ, value, traceback)||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/_utils.py", line 93, in collapse_excgroups||    raise exc||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__||    response = await self.dispatch_func(request, call_next)||               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/billingproxy/main.py", line 377, in dispatch||    response = await call_next(request)||               ^^^^^^^^^^^^^^^^^^^^^^^^||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py", line 165, in call_next||    raise app_exc||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py", line 151, in coro||    await self.app(scope, receive_or_disconnect, send_no_error)||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py", line 189, in __call__||    with collapse_excgroups():||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/contextlib.py", line 158, in __exit__||    self.gen.throw(typ, value, traceback)||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/_utils.py", line 93, in collapse_excgroups||    raise exc||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__||    response = await self.dispatch_func(request, call_next)||               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/billingproxy/main.py", line 353, in dispatch||    response = await call_next(request)||               ^^^^^^^^^^^^^^^^^^^^^^^^||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py", line 165, in call_next||    raise app_exc||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py", line 151, in coro||    await self.app(scope, receive_or_disconnect, send_no_error)||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__||    await self.app(scope, receive, send)||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__||    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app||    raise exc||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app||    await app(scope, receive, sender)||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__||    await self.middleware_stack(scope, receive, send)||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/routing.py", line 776, in app||    await route.handle(scope, receive, send)||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle||    await self.app(scope, receive, send)||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/routing.py", line 77, in app||    await wrap_app_handling_exceptions(app, request)(scope, receive, send)||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app||    raise exc||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app||    await app(scope, receive, sender)||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/routing.py", line 72, in app||    response = await func(request)||               ^^^^^^^^^^^^^^^^^^^||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app||    raw_response = await run_endpoint_function(||                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function||    return await dependant.call(**values)||           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/billingproxy/apis/vllm/chat.py", line 322, in chat||    response = await downstream_client_getter().send(downstream_request)||          
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_client.py", line 1617, in send||    response = await self._send_handling_auth(||               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_client.py", line 1645, in _send_handling_auth||    response = await self._send_handling_redirects(||               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects||    response = await self._send_single_request(request)||               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_client.py", line 1719, in _send_single_request||    response = await transport.handle_async_request(request)||               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_transports/default.py", line 365, in handle_async_request||    with map_httpcore_exceptions():||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/contextlib.py", line 158, in __exit__||    self.gen.throw(typ, value, traceback)||  File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_transports/default.py", line 83, in map_httpcore_exceptions||    raise mapped_exc(message) from exc||httpx.ConnectError: All connection attempts failed||
Status: 500 (Internal Server Error)
ErrorCode: Internal Server Error

Content:
{"error":{"code":"Internal Server Error","message":"An error occurred during the request processing: All connection attempts failed | Type: ConnectError | Traceback (most recent call last):||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_transports/default.py\", line 66, in map_httpcore_exceptions||    yield||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_transports/default.py\", line 366, in handle_async_request||    resp = await self._pool.handle_async_request(req)||           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_async/connection_pool.py\", line 216, in handle_async_request||    raise exc from None||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_async/connection_pool.py\", line 196, in handle_async_request||    response = await connection.handle_async_request(||               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_async/connection.py\", line 99, in handle_async_request||    raise exc||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_async/connection.py\", line 76, in handle_async_request||    stream = await self._connect(request)||             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_async/connection.py\", line 122, in _connect||    stream = await self._network_backend.connect_tcp(**kwargs)||             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_backends/auto.py\", line 30, in connect_tcp||    return await self._backend.connect_tcp(||           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_backends/anyio.py\", line 115, in connect_tcp||    with map_exceptions(exc_map):||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/contextlib.py\", line 158, in __exit__||    self.gen.throw(typ, value, traceback)||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_exceptions.py\", line 14, in map_exceptions||    raise to_exc(exc) from exc||httpcore.ConnectError: All connection attempts failed||||The above exception was the direct cause of the following exception:||||Traceback (most recent call last):||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/errors.py\", line 164, in __call__||    await self.app(scope, receive, _send)||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/billingproxy/main.py\", line 411, in __call__||    return await handler_task||           ^^^^^^^^^^^^^^^^^^||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py\", line 189, in __call__||    with collapse_excgroups():||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/contextlib.py\", line 158, in __exit__||    self.gen.throw(typ, value, traceback)||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/_utils.py\", line 93, in collapse_excgroups||    raise exc||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py\", line 191, in __call__||    response = await self.dispatch_func(request, call_next)||               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/billingproxy/main.py\", line 377, in dispatch||    response = await call_next(request)||               ^^^^^^^^^^^^^^^^^^^^^^^^||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py\", line 165, in call_next||    raise app_exc||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py\", line 151, in coro||    await self.app(scope, receive_or_disconnect, send_no_error)||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py\", line 189, in __call__||    with collapse_excgroups():||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/contextlib.py\", line 158, in __exit__||    self.gen.throw(typ, value, traceback)||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/_utils.py\", line 93, in collapse_excgroups||    raise exc||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py\", line 191, in __call__||    response = await self.dispatch_func(request, call_next)||               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/billingproxy/main.py\", line 353, in dispatch||    response = await call_next(request)||               ^^^^^^^^^^^^^^^^^^^^^^^^||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py\", line 165, in call_next||    raise app_exc||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py\", line 151, in coro||    await self.app(scope, receive_or_disconnect, send_no_error)||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/cors.py\", line 85, in __call__||    await self.app(scope, receive, send)||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/exceptions.py\", line 65, in __call__||    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/_exception_handler.py\", line 64, in wrapped_app||    raise exc||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/_exception_handler.py\", line 53, in wrapped_app||    await app(scope, receive, sender)||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/routing.py\", line 756, in __call__||    await self.middleware_stack(scope, receive, send)||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/routing.py\", line 776, in app||    await route.handle(scope, receive, send)||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/routing.py\", line 297, in handle||    await self.app(scope, receive, send)||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/routing.py\", line 77, in app||    await wrap_app_handling_exceptions(app, request)(scope, receive, send)||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/_exception_handler.py\", line 64, in wrapped_app||    raise exc||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/_exception_handler.py\", line 53, in wrapped_app||    await app(scope, receive, sender)||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/routing.py\", line 72, in app||    response = await func(request)||               ^^^^^^^^^^^^^^^^^^^||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/fastapi/routing.py\", line 278, in app||    raw_response = await run_endpoint_function(||                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/fastapi/routing.py\", line 191, in run_endpoint_function||    return await dependant.call(**values)||           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/billingproxy/apis/vllm/chat.py\", line 322, in chat||    response = await downstream_client_getter().send(downstream_request)||               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_client.py\", line 1617, in send||    response = await self._send_handling_auth(||               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_client.py\", line 1645, in _send_handling_auth||    response = await self._send_handling_redirects(||               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_client.py\", line 1682, in _send_handling_redirects||    response = await self._send_single_request(request)||               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_client.py\", line 1719, in _send_single_request||    response = await transport.handle_async_request(request)||               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_transports/default.py\", line 365, in handle_async_request||    with map_httpcore_exceptions():||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/contextlib.py\", line 158, in __exit__||    self.gen.throw(typ, value, traceback)||  File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_transports/default.py\", line 83, in map_httpcore_exceptions||    raise mapped_exc(message) from exc||httpx.ConnectError: All connection attempts failed||","status":500}}

Headers:
Date: Mon, 04 Nov 2024 23:54:21 GMT
Transfer-Encoding: chunked
Connection: keep-alive
azureml-model-group: REDACTED
request-context: REDACTED
x-ms-rai-invoked: REDACTED
X-Request-ID: REDACTED
ms-azureml-model-error-reason: REDACTED
ms-azureml-model-error-statuscode: REDACTED
x-ms-client-request-id: 09647581-2868-4eef-b19d-be48a4c54033
azureml-model-deployment: REDACTED
azureml-model-session: REDACTED
Strict-Transport-Security: REDACTED
X-Content-Type-Options: REDACTED
x-aml-cluster: REDACTED
x-request-time: REDACTED
Content-Type: application/json

   at Azure.Core.HttpPipelineExtensions.ProcessMessageAsync(HttpPipeline pipeline, HttpMessage message, RequestContext requestContext, CancellationToken cancellationToken)
   at Azure.AI.Inference.ChatCompletionsClient.CompleteAsync(RequestContent content, String extraParams, RequestContext context)
   at Azure.AI.Inference.ChatCompletionsClient.CompleteAsync(ChatCompletionsOptions chatCompletionsOptions, CancellationToken cancellationToken)
   at Microsoft.Extensions.AI.AzureAIInferenceChatClient.CompleteAsync(IList`1 chatMessages, ChatOptions options, CancellationToken cancellationToken)
   at Microsoft.Extensions.AI.ChatClientStructuredOutputExtensions.CompleteAsync[T](IChatClient chatClient, IList`1 chatMessages, JsonSerializerOptions serializerOptions, ChatOptions options, Nullable`1 useNativeJsonSchema, CancellationToken cancellationToken)
   at Program.<Main>$(String[] args) in C:\Dev\AIBookmarks\Program.cs:line 76
   at Program.<Main>(String[] args)

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

[stephentoub]

cc: @SteveSandersonMS

[SteveSandersonMS]

I'm not at all sure what's going on here. The error message seems to be All connection attempts failed which doesn't sound like it has anything to do with whether it uses structured output or not.

The difference to the request that structured output makes is:

1. Sets the requested response format to JSON
2. Adds a futher message saying (roughly) Reply as a JSON object matching this schema: {...}

If you have any idea why MaaS replies with this strange message (or if you managed to spot any more useful diagnostic message in the output elsewhere) please let us know.

[luisquintanilla]

It's possible there's an error connecting to the services.

The reason I ruled that out though was, both with the Azure AI Inference SDK as well as MEAI, when structured output is not used, they both return a response.

Also, using the same service, GPT-4o-mini returns a response with structured output which is what I'd expect since it supports it.