ChatBedrock doesn't support streaming when Bedrock guardrails are applied

[ardf]

Description

When using ChatBedrock with streaming enabled and Bedrock guardrails applied, the streaming functionality breaks. This appears to be due to differences in the response structure when guardrails are active.

Steps to Reproduce

1. Set up a ChatBedrock instance with streaming enabled.
2. Configure and apply Bedrock guardrails to the model.
3. Attempt to use the streaming functionality.

Current Behavior

The streaming functionality fails with a KeyError, specifically:

\venv\Lib\site-packages\langchain_aws\llms\bedrock.py", line 148, in _stream_response_to_generation_chunk
    |     "stop_sequence": stream_response["delta"]["stop_sequence"],
    |                      ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
    | KeyError: 'stop_sequence'

This occurs in the _stream_response_to_generation_chunk method of the llms\bedrock.py file.

Expected Behavior

The streaming functionality should work correctly regardless of whether guardrails are applied or not.

Additional Context

• The application works fine without guardrails and streaming responses are received as expected.
• The issue only occurs when Bedrock guardrails are applied to the ChatBedrock instance.

Environment

• langchain_aws version: 0.1.17
• Python version: 3.12.2
• Operating System: Windows 11

Possible Solution

The _stream_response_to_generation_chunk method in bedrock.py should be updated to handle cases where the response structure differs when guardrails are applied. This may involve checking for the existence of certain keys before accessing them, or handling different response structures based on whether guardrails are active.

[john-mihailescu]

I am also seeing this issue. In addition - ChatBedrock still does not support Guardrails configurations, which are necessary to properly do prompt attack filtering via input tagging and to filter responses asynchronously.
There is already a PR open here for this issue that several people are waiting for - #59

[ardf]

I am also seeing this issue. In addition - ChatBedrock still does not support Guardrails configurations, which are necessary to properly do prompt attack filtering via input tagging and to filter responses asynchronously.
There is already a PR open here for this issue that several people are waiting for - #59

Well, AFAIK, it does support bedrock guardrails, however streaming has to be disabled for the same. The problem occurs when streaming is enabled.

I've migrated to ChatBedrockConverse for my project, which will eventually replace ChatBedrock class.

[dillycone]

I am also seeing this issue. In addition - ChatBedrock still does not support Guardrails configurations, which are necessary to properly do prompt attack filtering via input tagging and to filter responses asynchronously.
There is already a PR open here for this issue that several people are waiting for - #59

Well, AFAIK, it does support bedrock guardrails, however streaming has to be disabled for the same. The problem occurs when streaming is enabled.

I've migrated to ChatBedrockConverse for my project, which will eventually replace ChatBedrock class.

Does ChatBedrockConverse fix it?

[ardf]

I am also seeing this issue. In addition - ChatBedrock still does not support Guardrails configurations, which are necessary to properly do prompt attack filtering via input tagging and to filter responses asynchronously.
There is already a PR open here for this issue that several people are waiting for - #59

Well, AFAIK, it does support bedrock guardrails, however streaming has to be disabled for the same. The problem occurs when streaming is enabled.

I've migrated to ChatBedrockConverse for my project, which will eventually replace ChatBedrock class.

Does ChatBedrockConverse fix it?

Yes, I'm able to stream the response using ChatBedrockConverse with guardrails configured.

[dillycone]

Thank you.

[rpgeddam]

is ChatBedrockConverse a drop in replacement for ChatBedrock?

[3coins]

Fixed by #177