Bedrock invoke_model expects alternating roles

[ssg-kstewart]

I am attempting to use ChatBedrock with an Anthropic Claude Sonnet model (anthropic.claude-3-sonnet-20240229-v1:0) in an implementation of LangGraph's collaborative multi-agent system. I am receiving an error as shown in a below stack trace where there is an expectation that roles must alternate. Messages passed in to _prepare_input_and_invoke are structured as so (with the message content redacted):

[
  {
    "role": "user",
    "content": "REDACTED"
  },
  {
    "role": "assistant",
    "content": "REDACTED"
  },
  {
    "role": "assistant",
    "content": "REDACTED"
  }
]

I assume this is expected behavior in regards to the multi-agent system and see similar message structure when using ChatOpenAI in place of ChatBedrock where there are multiple messages of type "ai" in succession.

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/langchain_aws/llms/bedrock.py", line 628, in _prepare_input_and_invoke
    response = self.client.invoke_model(**request_options)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 565, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 1021, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "assistant" roles in a row

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/routers/spel.py", line 28, in spel_invoke
    content = chain.handle_query(message.input)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/services/ai_components.py", line 289, in handle_query
    return self.graph.invoke(
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/__init__.py", line 1333, in invoke
    for chunk in self.stream(
  File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/__init__.py", line 876, in stream
    _panic_or_proceed(done, inflight, step)
  File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/__init__.py", line 1422, in _panic_or_proceed
    raise exc
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/retry.py", line 66, in run_with_retry
    task.proc.invoke(task.input, task.config)
  File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 2399, in invoke
    input = step.invoke(
            ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/langgraph/utils.py", line 95, in invoke
    ret = context.run(self.func, input, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/services/ai_components.py", line 315, in agent_node
    result = agent.invoke(state)
             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 2399, in invoke
    input = step.invoke(
            ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 170, in invoke
    self.generate_prompt(
  File "/usr/local/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 599, in generate_prompt
    return self.generate(prompt_messages, stop=stop, callbacks=callbacks, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 456, in generate
    raise e
  File "/usr/local/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 446, in generate
    self._generate_with_cache(
  File "/usr/local/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 671, in _generate_with_cache
    result = self._generate(
             ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/langchain_aws/chat_models/bedrock.py", line 440, in _generate
    completion, llm_output = self._prepare_input_and_invoke(
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/langchain_aws/llms/bedrock.py", line 635, in _prepare_input_and_invoke
    raise ValueError(f"Error raised by bedrock service: {e}")
ValueError: Error raised by bedrock service: An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "assistant" roles in a row

[ssg-kstewart]

For additional context, it appears as though the tool calls are not occurring successfully as the response body results in a IndexError here. Messages with the response body injected appear as shown:

[
  {
    "role": "user",
    "content": "REDACTED"
  },
  {
    "role": "assistant",
    "content": "REDACTED"
  },
  {
    "role": "user",
    "content": "RESPONSEBODY: {'id': 'msg_bdrk_018qmbt8JrxDnLUhEY7XbCtd', 'type': 'message', 'role': 'assistant', 'model': 'claude-3-sonnet-20240229', 'content': [], 'stop_reason': 'end_turn', 'stop_sequence': None, 'usage': {'input_tokens': 531, 'output_tokens': 3}}"
  },
  {
    "role": "assistant",
    "content": "REDACTED"
  },
  {
    "role": "user",
    "content": "RESPONSEBODY: {'id': 'msg_bdrk_01Ww79eCr8Bz1ZsSyrkgbSUH', 'type': 'message', 'role': 'assistant', 'model': 'claude-3-sonnet-20240229', 'content': [], 'stop_reason': 'end_turn', 'stop_sequence': None, 'usage': {'input_tokens': 796, 'output_tokens': 3}}"
  }
]

EDIT:
Note, this is after forcing an alternating message pattern using the following logic. Without alternating, I run into the same issue requiring the alternation. Ultimately, it looks as though tools are not actually being called successfully.

for i in range(1, len(messages)):
    if messages[i]['role'] == messages[i - 1]['role']:
        messages[i]['role'] = 'assistant' if messages[i - 1]['role'] == 'user' else 'user'

[ssg-kstewart]

Attached is additional debug information including a sorted diff of two tool calls, one from the GPT-4o model and one from the Bedrock Anthropic implementation. The structure differences appear to be related to this PR which I am currently working from.

bedrock-tool-call-sorted.json
gpt-tool-call-sorted.json
sorted-diff.json

{                                                               {
  "generations": [                                                "generations": [
    [                                                               [
      {                                                               {
        "generation_info": null,                              |         "generation_info": {
                                                              >           "finish_reason": "tool_calls",
                                                              >           "logprobs": null
                                                              >         },
        "message": {                                                    "message": {
          "id": [                                                         "id": [
            "langchain",                                                    "langchain",
            "schema",                                                       "schema",
            "messages",                                                     "messages",
            "AIMessage"                                                     "AIMessage"
          ],                                                              ],
          "kwargs": {                                                     "kwargs": {
            "additional_kwargs": {                                          "additional_kwargs": {
              "model_id": "anthropic.claude-3-sonnet-20240229 <
              "stop_reason": "tool_use",                      <
              "tool_calls": [                                                 "tool_calls": [
                {                                                               {
                  "function": {                                                   "function": {
                    "arguments": {                            |                     "arguments": "{\"entity_name\":\"Person\"
                      "entity_name": "Person",                |                     "name": "test_calc_wrapper"
                      "expression": "firstName != null"       |                   },
                    },                                        |                   "id": "call_eszDySpx0r99Rq9OVuupxHiF",
                    "name": "test_calc_wrapper",              |                   "type": "function"
                    "type": "function"                        <
                  }                                           <
                }                                                               }
              ],                                              |               ]
              "usage": {                                      <
                "completion_tokens": 148,                     <
                "prompt_tokens": 362,                         <
                "total_tokens": 510                           <
              }                                               <
            },                                                              },
            "content": "REDACTED",                            |             "content": "",
            "id": "run-67445d61-86f4-49cf-9de7-031f35af3d5b-0 |             "id": "run-58754c7e-8b91-4b37-9b7e-cabca937413c-0
            "invalid_tool_calls": [],                                       "invalid_tool_calls": [],
            "response_metadata": {                                          "response_metadata": {
              "model_id": "anthropic.claude-3-sonnet-20240229 |               "finish_reason": "tool_calls",
              "stop_reason": "tool_use",                      |               "logprobs": null,
              "tool_calls": [                                 |               "model_name": "gpt-4o",
                {                                             |               "system_fingerprint": "fp_319be4768e",
                  "function": {                               |               "token_usage": {
                    "arguments": {                            |                 "completion_tokens": 23,
                      "entity_name": "Person",                |                 "prompt_tokens": 14037,
                      "expression": "firstName != null"       |                 "total_tokens": 14060
                    },                                        <
                    "name": "test_calc_wrapper",              <
                    "type": "function"                        <
                  }                                           <
                }                                             <
              ],                                              <
              "usage": {                                      <
                "completion_tokens": 148,                     <
                "prompt_tokens": 362,                         <
                "total_tokens": 510                           <
              }                                                               }
            },                                                              },
            "tool_calls": [],                                 |             "tool_calls": [
            "type": "ai"                                      |               {
                                                              >                 "args": {
                                                              >                   "entity_name": "Person",
                                                              >                   "expression": "firstName != null"
                                                              >                 },
                                                              >                 "id": "call_eszDySpx0r99Rq9OVuupxHiF",
                                                              >                 "name": "test_calc_wrapper"
                                                              >               }
                                                              >             ],
                                                              >             "type": "ai",
                                                              >             "usage_metadata": {
                                                              >               "input_tokens": 14037,
                                                              >               "output_tokens": 23,
                                                              >               "total_tokens": 14060
                                                              >             }
          },                                                              },
          "lc": 1,                                                        "lc": 1,
          "type": "constructor"                                           "type": "constructor"
        },                                                              },
        "text": "REDACTED",                                   |         "text": "",
        "type": "ChatGeneration"                                        "type": "ChatGeneration"
      }                                                               }
    ]                                                               ]
  ],                                                              ],
  "llm_output": {                                                 "llm_output": {
    "model_id": "anthropic.claude-3-sonnet-20240229-v1:0",    |     "model_name": "gpt-4o",
    "stop_reason": "tool_use",                                |     "system_fingerprint": "fp_319be4768e",
    "tool_calls": [                                           |     "token_usage": {
      {                                                       |       "completion_tokens": 23,
        "function": {                                         |       "prompt_tokens": 14037,
          "arguments": {                                      |       "total_tokens": 14060
            "entity_name": "Person",                          <
            "expression": "firstName != null"                 <
          },                                                  <
          "name": "test_calc_wrapper",                        <
          "type": "function"                                  <
        }                                                     <
      }                                                       <
    ],                                                        <
    "usage": {                                                <
      "completion_tokens": 148,                               <
      "prompt_tokens": 362,                                   <
      "total_tokens": 510                                     <
    }                                                               }
  },                                                              },
  "run": null                                                     "run": null
}                                                               }

[ssg-kstewart]

I have resolved my issue via a PR. For addtional details, refer to this comment.