Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] model interface input schema validation failed for cohere model #2829

Closed
mingshl opened this issue Aug 14, 2024 · 3 comments · Fixed by #2847
Closed

[BUG] model interface input schema validation failed for cohere model #2829

mingshl opened this issue Aug 14, 2024 · 3 comments · Fixed by #2847
Assignees
Labels
bug Something isn't working

Comments

@mingshl
Copy link
Collaborator

mingshl commented Aug 14, 2024

What is the bug?
model interface input schema validation failed for cohere model when there is pre processing function, suspect that the model interface validation happens before the pre-processing function, so it throw exceptions before it reaches pre-processing function

following the blueprint , got status_exception

How can one reproduce the bug?
Steps to reproduce the behavior:

step 1: create connector follow the blue print:

POST /_plugins/_ml/connectors/_create
{
    "name": "Amazon Bedrock Connector: Cohere embed-multilingual-v3",
    "description": "Test connector for Amazon Bedrock Cohere embed-multilingual-v3",
    "version": 1,
    "protocol": "aws_sigv4",
    "credential": {
    },
    "parameters": {
        "region": "us-west-2",
        "service_name": "bedrock",
        "truncate": "NONE",
        "input_type": "search_document",
        "model": "cohere.embed-multilingual-v3"
    },
    "actions": [
        {
            "action_type": "predict",
            "method": "POST",
            "headers": {
                "x-amz-content-sha256": "required",
                "content-type": "application/json"
            },
            "url": "https://bedrock-runtime.${parameters.region}.amazonaws.com/model/${parameters.model}/invoke",
            "request_body": "{ \"texts\": ${parameters.texts}, \"truncate\": \"${parameters.truncate}\", \"input_type\": \"${parameters.input_type}\" }",
            "pre_process_function": "connector.pre_process.cohere.embedding",
            "post_process_function": "connector.post_process.cohere.embedding"
        }
    ]
}

Step 2: register and deploy model

POST /_plugins/_ml/models/_register
{
  "name": "Bedrock   model ",
  "function_name": "remote",
  "description": "test model",
  "connector_id": "sDsGU5EBVjLDgS3RQdWR"}
  
POST /_plugins/_ml/models/uDsGU5EBVjLDgS3RWdWZ/_deploy

Step 3: call predict using the same example from blue print

POST /_plugins/_ml/models/uDsGU5EBVjLDgS3RWdWZ/_predict  
{
  "parameters": {
    "texts" : ["上海", "This is a test"]
  }
}


got exceptions:

{
  "error": {
    "root_cause": [
      {
        "type": "status_exception",
        "reason": "Error validating input schema: Validation failed: [$.parameters.texts: string found, array expected] for instance: {\"algorithm\":\"REMOTE\",\"parameters\":{\"texts\":\"[\\\"上海\\\",\\\"This is a test\\\"]\"},\"action_type\":\"PREDICT\"} with schema: {\n    \"type\": \"object\",\n    \"properties\": {\n        \"parameters\": {\n            \"type\": \"object\",\n            \"properties\": {\n                \"texts\": {\n                    \"type\": \"array\",\n                    \"items\": {\n                        \"type\": \"string\"\n                    }\n                }\n            },\n            \"required\": [\n                \"texts\"\n            ]\n        }\n    },\n    \"required\": [\n        \"parameters\"\n    ]\n}"
      }
    ],
    "type": "status_exception",
    "reason": "Error validating input schema: Validation failed: [$.parameters.texts: string found, array expected] for instance: {\"algorithm\":\"REMOTE\",\"parameters\":{\"texts\":\"[\\\"上海\\\",\\\"This is a test\\\"]\"},\"action_type\":\"PREDICT\"} with schema: {\n    \"type\": \"object\",\n    \"properties\": {\n        \"parameters\": {\n            \"type\": \"object\",\n            \"properties\": {\n                \"texts\": {\n                    \"type\": \"array\",\n                    \"items\": {\n                        \"type\": \"string\"\n                    }\n                }\n            },\n            \"required\": [\n                \"texts\"\n            ]\n        }\n    },\n    \"required\": [\n        \"parameters\"\n    ]\n}"
  },
  "status": 400
}

What is the expected behavior?
A clear and concise description of what you expected to happen.

What is your host/environment?

  • OS:
  • Version 2.16
  • Plugins

Do you have any screenshots?
If applicable, add screenshots to help explain your problem.

Do you have any additional context?
Add any other context about the problem.

@mingshl mingshl added bug Something isn't working untriaged labels Aug 14, 2024
@b4sjoo b4sjoo self-assigned this Aug 14, 2024
@b4sjoo b4sjoo removed the untriaged label Aug 14, 2024
@b4sjoo
Copy link
Collaborator

b4sjoo commented Aug 16, 2024

After testing, this bug will occur in both cohere.embed-multilingual-v31 and cohere.embed-english-v3 bedrock model. And the exception behavior is flaky. That means sometimes it will get successfully parsed, but most of the time not

@b4sjoo
Copy link
Collaborator

b4sjoo commented Aug 16, 2024

I would suggest two ways:

  1. To cancel the automated model interface generation support towards bedrock cohere model until we find a way to fix it entirely
  2. Fix the json schema to another structure, which is more likely to pass the check (I would say it will pass 90% of the time)

@b4sjoo
Copy link
Collaborator

b4sjoo commented Aug 26, 2024

Checked, this issue will only impact Bedrock Cohere models

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Development

Successfully merging a pull request may close this issue.

2 participants