Documentation for returning an array of objects

[atakanokan]

Is there any way to return an array of objects (e.g. return multiple car objects):

{"type": "object", "properties": {"car": {"type": "object", "properties": {"make": {"type": "string"}, "model": {"type": "string"}, "year": {"type": "number"}, "colors": {"type": "array", "items": {"type": "string"}}, "features": {"type": "object", "properties": {"audio": {"type": "object", "properties": {"brand": {"type": "string"}, "speakers": {"type": "number"}, "hasBluetooth": {"type": "boolean"}}}, "safety": {"type": "object", "properties": {"airbags": {"type": "number"}, "parkingSensors": {"type": "boolean"}, "laneAssist": {"type": "boolean"}}}, "performance": {"type": "object", "properties": {"engine": {"type": "string"}, "horsepower": {"type": "number"}, "topSpeed": {"type": "number"}}}}}}}, "owner": {"type": "object", "properties": {"firstName": {"type": "string"}, "lastName": {"type": "string"}, "age": {"type": "number"}}}}}

Here is an example I tried that gave the below error:

json_schema = {
    "type": "array",
    "properties": {
        "type": "object",
        "properties": {
            "car": {
                "type": "object",
                "properties": {
                    "make": {"type": "string"},
                    "model": {"type": "string"},
                    "horsepower": {"type": "number"}
                }
            }
        }        
    }
}

error:

TypeError: string indices must be integers

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-0fa4792b-fa73-408b-b0a8-ecf9f5e56538/lib/python3.10/site-packages/jsonformer/main.py:242, in Jsonformer.__call__(self)
    240 def __call__(self) -> Dict[str, Any]:
    241     self.value = {}
--> 242     generated_data = self.generate_object(
    243         self.json_schema["properties"], self.value
    244     )
    245     return generated_data

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-0fa4792b-fa73-408b-b0a8-ecf9f5e56538/lib/python3.10/site-packages/jsonformer/main.py:147, in Jsonformer.generate_object(self, properties, obj)
    145 for key, schema in properties.items():
    146     self.debug("[generate_object] generating value for", key)
--> 147     obj[key] = self.generate_value(schema, obj, key)
    148 return obj

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-0fa4792b-fa73-408b-b0a8-ecf9f5e56538/lib/python3.10/site-packages/jsonformer/main.py:156, in Jsonformer.generate_value(self, schema, obj, key)
    150 def generate_value(
    151     self,
    152     schema: Dict[str, Any],
    153     obj: Union[Dict[str, Any], List[Any]],
    154     key: Union[str, None] = None,
    155 ) -> Any:
--> 156     schema_type = schema["type"]
    157     if schema_type == "number":
    158         if key:

[botka1998]

Hey @atakanokan
When defining an array you want to use the keyword "items" instead of "properties"
So you would want to use something like this:

json_schema = {
    "type": "array",
    "items": {
        "type": "object",
        "properties": {
            "car": {
                "type": "object",
                "properties": {
                    "make": {"type": "string"},
                    "model": {"type": "string"},
                    "horsepower": {"type": "number"}
                }
            }
        }        
    }
}

This should give you an array of car objects as long as you include that instruction inside the prompt.
If you are still having trouble getting arrays as in issue #46 check out my solution #47

[mpetruc]

This doesn't work, at least not in version 0.12.0. Here's the traceback:

{
	"name": "KeyError",
	"message": "'properties'",
	"stack": "---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[28], line 17
      1 json_schema = {
      2     \"type\": \"array\",
      3     \"items\": {
   (...)
     12             }
     13         }
     16 jsonformer = Jsonformer(model, tokenizer, json_schema, prompt=prompt, max_number_tokens=4000, temperature=1, debug=True,max_string_token_length=4000)
---> 17 generated_data = jsonformer()
     19 print(generated_data)

File /usr/local/lib/python3.10/dist-packages/jsonformer/main.py:243, in Jsonformer.__call__(self)
    240 def __call__(self) -> Dict[str, Any]:
    241     self.value = {}
    242     generated_data = self.generate_object(
--> 243         self.json_schema[\"properties\"], self.value
    244     )
    245     return generated_data

KeyError: 'properties'"
}

It seems that Jsonformer is ignoring the 'items' key and instead is looking just for 'properties. Or am i doing it wrongly? Thank you.

[mpetruc]

@botka1998 a few more details:

• i installed your bugfix branch (pip install --no-cache-dir git+https://github.com/botka1998/jsonformer.git@bugfix/change-array-generation-stopping-criteria)
• modified the calling function in main.py like this:

def __call__(self) -> Dict[str, Any]:
    self.value = {}
    if "items" in self.json_schema.keys():
        generated_data = self.generate_object(
            self.json_schema["items"]["properties"], self.value
    )
    else:
        generated_data = self.generate_object(
            self.json_schema["properties"], self.value
    )

    return generated_data

• using the json_schema from your previous post and

prompt = "You are a car expert. List information for 10 brands of cars: toyota, ford, gm, tesla, mazda, honda, suzuki, kia, audi, jeep. Use on the following schema:"
jsonformer = Jsonformer(model, tokenizer, json_schema, prompt=prompt,  temperature=1)
generated_data = jsonformer()
print(generated_data)

I am not getting the error message from above anymore, but neither do i get the array. Here is the response:

{'car': {'make': 'Toyota', 'model': 'Camry', 'horsepower': 200.0}}

Do you have any suggestions about what i should be doing differently? Thank you.

[botka1998]

Hey @mpetruc can you please share your json schema definition? Since it’s throwing a keyError, the schema is most likely invalid

[mpetruc]

@botka1998 Here it is:

json_schema = {
    "type": "array",
    "items": {
        "type": "object",
        "properties": {
            "car": {
                "type": "object",
                "properties": {
                    "make": {"type": "string"},
                    "model": {"type": "string"},
                    "horsepower": {"type": "number"}
                }
            }
        }        
    }
}

But like i said, after updating the call function i'm not getting the error anymore. It's just that i'm still getting only one item returned, as before. Thank you!

[mpetruc]

Ok, I think the JSON schema i was using was wrong. Here's a very simple example of how it should really look like:

json_schema = {
    "type": "object",
    "properties": {
        "items": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "drug_name": {"type": "string"},
                    "dose": {"type": "string"},
                    "administration_route": {"type": "string"}
                }
            }
        }
    }

This is the schema structure expected by the original call function in main.py.

So now i am using this type of schema with the original main.py as it gets installed from @botka1998 branch. However, i'm still not able to consistently get the arrays with prompts like this:

prompt = "Create a list of the top 10 most used hypertension medications. "

Calling the Jsonformer like this:

jsonformer = Jsonformer(model, tokenizer, json_schema, prompt=prompt,  temperature=0.9, max_array_length=15, max_string_token_length=4000, max_number_tokens=4000)
generated_data = jsonformer()

I rarely get more than 1 item generated, and never more than 3. Here's one example of response:

{'items': [{'drug_name': 'Amity-24', 'dose': '100 mg', 'administration_route': 'oral'}, {'drug_name': 'Atenolol', 'dose': '50 mg', 'administration_route': 'oral'}]}

I tried using several models: mistral 7b, a couple llama3 fine tunes, openbmb/MiniCPM-2B-sft-fp32. I don't think i see a difference, they all terminate earlier than I expect them to. I also think that it's not the models' limitation because when i generate with the transformers' model.generate() method the response includes the specified number of items (10 in this case).

Btw, the example above is obviously a toy example. What I'm really trying to use this for is for medication extraction, where i need to extract multiple drug names and their info from text snippets.

So, what am i missing? Are you guys really able to get larger arrays consistently? Thank you.

[botka1998]

Hey @mpetruc sorry for the late and sporadic responses, I’m on holiday and away from my laptop..

I’m glad you figured out the schema part!

Here’s my experience:
So I use this implementation in one project for function calling where the LLM creates a list of functions to call and their input args from natural language. So in my case, the LLM gets all the context, all the functions it should include, in the prompt. Yes it works quite well for this, but consistency is always hard with LLM-s, in my case, I am fine with missing a function call from time to time because I couldn’t get better results using just transformer.generate(). Using a better LLM and prompt engineering allowed me to get acceptable performance.

Here’s what you could try:
I see you’ve tried different models, I could suggest finding the best performing one that you can run on your hardware. I hear you don’t see a difference but this really made a huge difference for me.
Prompt engineering goes a long way! Write a longer, more instructive prompt that emphasizes the importance of the output containing a list, making sure to include all the instances in the list.

What I actually did in the fork:
I explain here what I did to achieve array generation #47 (comment)

Note that this isn’t the best possible solution, before I added this, array generation didn’t work at all. I stopped working on this as soon as I got good results for my application. I am not a maintainer of the original repo and the owners are not interested in maintaining so i didn’t bother doing more than this PR..
A good idea, if you’re up to do some coding, is to force the model to generate only the token that indicate the decision to us (comma or closed bracket) using logits. This might be a better solution, if all else fails, I might try and implement this when I get back to work, but I wouldn’t rely on it if I were you.

Here’s the model i used:
https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GPTQ

Here’s my prompt template

<s>[INST]
# TASK DESCRIPTION
You are to analyze user queries regarding industrial robotic operations and translate these into specific function calls. Your role is to bridge human instructions and the technical execution of a robot.
You will extract the information and values from the USER QUERY and form a JSON output which will represent a fuction call, you will fill in the JSON 
with the necessary information. 

# JSON PARAMETER VALUES
- functions: This is an array that represents all the function calls needed to perform the actions requested buy the USER QUERY. The structure of each function is defined below. These
functions are separated in the USER QUERY by words like "and", "then" ... you need to fill the array using all the actions defined by the USER QUERY while respectiong all the rules defined below.
- function_name: Which function does the robot need to perform. Can be only one of these: move_tcp, move_joint, get_joint_values
- input_name: Input parameter name for the function. Every function has a defined set of input parameters, what they mean, how they are named and what type of value they can be.
- input_value: The value of the previously mentioned function input parameter. It's type is defined within the function explaination
- inputs: An array of input_name, input_value pairs. One inputs array defines all the input parameters for a function call. This array must be complete according to the definition of the chosen function_name 

# FUNCTION MEANING
- move_tcp: Moves the robot's tool center point (TCP). This function takes as inputs: x, y, z, q1, q2, q3, q4
x, y ,z can only have number values, they represent the desired position of the TCP for each axis and are expressed in milimeters, this unit is implicit. Example: "Move TCP to the right by 0.1m" results in input_name: x and input_value: 10 
q1, q2, q3, q4 can only have number values and represent the quaternion value of the TCP orientation, defalut should be the TCP pointing down with it's z axis 
- move_joint: Rotates or moves a specific robot joint. This funciton takes as inputs joint, angle 
joint is the index of the joint that needs to rotate. Joints are indext 0, 1, 2 ... The joint 0 is also often reffered to as "base joint", "robot base" or just "base"
angle is the amount the specified joint needs to rotate and is expressed in radians
Example: "Rotate the third joint by 90 degrees." results in joint: 2 angle: 1.5708
- get_joint_values: Retrieves current status of the robot's joints. This function takes no input parameters! Example: "What is the position of the fifth joint?"

# RESPONSE FORMAT
- Include only the necessary functions directly implied by the query.
- Maintain the order of functions as implied by the sequence of actions in the query.
- In case of ambiguity, provide the most likely function while noting the uncertainty.

# ADDITIONAL GUIDANCE
- Focus on the verbs and technical terms in the query to determine the appropriate function.
- If a query involves actions not covered by the functions, such as maintenance requests, indicate that the query falls outside the function list.
- Consider the practical aspects of robotic operations when interpreting instructions.
- Treat "base" as equivalent to the first robot "joint".

USER QUERY: {user_query}
[/INST]

[mpetruc]

Thank you so much @botka1998 for the detailed, thoughtful response, and for taking the time to get back with me while you're on vacation! Much appreciated!
I believe that at this point the only way forward is to continue to tweak and optimize on your approach. I believe i've mentioned that my prompt is already able to (more or less) consistently extract all the information expected, just not in JSON. It is a chatml -type prompt, with system and user roles. Jsonformers does not support these prompts (as far as i can tell). So my jsonformers prompt consists of concatenating the system and user portions of the prompt. Even so, it never returns more than 3 items. So i will try to look into your suggestion for force the model to generate those decision tokens. Will let you know how it goes.

[projects-g]

@mpetruc @botka1998 A little guidance here please.

Here is the schema i am using :

{
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "line_items": {
                        "type": "object",
                        "properties": {
                            "item": {"type": "string"},
                            "quantity": {"type": "number"},
                            "price": {"type": "number"}
                        }
                    }
                }
            }
        }

To extract line items from an invoice. A simple & straight forward use case, but the error thrown is KeyError: 'properties'. But the schema seems to be correct.

[botka1998]

Hey @projects-g I’m afk again so this is from memory, I can’t check this but… try it

Make sure to start with an object that contains the array instead of the array immediately!
Like so:

json_schema = {
    "type": "object",
    "properties": {
        "line_items": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "item": {"type": "string"},
                    "quantity": {"type": "number"},
                    "price": {"type": "number"}
                }
            }
        }
    }