Updated ollama handler to handle case when multiple tokens are return…

[ronneldavis]

Handling the case when multiple tokens are sent by ollama at the same time

Instead of decoding the entire payload, split it up based on the newline character and parse each one independently

[zya]

@ronneldavis By multiple tokens do you mean one streaming response from ollama could actually be multiple responses?
Can you let me know the following?

1. What are the cases where this happens?
2. How can I replicate it?
3. What is the current behaviour? Without your changes.

[zya]

Seems like the ollama folks themselves are handling the streaming response this way.
https://github.com/ollama-ui/ollama-ui/blob/main/api.js#L71-L97

[ronneldavis]

@zya I face this issue cause my output tokens have "\n" in it, its a code generation model.
You can use the following prompt for the wizardcoder moder to test it

Respond in the format <PLAN>...</PLAN><CODE>...</CODE>\nWithin the <CODE>...</CODE> block, write a function called \"main\" that returns a chart.js object to create a graph as a dictionary. \nThe function main should not accept any parameters.\nThe chart options should have responsive set as true and maintainAspectRatio set to false.  \nAdd a relevant title for the chart in options.plugins.title.text of the output object. \nIf the user asks for a metric, set the type of chart in config to me metric and return a single data point in the data.datasets.data array and the datasets.label should be an appropriate unit. \nUse only data from the given files to construct your response, expect the files to be unsorted. \nUse the pandas library load the necessary columns from the CSVs to reduce memory usage. The columns you load SHOULD exist in the dataset. Do not attempt to load columns that don't exist in a file!\nDo not use or import any other libraries other than the built-in python libraries, numpy and pandas.\nBefore performing numerical operations on a column like .sum(), use the convertNum() function to make sure its a numerical value (eg. df['column_name'] = df['column_name'].map(convertNum)). The convertNum has been written already and doesn't need to be created. Do not initialize a function called convertNum!\nWhen correlating different data sources using a common column, make sure to load the columns when importing the dataset. Do not try correlating two files if they both don't have the column you are looking for.\nWithin the <PLAN>...</PLAN> block, mention your plan as well as any assumptions behind the code as an ordered list.\nDo not write code to call or test the function at the end.\nDo not add any text or comments after the function. Do not explain the function after writing it.\n\n### DATASETS: \n  \nTitle: Sample Customers\nDescription: A sample datasource containing the details of customers in an e-commerce setting\nURL: /tmp/0e9180d1-48d9-4150-8388-dd4d67b7e0cd.csv\nColumns:\n[{\"title\":\"Customer ID\",\"description\":\"A unique identifier for the customer across tables\",\"samples\":[\"CG-12520\",\"DV-13045\",\"SO-20335\"]},{\"title\":\"City\",\"description\":\"City of residence\",\"samples\":[\"Henderson\",\"Los Angeles\",\"Fort Lauderdale\"]},{\"title\":\"Country\",\"description\":\"Country of residence\",\"samples\":[\"United States\",\"United States\",\"United States\"]},{\"title\":\"Segment\",\"description\":\"Segment of customer according to purchasing habits\",\"samples\":[\"Consumer\",\"Corporate\",\"Consumer\"]},{\"title\":\"State\",\"description\":\"State of residence\",\"samples\":[\"Kentucky\",\"California\",\"Florida\"]},{\"title\":\"Postal Code\",\"description\":\"Postal Code of customer\",\"samples\":[\"42420\",\"90036\",\"33311\"]},{\"title\":\"Region\",\"description\":\"Region of residence\",\"samples\":[\"South\",\"West\",\"South\"]}],\nTitle: Sample Orders\nDescription: A sample datasource containing the details of orders in an e-commerce setting\nURL: /tmp/f1a04cc3-b2c2-45bf-9eec-7561de369bb7.csv\nColumns:\n[{\"title\":\"Row ID\",\"description\":\"A serial number for every row\",\"samples\":[\"1\",\"2\",\"3\"]},{\"title\":\"Order ID\",\"description\":\"A unique ID for every order\",\"samples\":[\"CA-2017-152156\",\"CA-2017-152156\",\"CA-2017-138688\"]},{\"title\":\"Order Date\",\"description\":\"Order Date in YYYY-MM-DD format\",\"samples\":[\"2017-11-08\",\"2017-11-08\",\"2017-06-12\"]},{\"title\":\"Ship Date\",\"description\":\"Ship Date in YYYY-MM-DD format\",\"samples\":[\"2017-11-11\",\"2017-11-11\",\"2017-06-16\"]},{\"title\":\"Ship Mode\",\"description\":\"The type of shipping for the order\",\"samples\":[\"Second Class\",\"Second Class\",\"Second Class\"]},{\"title\":\"Customer ID\",\"description\":\"Customer ID\",\"samples\":[\"CG-12520\",\"CG-12520\",\"DV-13045\"]},{\"title\":\"Product ID\",\"description\":\"Product ID\",\"samples\":[\"FUR-BO-10001798\",\"FUR-CH-10000454\",\"OFF-LA-10000240\"]},{\"title\":\"Category\",\"description\":\"Category of product\",\"samples\":[\"Furniture\",\"Furniture\",\"Office Supplies\"]},{\"title\":\"Sub-Category\",\"description\":\"Sub-Category of product\",\"samples\":[\"Bookcases\",\"Chairs\",\"Labels\"]},{\"title\":\"Product Name\",\"description\":\"name of the product\",\"samples\":[\"Bush Somerset Collection Bookcase\",\"Hon Deluxe Fabric Upholstered Stacking Chairs, Rounded Back\",\"Self-Adhesive Address Labels for Typewriters by Universal\"]},{\"title\":\"Sales\",\"description\":\"Sales price in USD\",\"samples\":[\"261.96\",\"731.94\",\"14.62\"]},{\"title\":\"Quantity\",\"description\":\"Quantity as a whole number\",\"samples\":[\"2\",\"3\",\"2\"]},{\"title\":\"Discount\",\"description\":\"Discount as a percentage from 0-1\",\"samples\":[\"0\",\"0\",\"0\"]},{\"title\":\"Profit\",\"description\":\"Profit in USD (can be negative)\",\"samples\":[\"41.9136\",\"219.582\",\"6.8714\"]}]\n\n### PROMPT: Create a bar chart showing my orders month over month

You will notice that in a single streaming response, you will see multiple tokens that look like this:

{
    "model": "onvocoder:7b",
    "created_at": "2023-11-06T07:07:15.810698Z",
    "response": "python",
    "done": false
}
{
    "model": "onvocoder:7b",
    "created_at": "2023-11-06T07:07:15.847325Z",
    "response": "\n",
    "done": false
}
{
    "model": "onvocoder:7b",
    "created_at": "2023-11-06T07:07:15.883454Z",
    "response": "import",
    "done": false
}

[ronneldavis]

I think the Ollama folk are using the partialLine implementation to get around it as that's the only meaningful difference I see