Chat streaming is not working with Phi 2

[rupeshs]

Used sample code: https://github.com/openvinotoolkit/openvino.genai/blob/master/samples/python/chat_sample/chat_sample.py

ov_streaming_issue.mp4

[YuChern-Intel]

To use Phi-2 model, run with beam_search_causal_lm, greedy_causal_lm or multinomial_causal_lm.

[rupeshs]

@YuChern-Intel This is not mentioned in the documentation see https://github.com/openvinotoolkit/openvino.genai/blob/master/src/README.md.

[YuChern-Intel]

I had validated in running the microsoft/phi-2 model with these demos.

[olpipi]

Hi @rupeshs
The root cause is phi-2 model doesn't have chat template. Probably it is not tuned as chat. That's why it doesn't have template.
Genai pipeline crashes when you start chat scenario without chat template. I will fix it to return correct error.

If you want to run chat with phi-2 model, you should add chat_template field into tokenizer_config.json. See phi-3 for example.
I tried this template:
"chat_template": "{% for message in messages %}{% if (message['role'] == 'user') %}{{'Instruct: ' + message['content'] + '\nOutput:'}}{% elif (message['role'] == 'assistant') %}{{message['content'] + '\n'}}{% endif %}{% endfor %}"
It is not official, so I cannot guarantee it will work properly

[olpipi]

#697 - fix segfault