GenAI: Add a draft for NPU doc

[dmatveev]

Details:

• item1
• ...

Tickets:

• ticket-id

[dmatveev]

??

[TolyaTalamanov]

optimum-intel has different release cadence and usually taken from main branch

[TolyaTalamanov]

2024.3.1 hasn't been released yet!

[dmatveev]

Where does this point to? Is there a specific version or tag instead of a hash commit?

[TolyaTalamanov]

This is the hash commit from main branch that was verified to work with NPU.

In general, we may not specify the exact commit, hopefully it won't break NPU (not guaranteed).
openvino-notebooks takes optimum-intel just from main branch, see:
https://github.com/openvinotoolkit/openvino_notebooks/blob/a99c0ec648fc6414fb5c169e2dd0ef396c71f613/notebooks/llm-question-answering/llm-question-answering.ipynb#L28

%pip install -q "torch>=2.1" "nncf>=2.7" "transformers>=4.40.0" onnx "optimum>=1.16.1" "accelerate" "datasets>=2.14.6" "gradio>=4.19" "git+https://github.com/huggingface/optimum-intel.git" --extra-index-url https://download.pytorch.org/whl/cpu

[dmatveev]

can you find a proper fixed version or tag here?

[dmatveev]

Is chat-tuned the right term here?

[TolyaTalamanov]

Why not, model finetuned for chat scenarios

[kblaszczak-intel]

I can confirm that this term is used in the main article as well and can be found in the net scarcely. Changing it to "fine-tuned for chat" may be a good idea. We would change it in the genAI article too, in that case.

[dmatveev]

Doesn't alter the way it looks in RST but please limit lines to ~80-100 characters long.

In emacs it is easy, in vim I don't know.

[TolyaTalamanov]

I thought it's your piece, didn't touch this. No prob, will format it

[kblaszczak-intel]

Don't worry too much about formatting, we can polish the whole thing, make sure line breaks are fine, references work, directives render properly, and all that stuff :)

[TolyaTalamanov]

Preview: http://openvino-doc.iotg.sclab.intel.com/genai-npu-preview/learn-openvino/llm_inference_guide/genai-guide-npu.html

@dmatveev Could you have a look one more time, please?

[dmatveev]

Preview: http://openvino-doc.iotg.sclab.intel.com/genai-npu-preview/learn-openvino/llm_inference_guide/genai-guide-npu.html

@dmatveev Could you have a look one more time, please?

Reviewed, looks good to me, but I think it is now a dry how-to rather some useful educational text. What I mean here, there's no explanation that or emphasis that a specific type of quantization preferable (asymmetric in this case). Why does this instruction exist at all? What makes it different from the default one except a device name?

Also there's no other options covered, e.g. how to achieve better performance at the cost of compile time.

[TolyaTalamanov]

Preview: http://openvino-doc.iotg.sclab.intel.com/genai-npu-preview/learn-openvino/llm_inference_guide/genai-guide-npu.html
@dmatveev Could you have a look one more time, please?

Reviewed, looks good to me, but I think it is now a dry how-to rather some useful educational text. What I mean here, there's no explanation that or emphasis that a specific type of quantization preferable (asymmetric in this case). Why does this instruction exist at all? What makes it different from the default one except a device name?

Also there's no other options covered, e.g. how to achieve better performance at the cost of compile time.

The option to achieve better performance is covered in "Additional configuration options" part

[TolyaTalamanov]

LGTM 👍

[dmatveev]

@kblaszczak-intel can you please put your approve here? It seems merging is still blocked for this PR as @TolyaTalamanov seem not enough

[kblaszczak-intel]

Some tweaks added but nothing major.