Add Qwen pipeline and example #12292
Conversation
hkvision
force-pushed
the
qwen-pipeline
branch
from
e8dde16 to
489d351
Compare
| parser.add_argument( | ||
| "--repo-id-or-model-path", | ||
| type=str, | ||
| default="Qwen/Qwen2-7B-Instruct", |
There was a problem hiding this comment.
If "Qwen/Qwen2.5-7B-Instruct" can directly run with current code, maybe use "Qwen/Qwen2.5-7B-Instruct" as default ?
We can also update this in next PR 😊
There was a problem hiding this comment.
Sure, changed to 2.5 as default and renamed the file to qwen.py
Will test qwen2.5 soon
| default="Qwen/Qwen2-7B-Instruct", | ||
| help="The huggingface repo id for the Baichuan2 model to be downloaded" | ||
| ", or the path to the huggingface checkpoint folder", | ||
| ) |
There was a problem hiding this comment.
Maybe add parser.add_argument("--lowbit-path", type=str, ... here.
| parser.add_argument("--n-predict", type=int, default=32, help="Max tokens to predict") | ||
| parser.add_argument("--max-context-len", type=int, default=1024) | ||
| parser.add_argument("--max-prompt-len", type=int, default=960) | ||
| parser.add_argument("--disable-transpose-value-cache", action="store_true", default=False) |
There was a problem hiding this comment.
It looks like Qwen2 GW is already ready ? If so, maybe add parser.add_argument("--quantization_group_size", type=int, default=0) here ?
There was a problem hiding this comment.
Not tested yet, will test and update in a following PR.
README updated. |
|
Merge it first. Will fix any issues in a following PR. |
No description provided.