Support Qwen in model builder

[BowenBao]

Resolves #718

cc @spandantiwari-amd

[BowenBao]

cc @kunal-vaishnavi for review

[kunal-vaishnavi]

Thanks for the contribution! Can you also update the following places?

1. Add the qwen2 model type that will show up in genai_config.json here
2. Add Qwen to the README here to show that Qwen is now supported

[BowenBao]

Thanks @kunal-vaishnavi, updated.

I realized we will need to introduce the Qwen tokenizer as well for it to run e2e in OGA. I'm putting Qwen in "Under Development" column for now.

[yufenglee]

Thanks @kunal-vaishnavi, updated.

I realized we will need to introduce the Qwen tokenizer as well for it to run e2e in OGA. I'm putting Qwen in "Under Development" column for now.

What tokenizer does Qwen use? Does it use same tokenizer as llama?

[yufenglee]

Thanks @kunal-vaishnavi, updated.
I realized we will need to introduce the Qwen tokenizer as well for it to run e2e in OGA. I'm putting Qwen in "Under Development" column for now.

What tokenizer does Qwen use? Does it use same tokenizer as llama?

Synced up with @wenbingl. There needs a small change to support Qwen tokenizer. He will add the change.

[wenbingl]

Thanks @kunal-vaishnavi, updated.
I realized we will need to introduce the Qwen tokenizer as well for it to run e2e in OGA. I'm putting Qwen in "Under Development" column for now.

What tokenizer does Qwen use? Does it use same tokenizer as llama?

Synced up with @wenbingl. There needs a small change to support Qwen tokenizer. He will add the change.

a quick PR is here: microsoft/onnxruntime-extensions#781

[BowenBao]

Thank you for the quick response @wenbingl, @yufenglee! I have validated your fix enables the qwen generation e2e.

Should I wait for the PR in extension to merge, and then include the commit update in this PR?

[kunal-vaishnavi]

Should I wait for the PR in extension to merge, and then include the commit update in this PR?

Yes, let's wait so that Qwen can be supported end-to-end with this PR.

[wenbingl]

Thank you for the quick response @wenbingl, @yufenglee! I have validated your fix enables the qwen generation e2e.

Should I wait for the PR in extension to merge, and then include the commit update in this PR?

the PR in ort-extensions was merged.

[BowenBao]

Updated. @kunal-vaishnavi PTAL.