Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

transformers==4.37, yi & yuan2 & vicuna #11805

Merged
merged 6 commits into from
Aug 15, 2024
Merged

transformers==4.37, yi & yuan2 & vicuna #11805

merged 6 commits into from
Aug 15, 2024

Conversation

JinheTang
Copy link
Contributor

  1. added sample output for yi & vicuna
  2. updated vicuna example model to 7b-v1.5 & 13b-v1.5
  3. added yi-6b-chat model to yi

@JinheTang JinheTang changed the title transformers==4.37, yuan2 & vicuna transformers==4.37, yi & yuan2 & vicuna Aug 15, 2024
rnwang04
rnwang04 reviewed Aug 15, 2024
View reviewed changes
python/llm/example/GPU/HuggingFace/LLM/vicuna/README.md Outdated
```log
Inference time: xxxx s
Inference time: 1.0269405841827393 s
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't show real performance data in our example, just keep Inference time: xxxx s.

python/llm/example/GPU/HuggingFace/LLM/vicuna/README.md Outdated
```log
Inference time: xxxx s
Inference time: 0.7162051200866699 s
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just keep Inference time: xxxx s.

python/llm/example/GPU/HuggingFace/LLM/yi/README.md Outdated
@@ -112,18 +112,51 @@ python ./generate.py

In the example, several arguments can be passed to satisfy your requirements:

- `--repo-id-or-model-path REPO_ID_OR_MODEL_PATH`: argument defining the huggingface repo id for the Yi model (e.g. `01-ai/Yi-6B`) to be downloaded, or the path to the huggingface checkpoint folder. It is default to be `'01-ai/Yi-6B'`.
- `--repo-id-or-model-path REPO_ID_OR_MODEL_PATH`: argument defining the huggingface repo id for the Yi model (e.g. `01-ai/Yi-6B`) to be downloaded, or the path to the huggingface checkpoint folder. It is default to be `'01-ai/Yi-6B-Chat'`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(e.g. 01-ai/Yi-6B and 01-ai/Yi-6B-Chat)

python/llm/example/GPU/HuggingFace/LLM/yi/README.md Outdated
- `--prompt PROMPT`: argument defining the prompt to be infered (with integrated prompt format for chat). It is default to be `'AI是什么？'`.
- `--n-predict N_PREDICT`: argument defining the max number of tokens to predict. It is default to be `32`.

#### Sample Output
#### [01-ai/Yi-6B](https://huggingface.co/01-ai/Yi-6B)

```log
Inference time: xxxx s
Inference time: 1.1255202293395996 s
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just keep Inference time: xxxx s.

python/llm/example/GPU/HuggingFace/LLM/yi/README.md Outdated

#### [01-ai/Yi-6B-Chat](https://huggingface.co/01-ai/Yi-6B-Chat)
```log
Inference time: 0.5318927764892578 s
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just keep Inference time: xxxx s.

python/llm/example/GPU/HuggingFace/LLM/yi/generate.py Outdated
@@ -32,10 +32,10 @@

if __name__ == '__main__':
parser = argparse.ArgumentParser(description='Predict Tokens using `generate()` API for Yi model')
parser.add_argument('--repo-id-or-model-path', type=str, default="01-ai/Yi-6B",
parser.add_argument('--repo-id-or-model-path', type=str, default="/home/arda/jinhe/Yi-6B-Chat",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

default="01-ai/Yi-6B-Chat"

rnwang04
rnwang04 reviewed Aug 15, 2024
View reviewed changes
python/llm/example/GPU/HuggingFace/LLM/yi/generate.py Outdated
@@ -32,10 +32,10 @@

if __name__ == '__main__':
parser = argparse.ArgumentParser(description='Predict Tokens using `generate()` API for Yi model')
parser.add_argument('--repo-id-or-model-path', type=str, default="01-ai/Yi-6B",
parser.add_argument('--repo-id-or-model-path', type=str, default="/01-ai/Yi-6B-Chat",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

default="01-ai/Yi-6B-Chat"

rnwang04
rnwang04 reviewed Aug 15, 2024
View reviewed changes
python/llm/example/GPU/HuggingFace/LLM/yi/generate.py Outdated
@@ -22,20 +22,13 @@
from transformers import AutoTokenizer

# Refer to https://huggingface.co/01-ai/Yi-6B-Chat#31-use-the-chat-model
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also delete this line

rnwang04
rnwang04 approved these changes Aug 15, 2024
View reviewed changes
Copy link
Contributor

@rnwang04 rnwang04 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@rnwang04 rnwang04 merged commit 2fbbb51 into intel-analytics:main Aug 15, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rnwang04 rnwang04 rnwang04 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@JinheTang @rnwang04