Skip to content

Commit

Permalink
doc: fix outdated default value of batch size (#6336)
Browse files Browse the repository at this point in the history
* doc: fix outdated default value of batch size

* doc: add doc for ubatch-size
  • Loading branch information
Sunt-ing authored Mar 28, 2024
1 parent 6902cb7 commit cfc4d75
Showing 1 changed file with 3 additions and 1 deletion.
4 changes: 3 additions & 1 deletion examples/main/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -296,7 +296,9 @@ These options help improve the performance and memory usage of the LLaMA models.

### Batch Size

- `-b N, --batch-size N`: Set the batch size for prompt processing (default: 512). This large batch size benefits users who have BLAS installed and enabled it during the build. If you don't have BLAS enabled ("BLAS=0"), you can use a smaller number, such as 8, to see the prompt progress as it's evaluated in some situations.
- `-b N, --batch-size N`: Set the batch size for prompt processing (default: `2048`). This large batch size benefits users who have BLAS installed and enabled it during the build. If you don't have BLAS enabled ("BLAS=0"), you can use a smaller number, such as 8, to see the prompt progress as it's evaluated in some situations.

- `-ub N`, `--ubatch-size N`: physical maximum batch size. This is for pipeline parallelization. Default: `512`.

### Prompt Caching

Expand Down

0 comments on commit cfc4d75

Please sign in to comment.