Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add yaml and modify concat_csv.py for transformers 4.43.1 #11758

Merged
merged 4 commits into from
Aug 13, 2024
Merged

add yaml and modify concat_csv.py for transformers 4.43.1 #11758

merged 4 commits into from
Aug 13, 2024

Conversation

ch1y0q
Copy link
Contributor

@ch1y0q ch1y0q commented Aug 11, 2024

add yaml and modify concat_csv.py for transformers 4.43.1

local run.py test

machine: xx01
models: gemma-2-2b-it, gemma-2-9b-it
apis: int4_fp16_gpu_win, int4_loadlowbit_gpu_win,

32-32, 1024-128, 2048-256, 3072-384 works fine for both models and both apis.

However, 4096-512 failed.

(qiyue-llm-0808) D:\qiyue\ipex-llm\python\llm\dev\benchmark\all-in-one>python run.py
C:\Users\arda\miniforge3\envs\qiyue-llm-0808\Lib\site-packages\transformers\deepspeed.py:24: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
  warnings.warn(
C:\Users\arda\miniforge3\envs\qiyue-llm-0808\Lib\site-packages\torchvision\io\image.py:13: UserWarning: Failed to load image Python extension: 'Could not find module 'C:\Users\arda\miniforge3\envs\qiyue-llm-0808\Lib\site-packages\torchvision\image.pyd' (or one of its dependencies). Try using the full path with constructor syntax.'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
  warn(
2024-08-11 23:50:32,405 - INFO - intel_extension_for_pytorch auto imported
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████| 2/2 [00:04<00:00,  2.43s/it]
2024-08-11 23:50:39,404 - INFO - Converting the current model to sym_int4 format......
C:\Users\arda\miniforge3\envs\qiyue-llm-0808\Lib\site-packages\torch\nn\init.py:412: UserWarning: Initializing zero-element tensors is a no-op
  warnings.warn("Initializing zero-element tensors is a no-op")
>> loading of model costs 8.510939300002065s and 1.412109375GB
<class 'transformers.models.gemma2.modeling_gemma2.Gemma2ForCausalLM'>
Assertion failed: mask->size(3) == context_length, file gemma2.cpp, line 362
Details 
repo_id:
  # - 'THUDM/chatglm2-6b'
  #- 'meta-llama/Llama-2-7b-chat-hf'
  - 'google/gemma-2-2b-it'
  - 'google/gemma-2-9b-it'
  # - 'meta-llama/Meta-Llama-3.1-8B-Instruct'
  # - 'liuhaotian/llava-v1.5-7b' # requires a LLAVA_REPO_DIR env variables pointing to the llava dir; added only for gpu win related test_api now
local_model_hub: 'D:\llm-models\'
warm_up: 1 # must set >=2 when run "pipeline_parallel_gpu" test_api
num_trials: 3
num_beams: 1 # default to greedy search
low_bit: 'sym_int4' # default to use 'sym_int4' (i.e. symmetric int4)
batch_size: 1 # default to 1
in_out_pairs:
  #- '32-32'
  #- '1024-128'
  #- '2048-256'
  #- '3072-384'
  - '4096-512'
test_api:
  #- "transformer_int4_fp16_gpu"             # on Intel GPU, transformer-like API, (qtype=int4), (dtype=fp16)
   - "transformer_int4_fp16_gpu_win"       # on Intel GPU for Windows, transformer-like API, (qtype=int4), (dtype=fp16)
  # - "transformer_int4_gpu"                # on Intel GPU, transformer-like API, (qtype=int4), (dtype=fp32)
  # - "transformer_int4_gpu_win"            # on Intel GPU for Windows, transformer-like API, (qtype=int4), (dtype=fp32)
  ## - "transformer_int4_loadlowbit_gpu_win" # on Intel GPU for Windows, transformer-like API, (qtype=int4), use load_low_bit API. Please make sure you have used the save.py to save the converted low bit model
  # - "transformer_int4_fp16_loadlowbit_gpu_win" # on Intel GPU for Windows, transformer-like API, (qtype=int4), (dtype=fp16), use load_low_bit API. Please make sure you have used the save.py to save the converted low bit model
  # - "bigdl_fp16_gpu"                      # on Intel GPU, use ipex-llm transformers API, (dtype=fp16), (qtype=fp16)
  # - "optimize_model_gpu"                  # on Intel GPU, can optimize any pytorch models include transformer model
  # - "deepspeed_optimize_model_gpu"        # on Intel GPU, deepspeed autotp inference
  # - "pipeline_parallel_gpu"               # on Intel GPU, pipeline parallel inference
  # - "speculative_gpu"                     # on Intel GPU, inference with self-speculative decoding
  # - "transformer_int4"                    # on Intel CPU, transformer-like API, (qtype=int4)
  # - "native_int4"                         # on Intel CPU
  # - "optimize_model"                      # on Intel CPU, can optimize any pytorch models include transformer model
  # - "pytorch_autocast_bf16"               # on Intel CPU
  # - "transformer_autocast_bf16"           # on Intel CPU
  # - "bigdl_ipex_bf16"                     # on Intel CPU, (qtype=bf16)
  # - "bigdl_ipex_int4"                     # on Intel CPU, (qtype=int4)
  # - "bigdl_ipex_int8"                     # on Intel CPU, (qtype=int8)
  # - "speculative_cpu"                     # on Intel CPU, inference with self-speculative decoding
  # - "deepspeed_transformer_int4_cpu"      # on Intel CPU, deepspeed autotp inference
  # - "transformers_int4_npu_win"           # on Intel NPU for Windows,  transformer-like API, (qtype=int4)
cpu_embedding: True # whether put embedding to CPU (only avaiable now for gpu win related test_api)
conda env 
(qiyue-llm-0808) D:\qiyue\ipex-llm\python\llm\dev\benchmark\all-in-one>conda list
# packages in environment at C:\Users\arda\miniforge3\envs\qiyue-llm-0808:
#
# Name                    Version                   Build  Channel
accelerate                0.23.0                   pypi_0    pypi
aiohappyeyeballs          2.3.5                    pypi_0    pypi
aiohttp                   3.10.3                   pypi_0    pypi
aiosignal                 1.3.1                    pypi_0    pypi
annotated-types           0.7.0                    pypi_0    pypi
antlr4-python3-runtime    4.9.3                    pypi_0    pypi
attrs                     24.2.0                   pypi_0    pypi
bigdl-core-xe-21          2.5.0b20240807           pypi_0    pypi
bigdl-core-xe-addons-21   2.5.0b20240807           pypi_0    pypi
bigdl-core-xe-batch-21    2.5.0b20240807           pypi_0    pypi
bzip2                     1.0.8                h2bbff1b_6    https://mirrors.sustech.edu.cn/anaconda/pkgs/main
ca-certificates           2024.7.2             haa95532_0    https://mirrors.sustech.edu.cn/anaconda/pkgs/main
certifi                   2024.7.4                 pypi_0    pypi
charset-normalizer        3.3.2                    pypi_0    pypi
colorama                  0.4.6                    pypi_0    pypi
datasets                  2.20.0                   pypi_0    pypi
dill                      0.3.8                    pypi_0    pypi
docstring-parser          0.16                     pypi_0    pypi
dpcpp-cpp-rt              2024.0.2                 pypi_0    pypi
filelock                  3.15.4                   pypi_0    pypi
frozenlist                1.4.1                    pypi_0    pypi
fsspec                    2024.5.0                 pypi_0    pypi
huggingface-hub           0.24.5                   pypi_0    pypi
idna                      3.7                      pypi_0    pypi
intel-cmplr-lib-rt        2024.0.2                 pypi_0    pypi
intel-cmplr-lic-rt        2024.0.2                 pypi_0    pypi
intel-extension-for-pytorch 2.1.10+xpu               pypi_0    pypi
intel-opencl-rt           2024.0.2                 pypi_0    pypi
intel-openmp              2024.0.2                 pypi_0    pypi
ipex-llm                  2.1.0b20240807           pypi_0    pypi
jinja2                    3.1.4                    pypi_0    pypi
libffi                    3.4.4                hd77b12b_1    https://mirrors.sustech.edu.cn/anaconda/pkgs/main
libuv                     1.48.0               h827c3e9_0    https://mirrors.sustech.edu.cn/anaconda/pkgs/main
markdown-it-py            3.0.0                    pypi_0    pypi
markupsafe                2.1.5                    pypi_0    pypi
mdurl                     0.1.2                    pypi_0    pypi
mkl                       2024.0.0                 pypi_0    pypi
mkl-dpcpp                 2024.0.0                 pypi_0    pypi
mpmath                    1.3.0                    pypi_0    pypi
multidict                 6.0.5                    pypi_0    pypi
multiprocess              0.70.16                  pypi_0    pypi
networkx                  3.3                      pypi_0    pypi
numpy                     1.26.4                   pypi_0    pypi
omegaconf                 2.3.0                    pypi_0    pypi
onednn                    2024.0.0                 pypi_0    pypi
onemkl-sycl-blas          2024.0.0                 pypi_0    pypi
onemkl-sycl-datafitting   2024.0.0                 pypi_0    pypi
onemkl-sycl-dft           2024.0.0                 pypi_0    pypi
onemkl-sycl-lapack        2024.0.0                 pypi_0    pypi
onemkl-sycl-rng           2024.0.0                 pypi_0    pypi
onemkl-sycl-sparse        2024.0.0                 pypi_0    pypi
onemkl-sycl-stats         2024.0.0                 pypi_0    pypi
onemkl-sycl-vm            2024.0.0                 pypi_0    pypi
openssl                   3.0.14               h827c3e9_0    https://mirrors.sustech.edu.cn/anaconda/pkgs/main
packaging                 24.1                     pypi_0    pypi
pandas                    2.2.2                    pypi_0    pypi
pillow                    10.4.0                   pypi_0    pypi
pip                       24.0            py311haa95532_0    https://mirrors.sustech.edu.cn/anaconda/pkgs/main
protobuf                  5.28.0rc1                pypi_0    pypi
psutil                    6.0.0                    pypi_0    pypi
py-cpuinfo                9.0.0                    pypi_0    pypi
pyarrow                   17.0.0                   pypi_0    pypi
pyarrow-hotfix            0.6                      pypi_0    pypi
pydantic                  2.8.2                    pypi_0    pypi
pydantic-core             2.20.1                   pypi_0    pypi
pygments                  2.18.0                   pypi_0    pypi
python                    3.11.9               he1021f5_0    https://mirrors.sustech.edu.cn/anaconda/pkgs/main
python-dateutil           2.9.0.post0              pypi_0    pypi
pytz                      2024.1                   pypi_0    pypi
pyyaml                    6.0.2                    pypi_0    pypi
regex                     2024.7.24                pypi_0    pypi
requests                  2.32.3                   pypi_0    pypi
rich                      13.7.1                   pypi_0    pypi
safetensors               0.4.4                    pypi_0    pypi
sentencepiece             0.2.0                    pypi_0    pypi
setuptools                69.5.1                   pypi_0    pypi
shtab                     1.7.1                    pypi_0    pypi
six                       1.16.0                   pypi_0    pypi
sqlite                    3.45.3               h2bbff1b_0    https://mirrors.sustech.edu.cn/anaconda/pkgs/main
sympy                     1.13.1                   pypi_0    pypi
tabulate                  0.9.0                    pypi_0    pypi
tbb                       2021.13.1                pypi_0    pypi
tk                        8.6.14               h0416ee5_0    https://mirrors.sustech.edu.cn/anaconda/pkgs/main
tokenizers                0.19.1                   pypi_0    pypi
torch                     2.1.0a0+cxx11.abi          pypi_0    pypi
torchvision               0.16.0a0+cxx11.abi          pypi_0    pypi
tqdm                      4.66.5                   pypi_0    pypi
transformers              4.43.1                   pypi_0    pypi
trl                       0.9.6                    pypi_0    pypi
typing-extensions         4.12.2                   pypi_0    pypi
tyro                      0.8.6                    pypi_0    pypi
tzdata                    2024.1                   pypi_0    pypi
urllib3                   2.2.2                    pypi_0    pypi
vc                        14.2                 h2eaa2aa_4    https://mirrors.sustech.edu.cn/anaconda/pkgs/main
vs2015_runtime            14.29.30133          h43f2093_4    https://mirrors.sustech.edu.cn/anaconda/pkgs/main
wheel                     0.43.0          py311haa95532_0    https://mirrors.sustech.edu.cn/anaconda/pkgs/main
xxhash                    3.4.1                    pypi_0    pypi
xz                        5.4.6                h8cc25b3_1    https://mirrors.sustech.edu.cn/anaconda/pkgs/main
yarl                      1.9.4                    pypi_0    pypi
zlib                      1.2.13               h8cc25b3_1    https://mirrors.sustech.edu.cn/anaconda/pkgs/main

@Oscilloscope98 Oscilloscope98 marked this pull request as ready for review August 13, 2024 03:04
@Oscilloscope98 Oscilloscope98 changed the base branch from main to igpu-workflow-transformers-443 August 13, 2024 03:05
@Oscilloscope98
Copy link
Contributor

Merge for test now

@Oscilloscope98 Oscilloscope98 merged commit 71db6f8 into intel-analytics:igpu-workflow-transformers-443 Aug 13, 2024
Oscilloscope98 pushed a commit that referenced this pull request Aug 13, 2024
@ch1y0q @Oscilloscope98
add yaml and modify concat_csv.py for transformers 4.43.1 (#11758)
a9c65f2 
* add yaml and modify `concat_csv.py` for `transformers` 4.43.1

* remove 4.43 for arc; fix;

* remove 4096-512 for 4.43

* comment some models
rnwang04
rnwang04 reviewed Aug 13, 2024
View reviewed changes
.github/workflows/llm_performance_tests.yml
set BIGDL_LLM_XMX_DISABLED=1

cd python\llm\dev\benchmark\all-in-one
move ..\..\..\test\benchmark\igpu-perf\3072-384_int4_fp16_438.yaml config.yaml
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move ..\..\..\test\benchmark\igpu-perf\3072-384_int4_fp16_443.yaml config.yaml

Oscilloscope98 added a commit that referenced this pull request Aug 13, 2024
@Oscilloscope98 @ch1y0q
Add gemma-2-2b-it and gemma-2-9b-it to igpu nightly performance t…
ec184af 
…est (#11778)

* add yaml and modify `concat_csv.py` for `transformers` 4.43.1 (#11758)

* add yaml and modify `concat_csv.py` for `transformers` 4.43.1

* remove 4.43 for arc; fix;

* remove 4096-512 for 4.43

* comment some models

* Small fix

* uncomment models (#11777)

---------

Co-authored-by: Ch1y0q <[email protected]>
@rnwang04 rnwang04 mentioned this pull request Aug 14, 2024
Merged
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rnwang04 rnwang04 rnwang04 left review comments

@Oscilloscope98 Oscilloscope98 Oscilloscope98 approved these changes

@glorysdj glorysdj Awaiting requested review from glorysdj

@liu-shaojun liu-shaojun Awaiting requested review from liu-shaojun

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ch1y0q @Oscilloscope98 @rnwang04