Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: change oneccl #12296

Merged
merged 8 commits into from
Oct 31, 2024
Merged

feat: change oneccl #12296

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
2d965cd
feat: change oneccl
cranechu0131 Oct 30, 2024
d11d88c
fix: restore llama-70b
cranechu0131 Oct 30, 2024
0b2029a
fix: remove tab
cranechu0131 Oct 30, 2024
46d9d51
fix: remove extra blank
cranechu0131 Oct 30, 2024
fe2e1a9
small fix
cranechu0131 Oct 30, 2024
5ff4e23
add comments
cranechu0131 Oct 30, 2024
303d5c8
fix: add a blank space
cranechu0131 Oct 31, 2024
a71f215
Merge branch 'main' into change_oneccl
cranechu0131 Oct 31, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion python/llm/example/GPU/Deepspeed-AutoTP/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@ conda activate llm
# below command will install intel_extension_for_pytorch==2.1.10+xpu as default
pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
pip install transformers==4.37.0
pip install oneccl_bind_pt==2.1.100 --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
wget https://sourceforge.net/projects/oneccl-wks/files/2024.0.0.5.1-release/oneccl_wks_installer_2024.0.0.5.1.sh
bash oneccl_wks_installer_2024.0.0.5.1.sh
# configures OneAPI environment variables
source /opt/intel/oneapi/setvars.sh
pip install git+https://github.com/microsoft/DeepSpeed.git@ed8aed5
Expand Down
5 changes: 3 additions & 2 deletions python/llm/example/GPU/Deepspeed-AutoTP/deepspeed_autotp.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,8 @@ def get_int_from_env(env_keys, default):
deepspeed.comm.comm.cdb = None
from deepspeed.comm.comm import init_distributed
init_distributed()

from ipex_llm.utils import BenchmarkWrapper
model = BenchmarkWrapper(model)
print(model)

# Load tokenizer
Expand Down Expand Up @@ -135,7 +136,7 @@ def get_int_from_env(env_keys, default):
actual_output_len = output.shape[1] - input_ids.shape[1]
output_str = tokenizer.decode(output[0], skip_special_tokens=True)
avg_time = (end - st) / actual_output_len * 1000
print(f'Inference time of generating {actual_output_len} tokens: {end-st} s, average token latency is {avg_time} ms/token.')
print(f'Inference time of generating {actual_output_len} tokens: {end-st} s,first token cost {model.first_cost} s, rest tokens average cost {model.rest_cost_mean} s')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

space before ,

print('-'*20, 'Prompt', '-'*20)
print(prompt)
print('-'*20, 'Output', '-'*20)
Expand Down
3 changes: 2 additions & 1 deletion python/llm/example/GPU/Deepspeed-AutoTP/run_llama2_70b_pvc_1550_1_card.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,8 @@ export CCL_ZE_IPC_EXCHANGE=sockets
export LD_PRELOAD=${LD_PRELOAD}:${CONDA_PREFIX}/lib/libtcmalloc.so:${LD_PRELOAD}
basekit_root=/opt/intel/oneapi
source $basekit_root/setvars.sh --force
source $basekit_root/ccl/latest/env/vars.sh --force
# source $basekit_root/ccl/latest/env/vars.sh --force deprecate oneccl_bind_pt and use internal oneccl for better performance
source /opt/intel/1ccl-wks/setvars.sh

export OMP_NUM_THREADS=$((56/$NUM_GPUS))
export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=2
Expand Down
3 changes: 2 additions & 1 deletion python/llm/example/GPU/Deepspeed-AutoTP/run_mistral_7b_instruct_flex_2_card.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@ export CCL_ZE_IPC_EXCHANGE=sockets
export LD_PRELOAD=${LD_PRELOAD}:${CONDA_PREFIX}/lib/libtcmalloc.so:${LD_PRELOAD}
basekit_root=/opt/intel/oneapi
source $basekit_root/setvars.sh --force
source $basekit_root/ccl/latest/env/vars.sh --force
# source $basekit_root/ccl/latest/env/vars.sh --force deprecate oneccl_bind_pt and use internal oneccl for better performance
source /opt/intel/1ccl-wks/setvars.sh

NUM_GPUS=2 # number of used GPU
export USE_XETLA=OFF
Expand Down
3 changes: 2 additions & 1 deletion python/llm/example/GPU/Deepspeed-AutoTP/run_qwen_14b_arc_2_card.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,8 @@ export CCL_ZE_IPC_EXCHANGE=sockets
export LD_PRELOAD=${LD_PRELOAD}:${CONDA_PREFIX}/lib/libtcmalloc.so:${LD_PRELOAD}
basekit_root=/opt/intel/oneapi
source $basekit_root/setvars.sh --force
source $basekit_root/ccl/latest/env/vars.sh --force
# source $basekit_root/ccl/latest/env/vars.sh --force deprecate oneccl_bind_pt and use internal oneccl for better performance
source /opt/intel/1ccl-wks/setvars.sh

NUM_GPUS=2 # number of used GPU
export USE_XETLA=OFF
Expand Down
3 changes: 2 additions & 1 deletion python/llm/example/GPU/Deepspeed-AutoTP/run_vicuna_33b_arc_2_card.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@ export CCL_ZE_IPC_EXCHANGE=sockets
export LD_PRELOAD=${LD_PRELOAD}:${CONDA_PREFIX}/lib/libtcmalloc.so:${LD_PRELOAD}
basekit_root=/opt/intel/oneapi
source $basekit_root/setvars.sh --force
source $basekit_root/ccl/latest/env/vars.sh --force
# source $basekit_root/ccl/latest/env/vars.sh --force deprecate oneccl_bind_pt and use internal oneccl for better performance
source /opt/intel/1ccl-wks/setvars.sh

NUM_GPUS=2 # number of used GPU
export USE_XETLA=OFF
Expand Down
Loading