使用accelerate和deepspeed进行多卡微调LLM卡住 #1683
Labels
Comments
|
我尝试过了，deepspeed都试过了，最后是在微调前加入一条指令。 |
hiyouga
added
good first issue
1 task
这里修改的指令，是将GPU之间的传输带宽等级提高。 |
1 task
|
export NCCL_P2P_LEVEL=NVL是直接在运行前加上这行命令就可以吗？ |
我加上这一句解决了使用nccl时卡死的问题，加上就可以多卡微调了 |
This was referenced Mar 2, 2024
2 tasks
|
请问没有nvlink，数据量比较大是不是就卡死了。 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Reminder
Reproduction
deepspeeddeepspeed --num_gpus 2 --master_port=9901 src/train_bash.py \ --deepspeed ds_config.json \ --stage sft \ --model_name_or_path /home/zhmao/model/Baichuan-13B-chat \ --do_train \ --dataset alpaca_gpt4_zh\ --template baichuan \ --finetuning_type lora \ --lora_rank 32 \ --lora_target all \ --output_dir /home/zhmao/model/Baichuan-13B-QLoRA \ --per_device_train_batch_size 4 \ --gradient_accumulation_steps 8 \ --preprocessing_num_workers 16 \ --cutoff_len 1024 \ --optim paged_adamw_32bit \ --lr_scheduler_type cosine \ --logging_steps 10 \ --save_steps 100 \ --eval_steps 100 \ --warmup_steps 100 \ --learning_rate 3e-5 \ --max_grad_norm 0.5 \ --num_train_epochs 2.0 \ --quantization_bit 4 \ --plot_loss \ --fp16deepspeed配置参数{ "train_micro_batch_size_per_gpu": "auto", "zero_allow_untested_optimizer": true, "fp16": { "enabled": "auto", "loss_scale": 0, "initial_scale_power": 16, "loss_scale_window": 1000, "hysteresis": 2, "min_loss_scale": 1 }, "zero_optimization": { "stage": 2, "allgather_partitions": true, "allgather_bucket_size": 5e8, "overlap_comm": false, "reduce_scatter": true, "reduce_bucket_size": 5e8, "contiguous_gradients" : true } }accelerateaccelerate launch /home/zhmao/.cache/huggingface/accelerate/default_config.yaml \ src/train_bash.py \ --stage sft \ --model_name_or_path /home/zhmao/model/Baichuan-13B-chat \ --do_train \ --dataset alpaca_gpt4_zh\ --template baichuan \ --finetuning_type lora \ --lora_rank 32 \ --lora_target all \ --output_dir /home/zhmao/model/Baichuan-13B-QLoRA \ --per_device_train_batch_size 4 \ --gradient_accumulation_steps 8 \ --preprocessing_num_workers 16 \ --cutoff_len 1024 \ --optim paged_adamw_32bit \ --lr_scheduler_type cosine \ --logging_steps 10 \ --save_steps 100 \ --eval_steps 100 \ --warmup_steps 100 \ --learning_rate 3e-5 \ --max_grad_norm 0.5 \ --num_train_epochs 2.0 \ --quantization_bit 4 \ --plot_loss \ --fp16例子中使用的Baichuan-13B-chat，但我尝试了ChatGLM2-6B也同样卡住
我已经查询过的issue：
#74
#1651
Expected behavior
期望正确的使用多卡进行加速
System Info
transformersversion: 4.33.2- distributed_type: MULTI_GPU
- mixed_precision: fp16
- use_cpu: False
- debug: False
- num_processes: 2
- machine_rank: 0
- num_machines: 1
- gpu_ids: all
- rdzv_backend: static
- same_network: True
- main_training_function: main
- downcast_bf16: no
- tpu_use_cluster: False
- tpu_use_sudo: False
- tpu_env: []
*GPU：A6000（40G）2
Others
终端所有输出The text was updated successfully, but these errors were encountered: