glm4在stage==rm微调时评估出现：CUDA error: device-side assert triggered

[ldknight]

Reminder

• I have read the README and searched the existing issues.

System Info

0526版本

Reproduction

[2024-07-02 06:42:39,946] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-07-02 06:42:39,946] [INFO] [comm.py:668:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
07/02/2024 06:42:40 - INFO - llamafactory.hparams.parser - Process rank: 0, device: cuda:0, n_gpu: 1, distributed training: True, compute dtype: torch.bfloat16
[INFO|tokenization_utils_base.py:2106] 2024-07-02 06:42:40,049 >> loading file tokenizer.model
[INFO|tokenization_utils_base.py:2106] 2024-07-02 06:42:40,049 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2106] 2024-07-02 06:42:40,049 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2106] 2024-07-02 06:42:40,049 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2106] 2024-07-02 06:42:40,049 >> loading file tokenizer.json
[WARNING|logging.py:314] 2024-07-02 06:42:40,924 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
07/02/2024 06:42:40 - INFO - llamafactory.data.template - Add <|user|>,<|observation|> to stop words.
07/02/2024 06:42:40 - INFO - llamafactory.data.loader - Loading dataset preferenceData.json...
Converting format of dataset (num_proc=16): 100%|███████████████████████████████| 48/48 [00:00<00:00, 103.56 examples/s]
Running tokenizer on dataset (num_proc=16): 100%|████████████████████████████████| 48/48 [00:41<00:00, 1.17 examples/s]
chosen_input_ids:
[151331, 151333, 151336, 198, 25234, 7600, 320, 6279, 8, 151337, 198, 62628, 4226, 320, 6279, 8, 151329]
chosen_inputs:
[gMASK] <|user|>
human instruction (required) <|assistant|>
chosen answer (required) <|endoftext|>
chosen_label_ids:
[-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 198, 62628, 4226, 320, 6279, 8, 151329]
chosen_labels:

chosen answer (required) <|endoftext|>
rejected_input_ids:
[151331, 151333, 151336, 198, 25234, 7600, 320, 6279, 8, 151337, 198, 94552, 4226, 320, 6279, 8, 151329]
rejected_inputs:
[gMASK] <|user|>
human instruction (required) <|assistant|>
rejected answer (required) <|endoftext|>
rejected_label_ids:
[-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 198, 94552, 4226, 320, 6279, 8, 151329]
rejected_labels:

rejected answer (required) <|endoftext|>
[INFO|configuration_utils.py:731] 2024-07-02 06:43:27,162 >> loading configuration file /home/models_dir/glm-4-9b/config.json
[INFO|configuration_utils.py:731] 2024-07-02 06:43:27,164 >> loading configuration file /home/models_dir/glm-4-9b/config.json
[INFO|configuration_utils.py:796] 2024-07-02 06:43:27,165 >> Model config ChatGLMConfig {
"_name_or_path": "/home/models_dir/glm-4-9b",
"add_bias_linear": false,
"add_qkv_bias": true,
"apply_query_key_layer_scaling": true,
"apply_residual_connection_post_layernorm": false,
"architectures": [
"ChatGLMModel"
],
"attention_dropout": 0.0,
"attention_softmax_in_fp32": true,
"auto_map": {
"AutoConfig": "configuration_chatglm.ChatGLMConfig",
"AutoModel": "modeling_chatglm.ChatGLMForConditionalGeneration",
"AutoModelForCausalLM": "modeling_chatglm.ChatGLMForConditionalGeneration",
"AutoModelForSeq2SeqLM": "modeling_chatglm.ChatGLMForConditionalGeneration",
"AutoModelForSequenceClassification": "modeling_chatglm.ChatGLMForSequenceClassification"
},
"bias_dropout_fusion": true,
"classifier_dropout": null,
"eos_token_id": [
151329,
151336,
151338
],
"ffn_hidden_size": 13696,
"fp32_residual_connection": false,
"hidden_dropout": 0.0,
"hidden_size": 4096,
"kv_channels": 128,
"layernorm_epsilon": 1.5625e-07,
"model_type": "chatglm",
"multi_query_attention": true,
"multi_query_group_num": 2,
"num_attention_heads": 32,
"num_layers": 40,
"original_rope": true,
"pad_token_id": 151329,
"padded_vocab_size": 151552,
"post_layer_norm": true,
"rmsnorm": true,
"rope_ratio": 1,
"seq_length": 8192,
"tie_word_embeddings": false,
"torch_dtype": "bfloat16",
"transformers_version": "4.41.2",
"use_cache": true,
"vocab_size": 151552
}

07/02/2024 06:43:27 - WARNING - llamafactory.model.utils.rope - Current model does not support RoPE scaling.
[INFO|modeling_utils.py:3471] 2024-07-02 06:43:27,226 >> loading weights file /home/models_dir/glm-4-9b/model.safetensors.index.json
[INFO|modeling_utils.py:3614] 2024-07-02 06:43:27,227 >> Detected DeepSpeed ZeRO-3: activating zero.init() for this model
[INFO|configuration_utils.py:962] 2024-07-02 06:43:27,238 >> Generate config GenerationConfig {
"eos_token_id": [
151329,
151336,
151338
],
"pad_token_id": 151329
}

[2024-07-02 06:43:50,333] [INFO] [partition_parameters.py:343:exit] finished initializing model - num_params = 283, num_elems = 9.40B
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████| 10/10 [00:10<00:00, 1.07s/it]
[INFO|modeling_utils.py:4280] 2024-07-02 06:44:01,101 >> All model checkpoint weights were used when initializing ChatGLMForConditionalGeneration.

[INFO|modeling_utils.py:4288] 2024-07-02 06:44:01,101 >> All the weights of ChatGLMForConditionalGeneration were initialized from the model checkpoint at /home/models_dir/glm-4-9b.
If your task is similar to the task the model of the checkpoint was trained on, you can already use ChatGLMForConditionalGeneration for predictions without further training.
[INFO|modeling_utils.py:3797] 2024-07-02 06:44:01,107 >> Generation config file not found, using a generation config created from the model config.
07/02/2024 06:44:01 - INFO - llamafactory.model.utils.checkpointing - Gradient checkpointing enabled.
07/02/2024 06:44:01 - INFO - llamafactory.model.utils.attention - Using vanilla attention implementation.
07/02/2024 06:44:01 - INFO - llamafactory.model.adapter - ZeRO3/FSDP/PureBF16/BAdam detected, remaining trainable params as their original precision.
07/02/2024 06:44:01 - INFO - llamafactory.model.adapter - Fine-tuning method: Freeze
07/02/2024 06:44:01 - INFO - llamafactory.model.adapter - Set trainable layers: 38,39
07/02/2024 06:44:01 - INFO - llamafactory.model.utils.valuehead - Failed to load value_head.safetensors: /home/models_dir/glm-4-9b does not appear to have a file named value_head.safetensors. Checkout 'https://huggingface.co//home/models_dir/glm-4-9b/tree/None' for available files.
07/02/2024 06:44:01 - INFO - llamafactory.model.utils.valuehead - Failed to load value_head.bin: /home/models_dir/glm-4-9b does not appear to have a file named value_head.bin. Checkout 'https://huggingface.co//home/models_dir/glm-4-9b/tree/None' for available files.
07/02/2024 06:44:01 - INFO - llamafactory.model.utils.valuehead - Provided path (/home/models_dir/glm-4-9b) does not contain value head weights.
07/02/2024 06:44:01 - INFO - llamafactory.model.utils.valuehead - Ignore these messages if you are not resuming the training of a value head model.
07/02/2024 06:44:01 - INFO - llamafactory.model.loader - trainable params: 1649443841 || all params: 9399955457 || trainable%: 17.5474
[INFO|trainer.py:641] 2024-07-02 06:44:01,202 >> Using auto half precision backend
[INFO|deepspeed.py:328] 2024-07-02 06:44:01,431 >> Detected ZeRO Offload and non-DeepSpeed optimizers: This combination should work as long as the custom optimizer has both CPU and GPU implementation (except LAMB)
Using /root/.cache/torch_extensions/py310_cu121 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /root/.cache/torch_extensions/py310_cu121/cpu_adam/build.ninja...
Building extension module cpu_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module cpu_adam...
Time to load cpu_adam op: 2.5703651905059814 seconds
Adam Optimizer #0 is created with AVX2 arithmetic capability.
Config: alpha=0.000050, betas=(0.900000, 0.999000), weight_decay=0.010000, adam_w=1
[2024-07-02 06:44:05,862] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.14.0, git-hash=unknown, git-branch=unknown
[2024-07-02 06:44:05,881] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
[2024-07-02 06:44:05,883] [INFO] [logging.py:96:log_dist] [Rank 0] Using client Optimizer as basic optimizer
[2024-07-02 06:44:05,883] [INFO] [logging.py:96:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer
[2024-07-02 06:44:05,886] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam
[2024-07-02 06:44:05,886] [INFO] [utils.py:56:is_zero_supported_optimizer] Checking ZeRO support for optimizer=DeepSpeedCPUAdam type=<class 'deepspeed.ops.adam.cpu_adam.DeepSpeedCPUAdam'>
[2024-07-02 06:44:05,886] [INFO] [logging.py:96:log_dist] [Rank 0] Creating fp16 ZeRO stage 3 optimizer, MiCS is enabled False, Hierarchical params gather False
[2024-07-02 06:44:05,886] [INFO] [logging.py:96:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 3 optimizer
[2024-07-02 06:44:06,153] [INFO] [utils.py:800:see_memory_usage] Stage 3 initialize beginning
[2024-07-02 06:44:06,154] [INFO] [utils.py:801:see_memory_usage] MA 0.0 GB Max_MA 2.31 GB CA 0.0 GB Max_CA 2 GB
[2024-07-02 06:44:06,154] [INFO] [utils.py:808:see_memory_usage] CPU Virtual Memory: used = 47.26 GB, percent = 18.8%
[2024-07-02 06:44:06,157] [INFO] [stage3.py:130:init] Reduce bucket size 16777216
[2024-07-02 06:44:06,157] [INFO] [stage3.py:131:init] Prefetch bucket size 15099494
[2024-07-02 06:44:06,387] [INFO] [utils.py:800:see_memory_usage] DeepSpeedZeRoOffload initialize [begin]
[2024-07-02 06:44:06,388] [INFO] [utils.py:801:see_memory_usage] MA 0.0 GB Max_MA 0.0 GB CA 0.0 GB Max_CA 0 GB
[2024-07-02 06:44:06,388] [INFO] [utils.py:808:see_memory_usage] CPU Virtual Memory: used = 47.24 GB, percent = 18.8%
Parameter Offload: Total persistent parameters: 520193 in 123 params
[2024-07-02 06:44:06,667] [INFO] [utils.py:800:see_memory_usage] DeepSpeedZeRoOffload initialize [end]
[2024-07-02 06:44:06,668] [INFO] [utils.py:801:see_memory_usage] MA 0.0 GB Max_MA 0.0 GB CA 0.0 GB Max_CA 0 GB
[2024-07-02 06:44:06,668] [INFO] [utils.py:808:see_memory_usage] CPU Virtual Memory: used = 47.25 GB, percent = 18.8%
[2024-07-02 06:44:06,916] [INFO] [utils.py:800:see_memory_usage] Before creating fp16 partitions
[2024-07-02 06:44:06,917] [INFO] [utils.py:801:see_memory_usage] MA 0.0 GB Max_MA 0.0 GB CA 0.0 GB Max_CA 0 GB
[2024-07-02 06:44:06,917] [INFO] [utils.py:808:see_memory_usage] CPU Virtual Memory: used = 47.24 GB, percent = 18.8%
[2024-07-02 06:44:09,647] [INFO] [utils.py:800:see_memory_usage] After creating fp16 partitions: 3
[2024-07-02 06:44:09,648] [INFO] [utils.py:801:see_memory_usage] MA 0.0 GB Max_MA 0.0 GB CA 0.0 GB Max_CA 0 GB
[2024-07-02 06:44:09,649] [INFO] [utils.py:808:see_memory_usage] CPU Virtual Memory: used = 51.32 GB, percent = 20.4%
[2024-07-02 06:44:09,878] [INFO] [utils.py:800:see_memory_usage] Before creating fp32 partitions
[2024-07-02 06:44:09,879] [INFO] [utils.py:801:see_memory_usage] MA 0.0 GB Max_MA 0.0 GB CA 0.0 GB Max_CA 0 GB
[2024-07-02 06:44:09,879] [INFO] [utils.py:808:see_memory_usage] CPU Virtual Memory: used = 51.32 GB, percent = 20.4%
[2024-07-02 06:44:10,976] [INFO] [utils.py:800:see_memory_usage] After creating fp32 partitions
[2024-07-02 06:44:10,977] [INFO] [utils.py:801:see_memory_usage] MA 0.0 GB Max_MA 0.0 GB CA 0.0 GB Max_CA 0 GB
[2024-07-02 06:44:10,977] [INFO] [utils.py:808:see_memory_usage] CPU Virtual Memory: used = 57.37 GB, percent = 22.8%
[2024-07-02 06:44:11,196] [INFO] [utils.py:800:see_memory_usage] Before initializing optimizer states
[2024-07-02 06:44:11,197] [INFO] [utils.py:801:see_memory_usage] MA 0.0 GB Max_MA 0.0 GB CA 0.0 GB Max_CA 0 GB
[2024-07-02 06:44:11,197] [INFO] [utils.py:808:see_memory_usage] CPU Virtual Memory: used = 57.37 GB, percent = 22.8%
[2024-07-02 06:44:17,113] [INFO] [utils.py:800:see_memory_usage] After initializing optimizer states
[2024-07-02 06:44:17,114] [INFO] [utils.py:801:see_memory_usage] MA 0.0 GB Max_MA 0.0 GB CA 0.0 GB Max_CA 0 GB
[2024-07-02 06:44:17,114] [INFO] [utils.py:808:see_memory_usage] CPU Virtual Memory: used = 65.53 GB, percent = 26.0%
[2024-07-02 06:44:17,115] [INFO] [stage3.py:486:setup_for_real_optimizer] optimizer state initialized
[2024-07-02 06:44:20,355] [INFO] [utils.py:800:see_memory_usage] After initializing ZeRO optimizer
[2024-07-02 06:44:20,356] [INFO] [utils.py:801:see_memory_usage] MA 0.03 GB Max_MA 2.34 GB CA 2.35 GB Max_CA 2 GB
[2024-07-02 06:44:20,357] [INFO] [utils.py:808:see_memory_usage] CPU Virtual Memory: used = 69.61 GB, percent = 27.7%
[2024-07-02 06:44:20,357] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Final Optimizer = DeepSpeedCPUAdam
[2024-07-02 06:44:20,357] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed using client LR scheduler
[2024-07-02 06:44:20,357] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed LR Scheduler = None
[2024-07-02 06:44:20,357] [INFO] [logging.py:96:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0, 0.0], mom=[(0.9, 0.999), (0.9, 0.999)]
[2024-07-02 06:44:20,358] [INFO] [config.py:996:print] DeepSpeedEngine configuration:
[2024-07-02 06:44:20,359] [INFO] [config.py:1000:print] activation_checkpointing_config {
"partition_activations": false,
"contiguous_memory_optimization": false,
"cpu_checkpointing": false,
"number_checkpoints": null,
"synchronize_checkpoint_boundary": false,
"profile": false
}
[2024-07-02 06:44:20,359] [INFO] [config.py:1000:print] aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True}
[2024-07-02 06:44:20,360] [INFO] [config.py:1000:print] amp_enabled .................. False
[2024-07-02 06:44:20,360] [INFO] [config.py:1000:print] amp_params ................... False
[2024-07-02 06:44:20,360] [INFO] [config.py:1000:print] autotuning_config ............ {
"enabled": false,
"start_step": null,
"end_step": null,
"metric_path": null,
"arg_mappings": null,
"metric": "throughput",
"model_info": null,
"results_dir": "autotuning_results",
"exps_dir": "autotuning_exps",
"overwrite": true,
"fast": true,
"start_profile_step": 3,
"end_profile_step": 5,
"tuner_type": "gridsearch",
"tuner_early_stopping": 5,
"tuner_num_trials": 50,
"model_info_path": null,
"mp_size": 1,
"max_train_batch_size": null,
"min_train_batch_size": 1,
"max_train_micro_batch_size_per_gpu": 1.024000e+03,
"min_train_micro_batch_size_per_gpu": 1,
"num_tuning_micro_batch_sizes": 3
}
[2024-07-02 06:44:20,360] [INFO] [config.py:1000:print] bfloat16_enabled ............. True
[2024-07-02 06:44:20,360] [INFO] [config.py:1000:print] bfloat16_immediate_grad_update False
[2024-07-02 06:44:20,360] [INFO] [config.py:1000:print] checkpoint_parallel_write_pipeline False
[2024-07-02 06:44:20,360] [INFO] [config.py:1000:print] checkpoint_tag_validation_enabled True
[2024-07-02 06:44:20,360] [INFO] [config.py:1000:print] checkpoint_tag_validation_fail False
[2024-07-02 06:44:20,360] [INFO] [config.py:1000:print] comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7fe59e7fd630>
[2024-07-02 06:44:20,360] [INFO] [config.py:1000:print] communication_data_type ...... None
[2024-07-02 06:44:20,360] [INFO] [config.py:1000:print] compile_config ............... enabled=False backend='inductor' kwargs={}
[2024-07-02 06:44:20,360] [INFO] [config.py:1000:print] compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
[2024-07-02 06:44:20,360] [INFO] [config.py:1000:print] curriculum_enabled_legacy .... False
[2024-07-02 06:44:20,361] [INFO] [config.py:1000:print] curriculum_params_legacy ..... False
[2024-07-02 06:44:20,361] [INFO] [config.py:1000:print] data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}
[2024-07-02 06:44:20,361] [INFO] [config.py:1000:print] data_efficiency_enabled ...... False
[2024-07-02 06:44:20,361] [INFO] [config.py:1000:print] dataloader_drop_last ......... False
[2024-07-02 06:44:20,361] [INFO] [config.py:1000:print] disable_allgather ............ False
[2024-07-02 06:44:20,361] [INFO] [config.py:1000:print] dump_state ................... False
[2024-07-02 06:44:20,361] [INFO] [config.py:1000:print] dynamic_loss_scale_args ...... None
[2024-07-02 06:44:20,361] [INFO] [config.py:1000:print] eigenvalue_enabled ........... False
[2024-07-02 06:44:20,361] [INFO] [config.py:1000:print] eigenvalue_gas_boundary_resolution 1
[2024-07-02 06:44:20,361] [INFO] [config.py:1000:print] eigenvalue_layer_name ........ bert.encoder.layer
[2024-07-02 06:44:20,361] [INFO] [config.py:1000:print] eigenvalue_layer_num ......... 0
[2024-07-02 06:44:20,361] [INFO] [config.py:1000:print] eigenvalue_max_iter .......... 100
[2024-07-02 06:44:20,361] [INFO] [config.py:1000:print] eigenvalue_stability ......... 1e-06
[2024-07-02 06:44:20,361] [INFO] [config.py:1000:print] eigenvalue_tol ............... 0.01
[2024-07-02 06:44:20,361] [INFO] [config.py:1000:print] eigenvalue_verbose ........... False
[2024-07-02 06:44:20,361] [INFO] [config.py:1000:print] elasticity_enabled ........... False
[2024-07-02 06:44:20,361] [INFO] [config.py:1000:print] flops_profiler_config ........ {
"enabled": false,
"recompute_fwd_factor": 0.0,
"profile_step": 1,
"module_depth": -1,
"top_modules": 1,
"detailed": true,
"output_file": null
}
[2024-07-02 06:44:20,361] [INFO] [config.py:1000:print] fp16_auto_cast ............... None
[2024-07-02 06:44:20,361] [INFO] [config.py:1000:print] fp16_enabled ................. False
[2024-07-02 06:44:20,361] [INFO] [config.py:1000:print] fp16_master_weights_and_gradients False
[2024-07-02 06:44:20,361] [INFO] [config.py:1000:print] global_rank .................. 0
[2024-07-02 06:44:20,361] [INFO] [config.py:1000:print] grad_accum_dtype ............. None
[2024-07-02 06:44:20,361] [INFO] [config.py:1000:print] gradient_accumulation_steps .. 1
[2024-07-02 06:44:20,361] [INFO] [config.py:1000:print] gradient_clipping ............ 1.0
[2024-07-02 06:44:20,362] [INFO] [config.py:1000:print] gradient_predivide_factor .... 1.0
[2024-07-02 06:44:20,362] [INFO] [config.py:1000:print] graph_harvesting ............. False
[2024-07-02 06:44:20,362] [INFO] [config.py:1000:print] hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8
[2024-07-02 06:44:20,362] [INFO] [config.py:1000:print] initial_dynamic_scale ........ 1
[2024-07-02 06:44:20,362] [INFO] [config.py:1000:print] load_universal_checkpoint .... False
[2024-07-02 06:44:20,362] [INFO] [config.py:1000:print] loss_scale ................... 1.0
[2024-07-02 06:44:20,362] [INFO] [config.py:1000:print] memory_breakdown ............. False
[2024-07-02 06:44:20,362] [INFO] [config.py:1000:print] mics_hierarchial_params_gather False
[2024-07-02 06:44:20,362] [INFO] [config.py:1000:print] mics_shard_size .............. -1
[2024-07-02 06:44:20,362] [INFO] [config.py:1000:print] monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') enabled=False
[2024-07-02 06:44:20,362] [INFO] [config.py:1000:print] nebula_config ................ {
"enabled": false,
"persistent_storage_path": null,
"persistent_time_interval": 100,
"num_of_version_in_retention": 2,
"enable_nebula_load": true,
"load_path": null
}
[2024-07-02 06:44:20,362] [INFO] [config.py:1000:print] optimizer_legacy_fusion ...... False
[2024-07-02 06:44:20,362] [INFO] [config.py:1000:print] optimizer_name ............... None
[2024-07-02 06:44:20,362] [INFO] [config.py:1000:print] optimizer_params ............. None
[2024-07-02 06:44:20,362] [INFO] [config.py:1000:print] pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True}
[2024-07-02 06:44:20,362] [INFO] [config.py:1000:print] pld_enabled .................. False
[2024-07-02 06:44:20,362] [INFO] [config.py:1000:print] pld_params ................... False
[2024-07-02 06:44:20,362] [INFO] [config.py:1000:print] prescale_gradients ........... False
[2024-07-02 06:44:20,362] [INFO] [config.py:1000:print] scheduler_name ............... None
[2024-07-02 06:44:20,362] [INFO] [config.py:1000:print] scheduler_params ............. None
[2024-07-02 06:44:20,362] [INFO] [config.py:1000:print] seq_parallel_communication_data_type torch.float32
[2024-07-02 06:44:20,363] [INFO] [config.py:1000:print] sparse_attention ............. None
[2024-07-02 06:44:20,363] [INFO] [config.py:1000:print] sparse_gradients_enabled ..... False
[2024-07-02 06:44:20,363] [INFO] [config.py:1000:print] steps_per_print .............. inf
[2024-07-02 06:44:20,363] [INFO] [config.py:1000:print] train_batch_size ............. 1
[2024-07-02 06:44:20,363] [INFO] [config.py:1000:print] train_micro_batch_size_per_gpu 1
[2024-07-02 06:44:20,363] [INFO] [config.py:1000:print] use_data_before_expert_parallel False
[2024-07-02 06:44:20,363] [INFO] [config.py:1000:print] use_node_local_storage ....... False
[2024-07-02 06:44:20,363] [INFO] [config.py:1000:print] wall_clock_breakdown ......... False
[2024-07-02 06:44:20,363] [INFO] [config.py:1000:print] weight_quantization_config ... None
[2024-07-02 06:44:20,363] [INFO] [config.py:1000:print] world_size ................... 1
[2024-07-02 06:44:20,363] [INFO] [config.py:1000:print] zero_allow_untested_optimizer True
[2024-07-02 06:44:20,363] [INFO] [config.py:1000:print] zero_config .................. stage=3 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=16777216 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=500,000,000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=DeepSpeedZeroOffloadParamConfig(device='cpu', nvme_path=None, buffer_count=5, buffer_size=100,000,000, max_in_cpu=1,000,000,000, pin_memory=True) offload_optimizer=DeepSpeedZeroOffloadOptimizerConfig(device='cpu', nvme_path=None, buffer_count=4, pin_memory=True, pipeline=False, pipeline_read=False, pipeline_write=False, fast_init=False, ratio=1.0) sub_group_size=1000000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=15099494 param_persistence_threshold=40960 model_persistence_threshold=sys.maxsize max_live_parameters=1000000000 max_reuse_distance=1000000000 gather_16bit_weights_on_model_save=True stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True
[2024-07-02 06:44:20,363] [INFO] [config.py:1000:print] zero_enabled ................. True
[2024-07-02 06:44:20,363] [INFO] [config.py:1000:print] zero_force_ds_cpu_optimizer .. True
[2024-07-02 06:44:20,363] [INFO] [config.py:1000:print] zero_optimization_stage ...... 3
[2024-07-02 06:44:20,363] [INFO] [config.py:986:print_user_config] json = {
"train_batch_size": 1,
"train_micro_batch_size_per_gpu": 1,
"gradient_accumulation_steps": 1,
"gradient_clipping": 1.0,
"zero_allow_untested_optimizer": true,
"fp16": {
"enabled": false,
"loss_scale": 0,
"loss_scale_window": 1000,
"initial_scale_power": 16,
"hysteresis": 2,
"min_loss_scale": 1
},
"bf16": {
"enabled": true
},
"zero_optimization": {
"stage": 3,
"offload_optimizer": {
"device": "cpu",
"pin_memory": true
},
"offload_param": {
"device": "cpu",
"pin_memory": true
},
"overlap_comm": true,
"contiguous_gradients": true,
"sub_group_size": 1.000000e+09,
"reduce_bucket_size": 1.677722e+07,
"stage3_prefetch_bucket_size": 1.509949e+07,
"stage3_param_persistence_threshold": 4.096000e+04,
"stage3_max_live_parameters": 1.000000e+09,
"stage3_max_reuse_distance": 1.000000e+09,
"stage3_gather_16bit_weights_on_model_save": true
},
"steps_per_print": inf
}
[INFO|trainer.py:2078] 2024-07-02 06:44:20,364 >> ***** Running training *****
[INFO|trainer.py:2079] 2024-07-02 06:44:20,364 >> Num examples = 43
[INFO|trainer.py:2080] 2024-07-02 06:44:20,364 >> Num Epochs = 1
[INFO|trainer.py:2081] 2024-07-02 06:44:20,364 >> Instantaneous batch size per device = 1
[INFO|trainer.py:2084] 2024-07-02 06:44:20,364 >> Total train batch size (w. parallel, distributed & accumulation) = 1
[INFO|trainer.py:2085] 2024-07-02 06:44:20,364 >> Gradient Accumulation steps = 1
[INFO|trainer.py:2086] 2024-07-02 06:44:20,364 >> Total optimization steps = 43
[INFO|trainer.py:2087] 2024-07-02 06:44:20,366 >> Number of trainable parameters = 1,649,443,841
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 2.5e-05, 'epoch': 0.23}
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 5e-05, 'epoch': 0.47}
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 5e-05, 'epoch': 0.7}
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 5e-05, 'epoch': 0.93}
100%|███████████████████████████████████████████████████████████████████████████████████| 43/43 [05:38<00:00, 7.78s/it][INFO|trainer.py:3719] 2024-07-02 06:49:58,959 >> ***** Running Evaluation *****
[INFO|trainer.py:3721] 2024-07-02 06:49:58,960 >> Num examples = 5
[INFO|trainer.py:3724] 2024-07-02 06:49:58,960 >> Batch size = 1
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [0,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
Traceback (most recent call last):
File "/home/LLaMA-Factory-240526/src/train.py", line 14, in
main()
File "/home/LLaMA-Factory-240526/src/train.py", line 5, in main
run_exp()
File "/home/LLaMA-Factory-240526/src/llamafactory/train/tuner.py", line 49, in run_exp
run_rm(model_args, data_args, training_args, finetuning_args, callbacks)
File "/home/LLaMA-Factory-240526/src/llamafactory/train/rm/workflow.py", line 51, in run_rm
train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint)
File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 1885, in train
return inner_training_loop(
File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 2311, in _inner_training_loop
self._maybe_log_save_evaluate(tr_loss, grad_norm, model, trial, epoch, ignore_keys_for_eval)
File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 2721, in _maybe_log_save_evaluate
metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 3572, in evaluate
output = eval_loop(
File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 3766, in evaluation_loop
losses = self.gather_function((loss.repeat(batch_size)))
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:44 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fe5be2fa617 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7fe5be2b598d in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7fe5be3ab838 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: + 0x1f586 (0x7fe5be374586 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #4: + 0x222bd (0x7fe5be3772bd in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #5: + 0x226dd (0x7fe5be3776dd in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #6: + 0x519446 (0x7fe61d7b0446 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #7: + 0x55ca7 (0x7fe5be2dfca7 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #8: c10::TensorImpl::~TensorImpl() + 0x1e3 (0x7fe5be2d7cb3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #9: c10::TensorImpl::~TensorImpl() + 0x9 (0x7fe5be2d7e49 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #10: + 0x7c9518 (0x7fe61da60518 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #11: THPVariable_subclass_dealloc(_object*) + 0x305 (0x7fe61da608a5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #12: /opt/conda/bin/python() [0x4f5d58]
frame #13: /opt/conda/bin/python() [0x505859]
frame #14: /opt/conda/bin/python() [0x5058bb]
frame #15: /opt/conda/bin/python() [0x5058bb]
frame #16: /opt/conda/bin/python() [0x5058bb]
frame #17: /opt/conda/bin/python() [0x5058bb]
frame #18: /opt/conda/bin/python() [0x5058bb]
frame #19: /opt/conda/bin/python() [0x4deed2]
frame #20: PyDict_SetItemString + 0x52 (0x4e20a2 in /opt/conda/bin/python)
frame #21: /opt/conda/bin/python() [0x5c3dd4]
frame #22: Py_FinalizeEx + 0x143 (0x5c2ad3 in /opt/conda/bin/python)
frame #23: Py_RunMain + 0x109 (0x5b48f9 in /opt/conda/bin/python)
frame #24: Py_BytesMain + 0x39 (0x584e49 in /opt/conda/bin/python)
frame #25: __libc_start_main + 0xf3 (0x7fe6496c4083 in /lib/x86_64-linux-gnu/libc.so.6)
frame #26: /opt/conda/bin/python() [0x584cfe]

Expected behavior

请教一下如何解决CUDA error: device-side assert triggered问题？

Others

No response

[hiyouga]

无法复现。

[XinyuanDong]

遇到了同样的问题

[hiyouga]

更新代码

[liduang]

0.9.1.dev0 模型评估也出现了这个问题，咋解决啊
[WARNING|logging.py:328] 2024-10-30 02:30:47,459 >> Starting from v4.46, the logits model output will have the same type as the model (except at train time, where it will always be FP32)
Processing subjects: 9%|███████▎ | 6/67 [15:33<2:38:12, 155.62s/it, 中国公务员考试]
Traceback (most recent call last):
File "/home/hicon/miniconda3/envs/llama_factory/bin/llamafactory-cli", line 8, in
sys.exit(main())
File "/home/hicon/LLaMA-Factory/src/llamafactory/cli.py", line 85, in main
run_eval()
File "/home/hicon/LLaMA-Factory/src/llamafactory/eval/evaluator.py", line 158, in run_eval
Evaluator().eval()
File "/home/hicon/LLaMA-Factory/src/llamafactory/eval/evaluator.py", line 127, in eval
preds = self.batch_inference(batch_input)
File "/home/hicon/miniconda3/envs/llama_factory/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/home/hicon/LLaMA-Factory/src/llamafactory/eval/evaluator.py", line 73, in batch_inference
logits = self.model(**batch_input).logits
File "/home/hicon/miniconda3/envs/llama_factory/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/hicon/miniconda3/envs/llama_factory/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "/home/hicon/miniconda3/envs/llama_factory/lib/python3.10/site-packages/accelerate/hooks.py", line 170, in new_forward
output = module._old_forward(*args, **kwargs)
File "/home/hicon/miniconda3/envs/llama_factory/lib/python3.10/site-packages/transformers/models/qwen2/modeling_qwen2.py", line 1187, in forward
logits = self.lm_head(hidden_states[:, -num_logits_to_keep:, :]).float()
RuntimeError: CUDA error: unspecified launch failure
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.