Cannot train super low res videos without OOM on Hunyuan

[devcheetah]

I am on Ubuntu, I have a 4090. Made sure every other process that uses the GPU is off. My dataset consist of 2 super low res videos. 288 x 162 and cropped really short. (Before this super small data set I tried 512)

I can however, run images with the same config file just fine with over 40 captioned 1024 res images.

The end result is just running out of memory after caching.

Here is the config - as you can see its basically just the example config edited with my paths.

output_dir = 'outputs'
dataset = 'training/dataset.toml'
epochs = 30
micro_batch_size_per_gpu = 1
pipeline_stages = 1
gradient_accumulation_steps = 4
gradient_clipping = 1.0
warmup_steps = 100

eval_every_n_epochs = 1
eval_before_first_step = true
eval_micro_batch_size_per_gpu = 1
eval_gradient_accumulation_steps = 1

save_every_n_epochs = 2
checkpoint_every_n_minutes = 120

activation_checkpointing = true
partition_method = 'parameters'
save_dtype = 'bfloat16'
caching_batch_size = 1
steps_per_print = 1
video_clip_mode = 'single_middle'

[model]
type = 'hunyuan-video'
transformer_path = '/home/me/Documents/AI/ComfyUI/models/diffusion_models/hunyuan_video_720_cfgdistill_fp8_e4m3fn.safetensors'
vae_path = '/home/me/Documents/AI/ComfyUI/models/vae/hunyuanVideoSafetensors_vae.safetensors'
llm_path = '/home/me/Documents/AI/ComfyUI/models/text_encoders/llava-llama-3-8b-text-encoder-tokenizer'
clip_path = '/home/me/Documents/AI/ComfyUI/models/clip/clip-vit-large-patch14'
dtype = 'bfloat16'
transformer_dtype = 'float8'
timestep_sample_method = 'logit_normal'

[adapter]
type = 'lora'
rank = 32
dtype = 'bfloat16'

[optimizer]
type = 'adamw_optimi'
lr = 2e-5
betas = [0.9, 0.99]
weight_decay = 0.01
eps = 1e-8`

And here is an example output log

[2024-12-23 10:06:17,868] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-12-23 10:06:18,960] [WARNING] [runner.py:215:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only. [2024-12-23 10:06:18,960] [INFO] [runner.py:607:main] cmd = /home/tyler/miniconda3/envs/diffusion-pipe/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMF19 --master_addr=127.0.0.1 --master_port=29500 --enable_each_rank_log=None train.py --deepspeed --config training/hunyuan_video.toml [2024-12-23 10:06:19,710] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-12-23 10:06:20,775] [INFO] [launch.py:139:main] 0 NCCL_IB_DISABLE=1 [2024-12-23 10:06:20,775] [INFO] [launch.py:139:main] 0 NCCL_P2P_DISABLE=1 [2024-12-23 10:06:20,775] [INFO] [launch.py:146:main] WORLD INFO DICT: {'localhost': [0]} [2024-12-23 10:06:20,775] [INFO] [launch.py:152:main] nnodes=1, num_local_procs=1, node_rank=0 [2024-12-23 10:06:20,775] [INFO] [launch.py:163:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0]}) [2024-12-23 10:06:20,775] [INFO] [launch.py:164:main] dist_world_size=1 [2024-12-23 10:06:20,775] [INFO] [launch.py:168:main] Setting CUDA_VISIBLE_DEVICES=0 [2024-12-23 10:06:20,776] [INFO] [launch.py:256:main] process 71649 spawned with command: ['/home/tyler/miniconda3/envs/diffusion-pipe/bin/python', '-u', 'train.py', '--local_rank=0', '--deepspeed', '--config', 'training/hunyuan_video.toml'] [2024-12-23 10:06:21,566] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-12-23 10:06:23,011] [INFO] [comm.py:652:init_distributed] cdb=None [2024-12-23 10:06:23,011] [INFO] [comm.py:683:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl The config attributes {'tile_sample_min_size': 256} were passed to AutoencoderKLCausal3D, but are not expected and will be ignored. Please verify your config.json configuration file. 2024-12-23 10:06:24.050 | INFO | utils.patches:load_text_encoder:64 - Loading text encoder model (llm) from: /home/tyler/Documents/AI/ComfyUI/models/text_encoders/llava-llama-3-8b-text-encoder-tokenizer Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:03<00:00, 1.21it/s] 2024-12-23 10:06:27.458 | INFO | utils.patches:load_text_encoder:87 - Text encoder to dtype: torch.bfloat16 2024-12-23 10:06:27.459 | INFO | hyvideo.text_encoder:load_tokenizer:64 - Loading tokenizer (llm) from: /home/tyler/Documents/AI/ComfyUI/models/text_encoders/llava-llama-3-8b-text-encoder-tokenizer 2024-12-23 10:06:27.648 | INFO | utils.patches:load_text_encoder:64 - Loading text encoder model (clipL) from: /home/tyler/Documents/AI/ComfyUI/models/clip/clip-vit-large-patch14 2024-12-23 10:06:27.892 | INFO | utils.patches:load_text_encoder:87 - Text encoder to dtype: torch.bfloat16 2024-12-23 10:06:27.893 | INFO | hyvideo.text_encoder:load_tokenizer:64 - Loading tokenizer (clipL) from: /home/tyler/Documents/AI/ComfyUI/models/clip/clip-vit-large-patch14 using video_clip_mode=single_middle [2024-12-23 10:06:28,404] [WARNING] [dataset.py:258:cache_metadata] Image file training/micro/22.mp4 does not have corresponding caption file. caching metadata num_proc must be <= 2. Reducing num_proc to 2 for dataset of size 2. Map: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 1118.93 examples/s] caching latents: training/micro caching latents: (0.6299605249474366, 65) caching latents: (416, 640, 65) num_proc must be <= 2. Reducing num_proc to 2 for dataset of size 2. caching text embeddings: (0.6299605249474366, 65) num_proc must be <= 2. Reducing num_proc to 2 for dataset of size 2. caching text embeddings: (0.6299605249474366, 65) num_proc must be <= 2. Reducing num_proc to 2 for dataset of size 2. [2024-12-23 10:06:28,588] [WARNING] [dataset.py:258:cache_metadata] Image file training/micro/22.mp4 does not have corresponding caption file. caching metadata num_proc must be <= 2. Reducing num_proc to 2 for dataset of size 2. Map: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 1256.34 examples/s] caching latents: training/micro caching latents: (0.6299605249474366, 65) caching latents: (416, 640, 65) num_proc must be <= 2. Reducing num_proc to 2 for dataset of size 2. caching text embeddings: (0.6299605249474366, 65) num_proc must be <= 2. Reducing num_proc to 2 for dataset of size 2. caching text embeddings: (0.6299605249474366, 65) num_proc must be <= 2. Reducing num_proc to 2 for dataset of size 2. trainable params: 161,218,560 || all params: 12,982,231,104 || trainable%: 1.2418 SEED_LAYERS=False BASE_SEED=1234 SEED_FN=None Using topology: {ProcessCoord(pipe=0, data=0): 0} [2024-12-23 10:06:29,883] [INFO] [module.py:396:_partition_layers] Partitioning pipeline stages with method parameters stage=0 layers=63 0: InitialLayer 1: DoubleBlock 2: DoubleBlock 3: DoubleBlock 4: DoubleBlock 5: DoubleBlock 6: DoubleBlock 7: DoubleBlock 8: DoubleBlock 9: DoubleBlock 10: DoubleBlock 11: DoubleBlock 12: DoubleBlock 13: DoubleBlock 14: DoubleBlock 15: DoubleBlock 16: DoubleBlock 17: DoubleBlock 18: DoubleBlock 19: DoubleBlock 20: DoubleBlock 21: concatenate_hidden_states 22: SingleBlock 23: SingleBlock 24: SingleBlock 25: SingleBlock 26: SingleBlock 27: SingleBlock 28: SingleBlock 29: SingleBlock 30: SingleBlock 31: SingleBlock 32: SingleBlock 33: SingleBlock 34: SingleBlock 35: SingleBlock 36: SingleBlock 37: SingleBlock 38: SingleBlock 39: SingleBlock 40: SingleBlock 41: SingleBlock 42: SingleBlock 43: SingleBlock 44: SingleBlock 45: SingleBlock 46: SingleBlock 47: SingleBlock 48: SingleBlock 49: SingleBlock 50: SingleBlock 51: SingleBlock 52: SingleBlock 53: SingleBlock 54: SingleBlock 55: SingleBlock 56: SingleBlock 57: SingleBlock 58: SingleBlock 59: SingleBlock 60: SingleBlock 61: SingleBlock 62: OutputLayer [2024-12-23 10:06:31,431] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed info: version=0.16.2, git-hash=unknown, git-branch=unknown [2024-12-23 10:06:31,431] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 1 [2024-12-23 10:06:31,484] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False [2024-12-23 10:06:31,486] [INFO] [logging.py:128:log_dist] [Rank 0] Using client callable to create basic optimizer [2024-12-23 10:06:31,486] [INFO] [logging.py:128:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer [2024-12-23 10:06:31,522] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Basic Optimizer = AdamW [2024-12-23 10:06:31,522] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Final Optimizer = AdamW [2024-12-23 10:06:31,522] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed using configured LR scheduler = None [2024-12-23 10:06:31,522] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed LR Scheduler = None [2024-12-23 10:06:31,522] [INFO] [logging.py:128:log_dist] [Rank 0] step=0, skipped=0, lr=[2e-05], mom=[0.0] [2024-12-23 10:06:31,526] [INFO] [config.py:999:print] DeepSpeedEngine configuration: [2024-12-23 10:06:31,526] [INFO] [config.py:1003:print] activation_checkpointing_config { "partition_activations": false, "contiguous_memory_optimization": false, "cpu_checkpointing": false, "number_checkpoints": null, "synchronize_checkpoint_boundary": false, "profile": false } [2024-12-23 10:06:31,526] [INFO] [config.py:1003:print] aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True, 'use_gds': False} [2024-12-23 10:06:31,526] [INFO] [config.py:1003:print] amp_enabled .................. False [2024-12-23 10:06:31,526] [INFO] [config.py:1003:print] amp_params ................... False [2024-12-23 10:06:31,526] [INFO] [config.py:1003:print] autotuning_config ............ { "enabled": false, "start_step": null, "end_step": null, "metric_path": null, "arg_mappings": null, "metric": "throughput", "model_info": null, "results_dir": "autotuning_results", "exps_dir": "autotuning_exps", "overwrite": true, "fast": true, "start_profile_step": 3, "end_profile_step": 5, "tuner_type": "gridsearch", "tuner_early_stopping": 5, "tuner_num_trials": 50, "model_info_path": null, "mp_size": 1, "max_train_batch_size": null, "min_train_batch_size": 1, "max_train_micro_batch_size_per_gpu": 1.024000e+03, "min_train_micro_batch_size_per_gpu": 1, "num_tuning_micro_batch_sizes": 3 } [2024-12-23 10:06:31,526] [INFO] [config.py:1003:print] bfloat16_enabled ............. False [2024-12-23 10:06:31,526] [INFO] [config.py:1003:print] bfloat16_immediate_grad_update False [2024-12-23 10:06:31,526] [INFO] [config.py:1003:print] checkpoint_parallel_write_pipeline False [2024-12-23 10:06:31,526] [INFO] [config.py:1003:print] checkpoint_tag_validation_enabled True [2024-12-23 10:06:31,526] [INFO] [config.py:1003:print] checkpoint_tag_validation_fail False [2024-12-23 10:06:31,526] [INFO] [config.py:1003:print] comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x766659f34f20> [2024-12-23 10:06:31,526] [INFO] [config.py:1003:print] communication_data_type ...... None [2024-12-23 10:06:31,526] [INFO] [config.py:1003:print] compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}} [2024-12-23 10:06:31,526] [INFO] [config.py:1003:print] curriculum_enabled_legacy .... False [2024-12-23 10:06:31,526] [INFO] [config.py:1003:print] curriculum_params_legacy ..... False [2024-12-23 10:06:31,526] [INFO] [config.py:1003:print] data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}} [2024-12-23 10:06:31,526] [INFO] [config.py:1003:print] data_efficiency_enabled ...... False [2024-12-23 10:06:31,526] [INFO] [config.py:1003:print] dataloader_drop_last ......... False [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] disable_allgather ............ False [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] dump_state ................... False [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] dynamic_loss_scale_args ...... None [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] eigenvalue_enabled ........... False [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] eigenvalue_gas_boundary_resolution 1 [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] eigenvalue_layer_name ........ bert.encoder.layer [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] eigenvalue_layer_num ......... 0 [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] eigenvalue_max_iter .......... 100 [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] eigenvalue_stability ......... 1e-06 [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] eigenvalue_tol ............... 0.01 [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] eigenvalue_verbose ........... False [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] elasticity_enabled ........... False [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] flops_profiler_config ........ { "enabled": false, "recompute_fwd_factor": 0.0, "profile_step": 1, "module_depth": -1, "top_modules": 1, "detailed": true, "output_file": null } [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] fp16_auto_cast ............... None [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] fp16_enabled ................. False [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] fp16_master_weights_and_gradients False [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] global_rank .................. 0 [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] grad_accum_dtype ............. None [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] gradient_accumulation_steps .. 4 [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] gradient_clipping ............ 1.0 [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] gradient_predivide_factor .... 1.0 [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] graph_harvesting ............. False [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8 [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] initial_dynamic_scale ........ 65536 [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] load_universal_checkpoint .... False [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] loss_scale ................... 0 [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] memory_breakdown ............. False [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] mics_hierarchial_params_gather False [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] mics_shard_size .............. -1 [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') comet=CometConfig(enabled=False, samples_log_interval=100, project=None, workspace=None, api_key=None, experiment_name=None, experiment_key=None, online=None, mode=None) wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] nebula_config ................ { "enabled": false, "persistent_storage_path": null, "persistent_time_interval": 100, "num_of_version_in_retention": 2, "enable_nebula_load": true, "load_path": null } [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] optimizer_legacy_fusion ...... False [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] optimizer_name ............... None [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] optimizer_params ............. None [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True} [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] pld_enabled .................. False [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] pld_params ................... False [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] prescale_gradients ........... False [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] scheduler_name ............... None [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] scheduler_params ............. None [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] seq_parallel_communication_data_type torch.float32 [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] sparse_attention ............. None [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] sparse_gradients_enabled ..... False [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] steps_per_print .............. 1 [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] timers_config ................ enabled=True synchronized=True [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] train_batch_size ............. 4 [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] train_micro_batch_size_per_gpu 1 [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] use_data_before_expert_parallel_ False [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] use_node_local_storage ....... False [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] wall_clock_breakdown ......... False [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] weight_quantization_config ... None [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] world_size ................... 1 [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] zero_allow_untested_optimizer False [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] zero_config .................. stage=0 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=500000000 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=500000000 overlap_comm=False load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=None sub_group_size=1000000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50000000 param_persistence_threshold=100000 model_persistence_threshold=9223372036854775807 max_live_parameters=1000000000 max_reuse_distance=1000000000 gather_16bit_weights_on_model_save=False module_granularity_threshold=0 use_all_reduce_for_fetch_params=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False zeropp_loco_param=None mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] zero_enabled ................. False [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] zero_force_ds_cpu_optimizer .. True [2024-12-23 10:06:31,527] [INFO] [config.py:1003:print] zero_optimization_stage ...... 0 [2024-12-23 10:06:31,527] [INFO] [config.py:989:print_user_config] json = { "train_micro_batch_size_per_gpu": 1, "gradient_accumulation_steps": 4, "gradient_clipping": 1.0, "steps_per_print": 1 } [2024-12-23 10:06:31,527] [INFO] [engine.py:105:__init__] CONFIG: micro_batches=4 micro_batch_size=1 [2024-12-23 10:06:31,527] [INFO] [engine.py:146:__init__] is_pipe_partitioned= False is_grad_partitioned= False [2024-12-23 10:06:31,548] [INFO] [engine.py:165:__init__] RANK=0 STAGE=0 LAYERS=63 [0, 63) STAGE_PARAMS=161218560 (161.219M) TOTAL_PARAMS=161218560 (161.219M) UNIQUE_PARAMS=161218560 (161.219M) [rank0]: Traceback (most recent call last): [rank0]: File "/home/tyler/Documents/AI/diffusion-pipe/train.py", line 458, in <module> [rank0]: loss = model_engine.train_batch().item() [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/home/tyler/miniconda3/envs/diffusion-pipe/lib/python3.12/site-packages/deepspeed/runtime/pipe/engine.py", line 389, in train_batch [rank0]: self._exec_schedule(sched) [rank0]: File "/home/tyler/miniconda3/envs/diffusion-pipe/lib/python3.12/site-packages/deepspeed/runtime/pipe/engine.py", line 1422, in _exec_schedule [rank0]: self._exec_instr(**cmd.kwargs) [rank0]: File "/home/tyler/miniconda3/envs/diffusion-pipe/lib/python3.12/site-packages/deepspeed/runtime/pipe/engine.py", line 735, in _exec_forward_pass [rank0]: outputs = super().forward(inputs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/home/tyler/miniconda3/envs/diffusion-pipe/lib/python3.12/site-packages/deepspeed/utils/nvtx.py", line 18, in wrapped_fn [rank0]: ret_val = func(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/home/tyler/miniconda3/envs/diffusion-pipe/lib/python3.12/site-packages/deepspeed/runtime/engine.py", line 1914, in forward [rank0]: loss = self.module(*inputs, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/home/tyler/miniconda3/envs/diffusion-pipe/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl [rank0]: return self._call_impl(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/home/tyler/miniconda3/envs/diffusion-pipe/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl [rank0]: return forward_call(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/home/tyler/miniconda3/envs/diffusion-pipe/lib/python3.12/site-packages/deepspeed/runtime/pipe/module.py", line 386, in forward [rank0]: x = self.activation_checkpoint_func(exec_range_func(start_idx, end_idx), *x) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/home/tyler/miniconda3/envs/diffusion-pipe/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 600, in _fn [rank0]: return fn(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^ [rank0]: File "/home/tyler/miniconda3/envs/diffusion-pipe/lib/python3.12/site-packages/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 953, in checkpoint [rank0]: CheckpointFunction.apply(function, all_outputs, *args) [rank0]: File "/home/tyler/miniconda3/envs/diffusion-pipe/lib/python3.12/site-packages/torch/autograd/function.py", line 574, in apply [rank0]: return super().apply(*args, **kwargs) # type: ignore[misc] [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/home/tyler/miniconda3/envs/diffusion-pipe/lib/python3.12/site-packages/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 544, in forward [rank0]: outputs = run_function(*inputs_cuda) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/home/tyler/miniconda3/envs/diffusion-pipe/lib/python3.12/site-packages/deepspeed/runtime/pipe/module.py", line 363, in exec_func [rank0]: inputs = layer(inputs) [rank0]: ^^^^^^^^^^^^^ [rank0]: File "/home/tyler/miniconda3/envs/diffusion-pipe/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl [rank0]: return self._call_impl(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/home/tyler/miniconda3/envs/diffusion-pipe/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl [rank0]: return forward_call(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/home/tyler/miniconda3/envs/diffusion-pipe/lib/python3.12/site-packages/torch/amp/autocast_mode.py", line 43, in decorate_autocast [rank0]: return func(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/home/tyler/Documents/AI/diffusion-pipe/models/hunyuan_video.py", line 592, in forward [rank0]: x = self.block(x, vec, txt_seq_len.item(), cu_seqlens, cu_seqlens, max_seqlen.item(), max_seqlen.item(), (freqs_cos, freqs_sin)) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/home/tyler/miniconda3/envs/diffusion-pipe/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl [rank0]: return self._call_impl(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/home/tyler/miniconda3/envs/diffusion-pipe/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl [rank0]: return forward_call(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/home/tyler/Documents/AI/diffusion-pipe/submodules/HunyuanVideo/hyvideo/modules/models.py", line 340, in forward [rank0]: self.linear1(x_mod), [3 * self.hidden_size, self.mlp_hidden_dim], dim=-1 [rank0]: ^^^^^^^^^^^^^^^^^^^ [rank0]: File "/home/tyler/miniconda3/envs/diffusion-pipe/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl [rank0]: return self._call_impl(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/home/tyler/miniconda3/envs/diffusion-pipe/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl [rank0]: return forward_call(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/home/tyler/miniconda3/envs/diffusion-pipe/lib/python3.12/site-packages/peft/tuners/lora/layer.py", line 621, in forward [rank0]: result = result + lora_B(lora_A(dropout(x))) * scaling [rank0]: ~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~ [rank0]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 736.00 MiB. GPU 0 has a total capacity of 23.54 GiB of which 732.31 MiB is free. Process 3676 has 384.00 MiB memory in use. Including non-PyTorch memory, this process has 21.70 GiB memory in use. Of the allocated memory 18.09 GiB is allocated by PyTorch, and 2.90 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) [2024-12-23 10:06:45,780] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 71649 [2024-12-23 10:06:45,780] [ERROR] [launch.py:325:sigkill_handler] ['/home/tyler/miniconda3/envs/diffusion-pipe/bin/python', '-u', 'train.py', '--local_rank=0', '--deepspeed', '--config', 'training/hunyuan_video.toml'] exits with return code = 1

And finally, the execution command.

NCCL_P2P_DISABLE="1" NCCL_IB_DISABLE="1" deepspeed --num_gpus=1 train.py --deepspeed --config training/hunyuan_video.toml

[tdrussell]

caching latents: (416, 640, 65)

It's using 512^2 resolution and assigning the videos to frame bucket 65. This will OOM on a single 4090.

It doesn't matter what resolution the video files on disk are. They are assigned to the closest aspect ratio bucket, then assigned to the closest frame bucket such that the video is >= that number of frames, then resized to the resolution(s) you configured.

If you set frame_buckets=[1, 33] and keep resolutions=[512], that should barely fit on a 4090. If you lowered the resolution, longer videos could fit, but I don't know what the exact tradeoff is.

[devcheetah]

caching latents: (416, 640, 65)

It's using 512^2 resolution and assigning the videos to frame bucket 65. This will OOM on a single 4090.

It doesn't matter what resolution the video files on disk are. They are assigned to the closest aspect ratio bucket, then assigned to the closest frame bucket such that the video is >= that number of frames, then resized to the resolution(s) you configured.

If you set frame_buckets=[1, 33] and keep resolutions=[512], that should barely fit on a 4090. If you lowered the resolution, longer videos could fit, but I don't know what the exact tradeoff is.

This makes sense. I cropped videos and set up the config as you suggested. Even with one small, short video it goes OOM before the second epoch. I'm thinking I'll just wait for video training for a while.