NaN Values in Generated Tensor with Text-Video-Image-to-Video Model

[wenhao728]

When using the Text-Video-Image-to-Video version of this model with the commands specified in the README (as below), the generated tensor includes nan values.

CCEdit/README.md

Lines 63 to 77 in 303f7f1

	python scripts/sampling/sampling_tv2v_ref.py \
	--seed 201574 \
	--config_path configs/inference_ccedit/keyframe_ref_cp_no2ndca_add_cfca_depthzoe.yaml \
	--ckpt_path models/tvi2v-no2ndca-depthmidas.ckpt \
	--H 512 --W 768 --original_fps 18 --target_fps 6 --num_keyframes 17 --batch_size 1 --num_samples 2 \
	--sample_steps 50 --sampler_name DPMPP2SAncestralSampler --cfg_scale 7 \
	--prompt 'A person walks on the grass, the Milky Way is in the sky, night' \
	--add_prompt 'masterpiece, best quality,' \
	--video_path assets/Samples/tshirtman.mp4 \
	--reference_path assets/Samples/tshirtman-milkyway.png \
	--save_path outputs/tvi2v/tshirtman-MilkyWay \
	--disable_check_repeat \
	--prior_coefficient_x 0.03 \
	--prior_type ref
	```

Debug Steps

1. Printed out all tensors' shapes and ranges.
2. Identified that the issue occurs in model.conditioner.get_unconditional_conditioning.
3. Observed that control_hint looks normal before model.conditioner.get_unconditional_conditioning but contains NaN values after.

CCEdit/scripts/sampling/sampling_tv2v_ref.py

Lines 377 to 395 in 303f7f1

	batch = {
	"txt": prompts,
	"control_hint": control_hint,
	'cond_img': ref,
	}
	negative_prompt = args.negative_prompt
	batch_uc = {
	"txt": [negative_prompt for _ in range(bs)],
	"control_hint": batch["control_hint"].clone(), # balance mode in controlnet-webui
	'cond_img': batch["cond_img"].clone(), # follow the balance mode
	}
	# batch["txt"] = ["masterpiece, best quality, " + each for each in batch["txt"]]
	if args.add_prompt:
	batch["txt"] = [args.add_prompt + ", " + each for each in batch["txt"]]
	c, uc = model.conditioner.get_unconditional_conditioning(
	batch_c=batch,
	batch_uc=batch_uc,
	)

Input tensor of model.conditioner.get_unconditional_conditioning:

2024-07-17 13:15:15,718 [DEBUG] Control hint shape: torch.Size([1, 3, 24, 512, 512]), range [-1.0, 1.0]

Output tensor of model.conditioner.get_unconditional_conditioning:

2024-07-17 13:15:20,593 [DEBUG] Conditioning control_hint shape: torch.Size([1, 3, 24, 512, 512]), range [nan, nan]

Warning Information

When loading the checkpoint tvi2v-no2ndca-depthmidas.ckpt, the following warning was observed:

Restored from /mnt/data1/pretrained/community/CCEdit/tvi2v-no2ndca-depthmidas.ckpt with 535 missing and 369 unexpected keys
Missing Keys: ['conditioner.embedders.1.model.core.core.pretrained.model.cls_token', 'conditioner.embedders.1.model.core.core.pretrained.model.patch_embed.proj.weight', 'conditioner.embedders.1.model.core.core.pretrained.model.patch_embed.proj.bias',  ...]

Unexpected Keys: ['conditioner.embedders.0.transformer.text_model.embeddings.position_ids', 'conditioner.embedders.1.model.model.pretrained.model.cls_token', 'conditioner.embedders.1.model.model.pretrained.model.pos_embed', 'conditioner.embedders.1.model.model.pretrained.model.patch_embed.backbone.stem.conv.weight', 'conditioner.embedders.1.model.model.pretrained.model.patch_embed.backbone.stem.norm.weight', ...]

Potential Cause

It appears that the Zoe depth conditioner (specified in the config files) weights cannot be loaded. The provided checkpoints are using the Midas depth conditioner, which is also implied by the checkpoint name.

CCEdit/configs/inference_ccedit/keyframe_ref_cp_no2ndca_add_cfca_depthzoe.yaml

Line 107 in 303f7f1

target: sgm.modules.encoders.modules.DepthZoeEncoder

Could you please verify if there's a mismatch between the config files and the checkpoint? If so, guidance on how to resolve this discrepancy would be appreciated.