Example Question (got error) : Try new 40B LLMs demo in Kaggle

[YooSungHyun]

i just running jupyter example.
i got error like this

  File "/ssd/data01/ysh/test/.venv/lib/python3.10/site-packages/peft/tuners/lora.py", line 565, in forward
    result = F.linear(x, transpose(self.weight, self.fan_in_fan_out), bias=self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (6x8192 and 1x18874368)

i used A100 80GB * 2

and if i want to using huggingface trainer, how can i use? just change, for train loop to huggingface trainer?

[YooSungHyun]

this is so weird, lora_A and B is weight shape is good, but, self.weight.shape is weird.
this looks like, when another devices values is calculated torch.Size([18874368, 1])
self.weight is on cuda:0 and value is not nan and some kind of 4bit things......

qlora is so hard to me 😢

[YooSungHyun]

hiyouga/LLaMA-Factory#15
is work for me

pip install -U git+https://github.com/huggingface/peft.git