FIX: Error in forward of 4bit linear lora layer

[BenjaminBossan]

This was introduced during the refactoring of the forward function. It should now be fixed and be equivalent to the forward function before the refactoring:

peft/src/peft/tuners/lora.py

Lines 1207 to 1231 in 4df9c5a

	def forward(self, x: torch.Tensor):
	result = super().forward(x)
	if self.disable_adapters or self.active_adapter not in self.lora_A.keys():
	return result
	elif self.r[self.active_adapter] > 0:
	result = result.clone()
	if not torch.is_autocast_enabled():
	expected_dtype = result.dtype
	x = x.to(self.lora_A[self.active_adapter].weight.dtype)
	output = (
	self.lora_B[self.active_adapter](
	self.lora_A[self.active_adapter](self.lora_dropout[self.active_adapter](x))
	).to(expected_dtype)
	* self.scaling[self.active_adapter]
	)
	else:
	output = (
	self.lora_B[self.active_adapter](
	self.lora_A[self.active_adapter](self.lora_dropout[self.active_adapter](x))
	)
	* self.scaling[self.active_adapter]
	)
	result += output
	return result

Bug reported by @jiqing-feng

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[younesbelkada]

Thanks for the fix!