Update lora_quantization_layers.py

[tugang-baidu]

Fix parallel QLoRA in reference to paddlenlp/quantization/quantization_linear.py

The original lora_quantization_layers.py is in paddlenlp/peft/lora

In class QuantizationLoRABaseLinear:
In method init:
insert codes '
self.state = 0
if self.weight_quantize_algo in ["a8w8linear", "a8w4linear", "fp8linear"]:
self.act_scale = self.create_parameter(
shape=[1],
dtype=self._dtype,
is_bias=False,
default_initializer=nn.initializer.Constant(value=0.0),
)
self.act_scale.is_distributed = False
self.act_scale.stop_gradient = True
self.group = get_act_scale_group(is_row=True)
else:
raise NotImplementedError(
f"Not supported weight_quantize_algo {self.weight_quantize_algo}"
)
'
between 'self.bias = layer.bias' and 'self.lora_config = lora_config'

In method forward:
insert 'act_state=(self.state, self.training, self.act_scale, self.group)' in the parameter list of 'output=quant_weight_linear'

insert codes '
if self.training:
self.state += 1
'
before 'return output'

However, after such change, in different cases, I found that loss would start to converge with different beginnings (I have deleted all checkpoints every time I start a new case):

1. No parallelism: 7.68577909

2. tensor parallelism with paddle.distributed.launch: 14.6875057

3. paddle.distributed.launch: 7.88057899

4. pipeline parallelism with paddle.distributed.launch: 13.96667099

[paddle-bot]

Thanks for your contribution!

[CLAassistant]

All committers have signed the CLA.