Bug in decoder_padding_mask in BPE training

[csukuangfj]

See the code below:

snowfall/snowfall/models/transformer.py

Line 162 in 3502531

eos_id = self.decoder_num_class - 1

snowfall/snowfall/models/transformer.py

Line 167 in 3502531

ys_in_pad = pad_list(ys_in, eos_id)

snowfall/snowfall/models/transformer.py

Line 179 in 3502531

tgt_key_padding_mask = decoder_padding_mask(ys_in_pad)

snowfall/snowfall/models/transformer.py

Line 709 in 3502531

def decoder_padding_mask(ys_pad: Tensor, ignore_id: int = -1) -> Tensor:

snowfall/snowfall/models/transformer.py

Lines 720 to 721 in 3502531

	ys_mask = ys_pad == ignore_id
	return ys_mask

---

You can see that ys_in_pad is padded with eos_id, which is a positive word piece ID.

However, it is using -1 to compute the mask for ys_in_pad.

---

This bug may explain why the WERs differ with respect to batch size. It also affects the training, I guess.