After switching to Flash Attention, why do we not mask the attention?? Wouldn't it behave like en encoder?

[TejasKalsait]

We register a masking bias when coding our attention block to avoid future tokens from communicating. (Acting as a decoder). However, when we switch to using Flash Attention, torch.nn.functional.scaled_dot_product_attention (q, k, v) we do not assign anything to the attn_mask attribute which is set to None by default. Will this not make it behave like an encoder now?

Am I missing something or no one noticed it?

[TejasKalsait]

Got it! the is_casual = True applies the mask in the background.