support advanced attention implementations (FA3, FlashInfer, xformers, etc.)

[feifeibear]

The xdit with parallel degree>2 will execute the attention in the following line in unified SP. The exactly position to execute attention is in the ring_attention's implementation. Because USP applies the Ulysses outside of the Ring.

xDiT/xfuser/core/long_ctx_attention/ring/ring_flash_attn.py

Line 75 in f9e35f7

block_out, _, _, _, _, block_lse, _, _ = _flash_attn_forward(

[antferdom]

Tritonbench: [performance] Torch SDPA cuDNN backend vs FlashAttention v3 #41