[Encoder Decoder] Add flash_attn kernel support for encoder-decoder models

[sroy745]

This PR adds support for flash attention kernel for encoder decoder models. For encoder-decoder models with dtype=bfloat16 the default backend choice is now FlashAttention instead of XFormers. However for llama-3.2-11b-vision-instruct we still use the Xformers backend even with dtype=bfloat16 because the model implementation (models/mllama.py) has dependency on PagedAttention.

For adding this support, we make the following changes in this pr

1. Updated flash_attn.py to add support for encoder-decoder models. Also updated the tests in tests/kernels/test_encoder_decoder.py to test FlashAttention backend along with the existing XFormers backend.
2. Updated test_bart.py , test_florence2.py and encoder_decoder/test_e2e_correctness.py to run with both backends.
3. Moved some methods from xformers.py to backend/utils.py so that they can be reused in both xformers.py and flash_attn.py
4. Updated the checks in worker/enc_dec_model_runner.py to now check that the backend is either FlashAttention or XFormers instead of only XFormers as we do currently.
5. Updated models/bart.py to invoke attention.forward with query of shape [num_tokens, hidden_size]. Currently it was invoking the forward with a query of shape [num_tokens, num_heads, head_size] which is not default.

#7366

[sroy745]

Thanks for the review. Addressed comments. PTAL

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. @sroy745 please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[heheda12345]

LGTM. Thanks for your hardwork on this. Looking forward for the follow-up PRs for test_encoder_decoder_attention and mllama support.

Also CC @WoosukKwon. You may need to sync this PR to v1 later.

[sroy745]

@ywang96 PTAL when you get a chance. PR has been LG'ed by @heheda12345 , is synced to head and all tests are passing.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. @sroy745 please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[ywang96]

Thanks for this great work!