[DO NOT MERGE] Vinayak/moe final hashem

[carlushuang]

This PR combined vinayak/moe_final with PR:#126, kernel from hashem
Note: to enable triton optimization, need download triton code:

export VLLM_USE_ROCM_CUSTOM_PAGED_ATTN=1
export HIP_FORCE_DEV_KERNARG=1
export VLLM_USE_TRITON_FLASH_ATTN=false

export VLLM_MOE_PADDING=128
export TRITON_HIP_USE_NEW_STREAM_PIPELINE=1
export VLLM_MOE_MFMASWIZZLE=1 
export VLLM_MOE_MFMASWIZZLE_M_THRSHLD=128
# if set VLLM_MOE_MFMASWIZZLE_M_THRSHLD=0, this is just doug + vinayak
export VLLM_SCHED_PREFILL_COUNT=6

you can use benchmark in benchmarks/kernels/benchmark_mixtral_moe_dec.py to do benchmark

# test hashem's kernel with tp-2
VLLM_MOE_MFMASWIZZLE_M_THRSHLD=128  python3 benchmarks/kernels/benchmark_mixtral_moe_dec.py  --tp-size 2
 
# test hashem's kernel with tp-1
VLLM_MOE_MFMASWIZZLE_M_THRSHLD=128  python3 benchmarks/kernels/benchmark_mixtral_moe_dec.py  --tp-size 1
 
# test vinayak's triton kernel with tp-2
VLLM_MOE_MFMASWIZZLE_M_THRSHLD=0  python3 benchmarks/kernels/benchmark_mixtral_moe_dec.py  --tp-size 2
 
# test vinayak's triton kernel with tp-1
VLLM_MOE_MFMASWIZZLE_M_THRSHLD=0  python3 benchmarks/kernels/benchmark_mixtral_moe_dec.py  --tp-size 1

[github-actions]

This pull request has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this pull request should remain open. Thank you!