Upstream sync 2024 03 14

[robertgshaw2-neuralmagic]

SUMMARY:

• upstream merge (sync) up to 54be8a0

NOTES

• Updated ruff configs had line limits. Had to clean up a lot of files manually. I think ./format.sh runs yapf and ruff only on the nm-vllm/vllm directory whereas our automation runs on everything in the nm-vllm, so it was a bit tricky for me to catch why the automation was failing. cc @varun-sundar-rabindranath please review the benchmark directory in detail

Primary upstream changes:

Kernels

• batched_rotary_embedding
• gelu_tanh_and_mul

Core

• LLMEngine refactor <<< adds new layer of abstraction to vLLM. All should look at this

TEST PLAN:

• nightly automation

[mgoin]

this looks duplicated?

[varun-sundar-rabindranath]

Hey @robertgshaw2-neuralmagic - can you update the PR description with this template 7bbd2cc

something like

SUMMARY:
* upstream merge (sync) up to `54be8a0`

NOTES:
* Updated ruff configs had line limits. Had to clean up a lot of files manually. I think ./format.sh runs yapf and ruff only on the nm-vllm/vllm directory whereas our automation runs on everything in the nm-vllm, so it was a bit tricky for me to catch why the automation was failing. cc @varun-sundar-rabindranath please review the benchmark directory in detail

Primary upstream changes:
Kernels
[batched_rotary_embedding](https://github.com/vllm-project/vllm/commit/7e9bd08f60a4b18e3646ff986caeacde9ffffa53)
[gelu_tanh_and_mul](https://github.com/neuralmagic/nm-vllm/pull/127)

Core
https://github.com/vllm-project/vllm/pull/3191 <<< adds new layer of abstraction to vLLM. All should look at this

TEST PLAN:
nightly automation