[CI] Perform snapshot based model result test

[simon-mo]

This PR implement snapshot based testing in CI such that the model output in float32 matches exactly as of Huggingface version.

[sahilsuneja1]

Hi @simon-mo , what's the status on this PR? I updated it a bit based on the current master (PR here) I do observe different outputs for vllm vs HF for Deci/DeciLM-7b. Example below:

prompt: vLLM is a high-throughput and memory-efficient inference and serving engine for LLMs.

HF: vLLM is a high-throughput and memory-efficient inference and serving engine for LLMs. It is designed to be a drop-in replacement for HuggingFace's Transformers library, and it is compatible with all Transformers models.\nLLM is a high-throughput and memory-efficient inference and serving engine for LLMs. It is designed to be a drop-in replacement for HuggingFace's Transformers library, and it is compatible with all Transformers models.\nLLM is a high-throughput and memory-efficient inference and serving engine for LLMs. It is designed to be a drop-in replacement for HuggingFace's Transformers library

vLLM: vLLM is a high-throughput and memory-efficient inference and serving engine for LLMs. It is designed to be a drop-in replacement for HuggingFace's Transformers library, and is compatible with all of the same models and datasets.\nThe LLM is a high-throughput and memory-efficient inference and serving engine for LLMs. It is designed to be a drop-in replacement for HuggingFace's Transformers library, and is compatible with all of the same models and datasets.\nLLM is a high-throughput and memory-efficient inference and serving engine for LLMs. It is designed to be a drop-in replacement for HuggingFace's Trans

[simon-mo]

@sahilsuneja1, thanks! I don't have bandwidth to work on this PR at the moment so if you can create another PR based on your fixes that would be awesome. As you can see the primary issue in the CI testing is that it runs out of memory even with --tensor-paralle-size 2 on our CI machine (L4x2), this is quite weird because I was able to run the test locally on the same hardware. If you can take over and iterate that would be appreciated!

[sahilsuneja1]

Got it, thanks @simon-mo, will do.
Where do you see the memory error? Probably in the run prior to the latest merge? How can I see that OOM run log?

[sahilsuneja1]

Found the run history: https://buildkite.com/vllm/ci/builds?branch=simon-mo%3Amodels-snapshot-testing
Where's the OOM error?

[simon-mo]

Actually I'm going to close this PR because I think snapshot testing is still too brittle (need to update it on model by model basis). We should just get A100 machine to run these tests.

[sahilsuneja1]

OK, I'll close the update PR as well

[simon-mo]

sorry about the back and forth!

[sahilsuneja1]

No worries!