[ROCm] Fix some kernels failed unit tests

[hongxiayang]

This pull request is to fix some of the failed unit tests in ROCm platform, specifically,

• adjusted tests in test_activation.py, test_attention.py , test_pos_encoding.py and test_cache.py in tests/kernels to fix the failed tests
• refactored the code to get default atol and rtol values for use in torch.allclose

Tests:
To run the associated tests on ROCm platform, first we need to be inside the docker container with ROCm pytorch and vllm install, and then copy the tests directory to the vllm installed location, and then run the test using pytest.

Use the environment variable PYTORCH_TEST_WITH_ROCM=1 if inside the ROCm 5.7 docker image as shown in the below example:
This environment variable is already set inside the ROCm 6.0 docker image.

root@xxx:/opt/conda/envs/py_3.10/lib/python3.10/site-packages/vllm-0.2.7+rocm573-py3.10-linux-x86_64.egg/vllm/tests/kernels# PYTORCH_TEST_WITH_ROCM=1 pytest test_activation.py
====================================================================================== test session starts ======================================================================================
platform linux -- Python 3.10.13, pytest-7.4.2, pluggy-1.3.0
rootdir: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/vllm-0.2.7+rocm573-py3.10-linux-x86_64.egg/vllm/tests/kernels
plugins: shard-0.1.2, xdoctest-1.1.0, xdist-3.3.1, flakefinder-1.1.0, rerunfailures-12.0, hypothesis-5.35.1, anyio-3.7.1
collected 216 items                                                                                                                                                                             
Running 216 items in this shard

test_activation.py ...................................................................................................................................................................... [ 76%]
..................................................                                                                                                                                        [100%]

======================================================================================= warnings summary ========================================================================================
../../../../torch/cuda/__init__.py:546
  /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:546: UserWarning: Can't initialize NVML
    warnings.warn("Can't initialize NVML")

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================================================ 216 passed, 1 warning in 1.82s =================================================================================

[hongxiayang]

cc @tjtanaa

[tjtanaa]

Continuing the discussion from
#2409 (comment)

Thanks for pointing out which tests failed in this case. We could handle this in a different PR?

I think resolving it in another PR could be better, if we still analysing the issue.
Based on my understanding, I remember fp16 and half is supported as a native operator in ROCm whereas bfloat16 operations are mostly handled by casting it to float as intermediate type, as shown in /opt/rocm/include/hip/amd_detail/amd_hip_bf16.h
https://github.com/ROCm/clr/blob/rocm-6.0.x/hipamd/include/hip/amd_detail/amd_hip_bf16.h
https://github.com/ROCm/clr/blob/rocm-6.0.x/hipamd/include/hip/amd_detail/amd_hip_bfloat16.h
https://github.com/ROCm/clr/tree/rocm-5.7.x/hipamd/include/hip/amd_detail/amd_hip_bf16.h
https://github.com/ROCm/clr/tree/rocm-5.7.x/hipamd/include/hip/amd_detail/amd_hip_bfloat16.h

Do you perhaps by chance have any clue as to why natively supported float16 behaves this way?

[jeffdaily]

AMD HW does not have native bfloat16 arithmetic except for the matrix multiply operations.

[tjtanaa]

LGTM.

[simon-mo]

Hmmm it seems some of the tests are failing due to allclose.

https://buildkite.com/vllm/ci/builds/495#018d323c-6fa7-47ed-a078-04a2bc14f21c/51-7825

We are soft-failing kernel test due to GPU memory, but it shouldn't have any other error than CUDAOutOfMemory.

[hongxiayang]

Hmmm it seems some of the tests are failing due to allclose.

https://buildkite.com/vllm/ci/builds/495#018d323c-6fa7-47ed-a078-04a2bc14f21c/51-7825

We are soft-failing kernel test due to GPU memory, but it shouldn't have any other error than CUDAOutOfMemory.

Will check the failed ones. May be the message below of "All checks have passed" can be "improved"?

[simon-mo]

Yeah i'm trying to figure that out sorry.

[hongxiayang]

Yeah i'm trying to figure that out sorry.

@simon-mo Can this pull request be merged now? The longer it stays, more potential conflict will occur.

[simon-mo]

LGTM. Will merge once the tests are only OOM not accuracy off.