Fix RT-DETR inference with float16 and bfloat16

[qubvel]

What does this PR do?

Fix positional embeddings and anchor data types for RT-DETR.

Running the model in float16 or bfloat16 with the following code crashes with the error:

import torch
import requests

from PIL import Image
from transformers import RTDetrForObjectDetection, RTDetrImageProcessor

device = "cuda"
dtype = torch.float16

url = 'http://images.cocodataset.org/val2017/000000039769.jpg' 
image = Image.open(requests.get(url, stream=True).raw)

image_processor = RTDetrImageProcessor.from_pretrained("PekingU/rtdetr_r50vd")
model = RTDetrForObjectDetection.from_pretrained("PekingU/rtdetr_r50vd", attn_implementation="eager", torch_dtype=dtype, device_map=device)

inputs = image_processor(images=image, return_tensors="pt").to(device)
pixel_values = inputs.pixel_values.to(device).to(dtype)

with torch.no_grad():
    outputs = model(pixel_values)

assert outputs.logits.dtype == dtype

results = image_processor.post_process_object_detection(outputs, target_sizes=torch.tensor([image.size[::-1]]), threshold=0.3)

for result in results:
    for score, label_id, box in zip(result["scores"], result["labels"], result["boxes"]):
        score, label = score.item(), label_id.item()
        box = [round(i, 2) for i in box.tolist()]
        print(f"{model.config.id2label[label]}: {score:.2f} {box}")

Error:

RuntimeError: expected mat1 and mat2 to have the same dtype, but got: float != c10::Half

One approach to fix is to use torch.autocast(), but it could also be fixed with proper dtypes in the modeling file.

Steps:

1. Add test that fails with float16/bfloat16 dtypes
2. Add fix

Before submitting

• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[qubvel]

Added test failed for float16 and bfloat16 as expected:

[qubvel]

Tests are green with the fix

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[amyeroberts]

Thanks for fixing!

[qubvel]

Thanks for the review!