Cross-encoder and MTEB reranking tasks

[qiisziilbash]

I tried to use a cross-encoder model (e.g. BAAI/bge-reranker-base) with reranking tasks (e.g. SciDocsRR) and I was expecting the similarity scores to be computed directly from cross encoder using pair of (query, doc) but when I put a debugger and go into weeds of the code I can't see that but instead I see embeddings are still being computed and scores are computed from embeddings.

What am I missing?
In general I wonder:

• How an embedding model and a reranker model is being treated differently for a given reranking task?
• How a reranker model is used for retrieval tasks?

[KennethEnevoldsen]

@orionw I believe you are the one most familiar with this

[orionw]

Great question @qiisziilbash! Some example code for cross-encoders is here.

I think there could be two issues:

(1) the cross-encoder needs initial results to rerank. If you don't have any, perhaps it's using an initial base embedder to get the first stage results. You can skip this by providing the cross-encoder with previous_results path (see the example).

The (2)nd issue could be that you doing a Reranking AbstractTask (I assume based on the name). When adding the cross-encoder implementation I only tested it for retrieval -- not all tasks support cross-encoders (e.g. clustering can't, etc.). Obviously reranking would also be a good fit for a cross-encoder. It's quite possible that we're just missing something small to enable it for reranking tasks (see here for the retrieval code and it seems like we'd just need to change a bit here in the reranking code). Thanks for pointing this out, we'll make an issue to fix it.

[qiisziilbash]

Thank you so much @orionw
I also have noticed in the leaderboard if I only check cross-encoders there is no model shows up. but I see in one of your old comments here there are some models with that filter. Is it the case that there is no cross encoder in the leaderboard anymore or there is an issue with that filter?

[orionw]

The only place we evaluated cross-encoders is what I did for the instruction-retrieval tab. It's hard to put cross-encoders in the main tab since we average over tasks that cross-encoders can't do (bi-text mining, clustering, etc.). But it could be done for many of the other tabs.

I'd love for someone who has more bandwidth to add some results on the main retrieval/reranking tab, I think it would be really interesting results and is understudied right now.

[qiisziilbash]

I see; one more question: what makes a model to show up in cross-encoder checkbox? when I go to instruction-retrieval tab and I check only cross encoders for example this model shows up which I can't see its relationship to being a cross-encoder. would u help me understand that?

[orionw]

Great question! My understanding of a cross-encoder is a model that applies full attention over query and document to create a relevance score.

I used Flan-T5 as a cross-encoder in a zero-shot MonoT5 fashion - giving it an instruction and asking it to determine the relevance (true/false) and then taking the softmax of the logits of the true/false tokens to determine the score. Flan-T5 is surprisingly good at this for not being trained for it. In the FollowIR paper it was only few points worse than MonoT5 despite not training on any reranking data.

[KennethEnevoldsen]

We should probably consider adding a better UI for cross-encoders. I have seen others use a two-column approach, with the second being the cross-encoder. We could consider this an "experiment" (see #1211 )

[orionw]

What do you mean by “two column” @KennethEnevoldsen? Agree it could go well as an “experiment”!