Add voyage-code-3 results

[Liuhong99]

Add voyage-code-3 COIR results

Add voyageai__voyage-code-3 in paths.json

[KennethEnevoldsen]

This looks very reasonable

[Liuhong99]

Also PR in embeddings-benchmark/leaderboard#58. Thanks!

[Liuhong99]

Please let me know how I can help to merge the results. Ideally we hope it could be on the LB before leaderboard 2.0 is ready. (btw, congratulations on the upcoming LB 2.0 release---This is huge for the community!)

[KennethEnevoldsen]

LB 2.0 release---This is huge for the community!

Glad to hear! I'm looking forward to getting it online as well!

Please let me know how I can help to merge the results

It seems like I was a bit fast during my first review. It appears to fail due to the revision (we no longer allow "no_revision_available"). It seems like you used the implementation on MTEB then you can simply add "1".

Btw. the implementation on mteb is our reference implementation, so if you want to add the model there then we would be happy to review a PR (if there are disagreements between submitted scores and our implementation then we will prefer ours to ensure reproducibility)

[Liuhong99]

Thanks! @KennethEnevoldsen. The revision is now fixed. We used MTEB implementation.

[KennethEnevoldsen]

@Liuhong99 I have made some suggestions to the model_meta will you update it - otherwise I think we are good to go

[Liuhong99]

@Liuhong99 I have made some suggestions to the model_meta will you update it - otherwise I think we are good to go

@KennethEnevoldsen might be a dumb question--where can I locate the suggestions to model_meta?
Below is the current model_meta:

{"name": "voyageai/voyage-code-3", "revision": "1", "release_date": "2024-12-02", "languages": null, "n_parameters": null, "memory_usage": null, "max_tokens": 32000, "embed_dim": 1024, "license": null, "open_source": false, "similarity_fn_name": null, "framework": [], "loader": "VoyageWrapper"}

[KennethEnevoldsen]

@KennethEnevoldsen might be a dumb question--where can I locate the suggestions to model_meta?
Below is the current model_meta:

Do you mean where to add the changes?

You can either change the json directly, but ideally, we would want these to be submitted to the MTEB repository along with the model "voyageai/voyage-code-3".

[Liuhong99]

If the change refers to changing "no_revision_available" to "1", then it's already fixed and committed. Please let me know if there are other places that you suggest to change in model_meta. Thanks ❤️

[KennethEnevoldsen]

Sorry seems like I misunderstood the question, the you can find the suggestion here (screenshot from earlier in the thread):

[Liuhong99]

Got it! Makes perfect sense. Updated model_meta.json

[KennethEnevoldsen]

Wonderful - merged!