feat: evaluate arctic v2 models in MTEB(Medical) #66
Conversation
| ] | ||
| }, | ||
| "evaluation_time": 33.124486207962036, | ||
| "kg_co2_emissions": null |
There was a problem hiding this comment.
I am a bit sad not to have the co2 usage, but otherwise, this looks fine
There was a problem hiding this comment.
Is there any reason why codecarbon is currently not enabled by default? I'll modify my script to enable it from now on for new runs.
There was a problem hiding this comment.
Actually, I'll just run it again with it enabled, it doesn't take long.
| @@ -0,0 +1 @@ | |||
| {"name": "Snowflake/snowflake-arctic-embed-m-v2.0", "revision": "f2a7d59d80dfda5b1d14f096f3ce88bb6bf9ebdc", "release_date": null, "languages": null, "n_parameters": null, "memory_usage": null, "max_tokens": null, "embed_dim": null, "license": "apache-2.0", "open_weights": null, "public_training_data": false, "public_training_code": null, "framework": ["PyTorch"], "reference": null, "similarity_fn_name": null, "use_instructions": null, "training_datasets": null, "adapted_from": null, "superseded_by": null, "loader": null} | |||
There was a problem hiding this comment.
a lot of the metadata seems to be missing here - is it not run using the implementation in the package? (loader: null)
There was a problem hiding this comment.
It is using the package but without embeddings-benchmark/mteb#1574. It still used the sentence_transformers_loader. I let these models ran while I was making the other PR and then once the PR was finished I just ran the models again on a single task to validate that the previous results stayed the same. I'll upload the new metadata.
|
All good @KennethEnevoldsen ! |
...lake__snowflake-arctic-embed-l-v2.0/edc2df7b6c25794b340229ca082e7c78782e6374/model_meta.json
Outdated
Show resolved
Hide resolved
This PR evaluates the
arctic-embed-v2.0family of models by Snowflake on theMTEB(Medical)benchmark.Checklist
make test.make pre-push.Adding a model checklist
Models are added to
mteb/modelsin embeddings-benchmark/mteb#1574.