Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: evaluate arctic v2 models in MTEB(Medical) #66

Merged
merged 5 commits into from
Dec 11, 2024
Merged

feat: evaluate arctic v2 models in MTEB(Medical) #66

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b4a70a0
feat: evaluate arctic v2 models in MTEB(Medical)
dbuades Dec 9, 2024
e6a73a9
Merge branch 'main' into feat/medical-mteb-arctic-v2
dbuades Dec 10, 2024
a817b37
feat: track co2 emissions
dbuades Dec 10, 2024
1fa6b88
feat: new model_meta.json
dbuades Dec 10, 2024
5c4432b
fix: set `use_instructions` to true
dbuades Dec 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions ...lake-arctic-embed-l-v2.0/edc2df7b6c25794b340229ca082e7c78782e6374/CMedQAv2-reranking.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{
"dataset_revision": "23d186750531a14a0357ca22cd92d712fd512ea0",
"task_name": "CMedQAv2-reranking",
"mteb_version": "1.23.0",
"scores": {
"test": [
{
"map": 0.70069,
"mrr": 0.748533,
"nAUC_map_max": 0.598828,
"nAUC_map_std": 0.31886,
"nAUC_map_diff1": 0.541323,
"nAUC_mrr_max": 0.694851,
"nAUC_mrr_std": 0.382436,
"nAUC_mrr_diff1": 0.634483,
"main_score": 0.70069,
"hf_subset": "default",
"languages": [
"cmn-Hans"
]
}
]
},
"evaluation_time": 33.124486207962036,
"kg_co2_emissions": null
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am a bit sad not to have the co2 usage, but otherwise, this looks fine

Copy link
Contributor Author

@dbuades dbuades Dec 10, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any reason why codecarbon is currently not enabled by default? I'll modify my script to enable it from now on for new runs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I'll just run it again with it enabled, it doesn't take long.

}
Loading
Loading