Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MIEB] Change Flickr30k to test split #1449

Merged
merged 94 commits into from
Nov 15, 2024
Merged

[MIEB] Change Flickr30k to test split #1449

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
94 commits
Select commit Hold shift + click to select a range
1e50557
wip: start adding BLIP models
Jamie-Stirling Sep 6, 2024
8f8e05c
add other blip variants
Jamie-Stirling Sep 9, 2024
3ea4d3e
Merge branch 'mieb' of https://github.com/embeddings-benchmark/mteb i…
Jamie-Stirling Sep 9, 2024
be8b4bb
wip: add blip2_models.py
Jamie-Stirling Sep 11, 2024
b57a395
make lint
Jamie-Stirling Sep 11, 2024
236a94f
wip: implement blip2 wrapper
Jamie-Stirling Sep 13, 2024
1f2f8c3
feat: add blip2 models, still mismatched names
Jamie-Stirling Sep 15, 2024
8c64860
fix: remove projections from image and text embeddings
Jamie-Stirling Sep 15, 2024
20839ca
make lint
Jamie-Stirling Sep 15, 2024
ec47c69
wip: add coco BLIP2
Jamie-Stirling Sep 15, 2024
e8f4ae1
fix: BLIP2 better zero-shot classification without text_proj and visi…
Jamie-Stirling Sep 16, 2024
57bc3b8
tidy blip2
Jamie-Stirling Sep 19, 2024
4cbec1b
add imagenet-dog-15 dataset
Jamie-Stirling Sep 19, 2024
6da26dc
Merge branch 'mieb' of https://github.com/embeddings-benchmark/mteb i…
Jamie-Stirling Sep 19, 2024
35be38d
tidy and lint
Jamie-Stirling Sep 19, 2024
83d0f45
remove unused import
Jamie-Stirling Sep 19, 2024
a309de5
add cluster_accuracy, ari and nmi to Image.ClusteringEvaluator
Jamie-Stirling Sep 19, 2024
02c1f81
add imagenet-10 clustering task
Jamie-Stirling Sep 20, 2024
cede108
add SOPI2IRetrieval
Jamie-Stirling Sep 20, 2024
d226748
add results forclip on ImageNet10Clustering and ImageNetDog15Clustering
Jamie-Stirling Sep 20, 2024
2086904
Merge branch 'mieb-imagenet' of https://github.com/Jamie-Stirling/mte…
Jamie-Stirling Sep 20, 2024
c7c6893
add SOPI2IRetrieval results for clip 32
Jamie-Stirling Sep 20, 2024
5bf66fd
add results for clip vit 32/SOPI2IRetrieval
Jamie-Stirling Sep 23, 2024
3866647
merge incoming change from mieb
Jamie-Stirling Sep 23, 2024
b1759dc
resolve conflict
Jamie-Stirling Sep 23, 2024
a890171
add RP2kI2IRetrieval dataset
Jamie-Stirling Sep 23, 2024
d3396fa
add RP2kI2IRetrieval results with clip-vit-base-patch32
Jamie-Stirling Sep 23, 2024
aa58556
update image retrieval __init__.py
Jamie-Stirling Sep 23, 2024
91ad565
fix ImageTextPair dataloading for large datasets; more compositionali…
gowitheflow-1998 Sep 23, 2024
fc4d1ee
add RP2kI2IRetrieval and METI2IRetrieval
Jamie-Stirling Sep 24, 2024
7d4f332
add METI2IRetreival
Jamie-Stirling Sep 24, 2024
57593eb
add SOP results
Jamie-Stirling Sep 24, 2024
62e6e9f
make lign
Jamie-Stirling Sep 24, 2024
5d84ace
merge upstream mieb
Jamie-Stirling Sep 24, 2024
094a654
new revision for METI2IRetrieval
Jamie-Stirling Sep 25, 2024
2cc7ffe
make lint
Jamie-Stirling Sep 25, 2024
d3f9db2
reset corpus chunk size
Jamie-Stirling Sep 25, 2024
b94e4e2
remove wrong classification import
Jamie-Stirling Sep 26, 2024
a037081
Merge branch 'mieb' of https://github.com/embeddings-benchmark/mteb i…
Jamie-Stirling Sep 26, 2024
b54c227
add Flickr30k T2I and I2T
Jamie-Stirling Sep 27, 2024
08ce105
add Flickr30k T2I retriebal
Jamie-Stirling Sep 27, 2024
5a45bc4
reduced-size MET revision
Jamie-Stirling Sep 27, 2024
9b8c46d
fix: add Flickr30k T2I
Jamie-Stirling Sep 29, 2024
65c2bfb
make lint
Jamie-Stirling Sep 29, 2024
8d79c5b
add two landmark datasets and results
Jamie-Stirling Oct 1, 2024
fe0d053
add Sketchy i2i retrieval
Jamie-Stirling Oct 2, 2024
999d680
resolve conflicts
Jamie-Stirling Oct 8, 2024
a5011e3
Merge branch 'mieb' of https://github.com/embeddings-benchmark/mteb i…
gowitheflow-1998 Oct 12, 2024
72ace6e
add task metadata
Jamie-Stirling Oct 14, 2024
5afb2aa
add BLINKIT2IRetrieval dataset
Jamie-Stirling Oct 14, 2024
4ba56b1
add BLINKIT2TRetrieval
Jamie-Stirling Oct 14, 2024
dd99ddc
add ImageCoDeT2IRetrieval
Jamie-Stirling Oct 14, 2024
411e313
make lint
Jamie-Stirling Oct 14, 2024
74a2400
add vizwiz retrieval and results
Jamie-Stirling Oct 15, 2024
8067b17
fix vizwiz duplicate texts
Jamie-Stirling Oct 15, 2024
a80f3bb
add new vizwiz results
Jamie-Stirling Oct 15, 2024
4e1155c
add VQA2 results
Jamie-Stirling Oct 16, 2024
e91faf6
add GLD v2 I2T retrieval
Jamie-Stirling Oct 17, 2024
f2721c0
add gld v2 i2i retrieval
Jamie-Stirling Oct 17, 2024
c9df7ac
make lint
Jamie-Stirling Oct 18, 2024
a730ecc
add AbsTaskAny2AnyMultiChoice
Jamie-Stirling Oct 18, 2024
dab5913
make lint
Jamie-Stirling Oct 18, 2024
1123d53
remove GLDv2I2IRetrieval
Jamie-Stirling Oct 18, 2024
e053546
Merge branch 'mieb-datasets' into mieb-any2image-multi
Jamie-Stirling Oct 18, 2024
d7a1e25
Merge branch 'mieb' of https://github.com/embeddings-benchmark/mteb i…
gowitheflow-1998 Oct 20, 2024
6d22297
Merge branch 'mieb' of https://github.com/embeddings-benchmark/mteb i…
Jamie-Stirling Oct 20, 2024
d0cddd4
exclude AbsTaskAny2AnyMultiChoice from test_load_data
Jamie-Stirling Oct 20, 2024
bc8aa01
fix e5v&vista
gowitheflow-1998 Oct 21, 2024
7aa5bd2
Merge branch 'mieb' of https://github.com/embeddings-benchmark/mteb i…
gowitheflow-1998 Oct 21, 2024
6d8ed9b
remove duplicate corpus entries from BLINKIT2TRetreival dataset
Jamie-Stirling Oct 21, 2024
53065b8
task type fix for running tasks
gowitheflow-1998 Oct 21, 2024
3385584
update BLINKIT2T metadata
Jamie-Stirling Oct 21, 2024
3b93440
fix wrong meta
gowitheflow-1998 Oct 21, 2024
3998ccd
run mieb script
gowitheflow-1998 Oct 21, 2024
a3649a7
split ROxford, RParis into easy, medium and hard
Jamie-Stirling Oct 21, 2024
59a19a1
Merge branch 'mieb' of https://github.com/embeddings-benchmark/mteb i…
Jamie-Stirling Oct 21, 2024
de9c573
make lint
Jamie-Stirling Oct 21, 2024
4c345a7
Merge branch 'mieb-run-tasks-fix' of https://github.com/embeddings-be…
Jamie-Stirling Oct 21, 2024
2f7e2d9
resolve conflict with mieb
Jamie-Stirling Oct 22, 2024
50af768
add BLINK as multi choice tasks
Jamie-Stirling Oct 28, 2024
baf9674
resolve conflicts with mieb
Jamie-Stirling Oct 28, 2024
e646368
fix conflicts with mieb
Jamie-Stirling Oct 28, 2024
016c0d0
resolve conflict
Jamie-Stirling Oct 28, 2024
27444fd
fix: license metadata in wrong format
Jamie-Stirling Oct 29, 2024
3a3f812
Merge branch 'mieb' of https://github.com/embeddings-benchmark/mteb i…
Jamie-Stirling Oct 31, 2024
1abce84
remove null examples from corpus of ROxford and RParis
Jamie-Stirling Oct 31, 2024
a0e8caa
Merge branch 'mieb' of https://github.com/embeddings-benchmark/mteb i…
Jamie-Stirling Nov 7, 2024
ac51be8
fix: add/remove subtasks from BLINKIT2IMultiChoice and BLINKIT2TMulti…
Jamie-Stirling Nov 7, 2024
dca764a
update blink metadata
Jamie-Stirling Nov 7, 2024
2d371c7
add updated BLINK results
Jamie-Stirling Nov 7, 2024
220362d
Merge branch 'mieb' of https://github.com/embeddings-benchmark/mteb i…
Jamie-Stirling Nov 13, 2024
7e69e16
merge upstream mieb
Jamie-Stirling Nov 13, 2024
460a2b8
change Flickr30k to test split
Jamie-Stirling Nov 14, 2024
bada500
change flickr to test split
Jamie-Stirling Nov 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion mteb/tasks/Image/Any2AnyRetrieval/eng/Flickr30kI2TRetrieval.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ class Flickr30kI2TRetrieval(AbsTaskAny2AnyRetrieval):
reference="https://www.semanticscholar.org/paper/From-image-descriptions-to-visual-denotations%3A-New-Young-Lai/44040913380206991b1991daf1192942e038fe31",
dataset={
"path": "JamieSJS/flickr30k",
"revision": "a4cf34ac79215f9e2cd6a10342d84f606fc41cc3",
"revision": "24acb2d0b72e18b03388eb20a6225983c0e3f629",
},
type="Any2AnyRetrieval",
category="i2t",
Expand Down
2 changes: 1 addition & 1 deletion mteb/tasks/Image/Any2AnyRetrieval/eng/Flickr30kT2IRetrieval.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ class Flickr30kT2IRetrieval(AbsTaskAny2AnyRetrieval):
reference="https://www.semanticscholar.org/paper/From-image-descriptions-to-visual-denotations%3A-New-Young-Lai/44040913380206991b1991daf1192942e038fe31",
dataset={
"path": "JamieSJS/flickr30k",
"revision": "a4cf34ac79215f9e2cd6a10342d84f606fc41cc3",
"revision": "24acb2d0b72e18b03388eb20a6225983c0e3f629",
},
type="Any2AnyRetrieval",
category="t2i",
Expand Down
186 changes: 0 additions & 186 deletions ...clip-vit-base-patch32/3d74acf9a28c67741b2f4f2ea7635f0aaf6f0268/Flickr30kI2TRetrieval.json
View file
Open in desktop

This file was deleted.

186 changes: 0 additions & 186 deletions ...clip-vit-base-patch32/3d74acf9a28c67741b2f4f2ea7635f0aaf6f0268/Flickr30kT2IRetrieval.json
View file
Open in desktop

This file was deleted.

Loading