Skip to content

castorini/replicate-lce

Repository files navigation

Replicate-lce

This replication for lce-loss is based on Capreolus toolkit.

Quick Start

  1. Prerequisites: Python 3.7+ and Java 11. See the installation instructions
  2. Clone capreolus toolkit: git clone https://github.com/crystina-z/capreolus.git -b feature/eval+ptmaxp
  3. Run an experiment: python run.py --config_path yaml_config_neg/yaml_filename

Tables

Here are the three tables we stated and the corresponding yaml file for each row


Table1

Baseline - HN+FirstStage - Loss - n MRR@10 yaml_file_name
(1) monoELECTRABase-BM25-CE-1 0.378 test_line4_nneg_1_bm25_ce.yaml
(2) monoELECTRABase-BM25-HG-1 0.379 test_line5_nneg_1_bm25_hg.yaml
(3) monoELECTRABase-BM25-LCE-1 0.378 test_line24_nneg_1_bm25_lce.yaml
(4) monoELECTRABase-BM25-LCE-7 0.391 test_line24_nneg_1_bm25_lce.yaml
(5) monoELECTRABase-TCT-ColBERTv2-CE-1 0.365 test_line26_nneg_1_tct_ce.yaml
(6) monoELECTRABase-TCT-ColBERTv2-HG-1 0.375 test_line27_nneg_1_tct_hg.yaml
(7) monoELECTRABase-TCT-ColBERTv2-LCE-1 0.394 test_line16_nneg_1_tct_lce.yaml
(8) monoELECTRABase-TCT-ColBERTv2-LCE-7 0.401 test_line11_nneg_7_tct_lce.yaml

Please note that in table 2 both a and c, b and d are trained with the same yaml file but inference on different ones.

Table2

HN-FirstStage MRR@10 yaml_file_name
(a) BM25-BM25 0.391 test_line10_nneg_7_bm25_lce.yaml
(b) TCT-ColBERTv2-BM25 0.389 test_line11_nneg_7_tct_lce.yaml
(c) BM25-TCT-ColBERTv2 0.402 test_line10_nneg_7_bm25_lce.yaml
(d) TCT-ColBERTv2-TCT-ColBERTv2 0.401 test_line11_nneg_7_tct_lce.yaml

Table3

Group-size MRR@10 yaml_file_name
(a-1) 2 0.393 test_line26_nneg_1_tct_ce.yaml
(a-2) 4 0.400 test_line17_nneg_3_tct_lce.yaml
(a-3) 8 0.401 test_line11_nneg_7_tct_lce.yaml
(a-4) 16 0.408 test_line19_nneg_15_tct_lce.yaml
(a-5) 32 0.414 test_line20_nneg_31_tct_lce.yaml
(b-1) 2 0.381 test_table3_nneg_1-bm25-train.yaml
(b-2) 4 0.397 test_table3_nneg_3-bm25-train.yaml
(b-3) 8 0.402 test_table3_nneg_7-bm25-train.yaml
(b-4) 16 0.403 test_table3_nneg_15-bm25-train.yaml
(b-5) 32 0.407 test_table3_nneg_31-bm25-train.yaml

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published