finished passage replication

[ArthurChen189]

OS: Ubuntu 18.04.5 LTS (bionic)

Issues:

1. After the replication was finished, no changes were found by git before changing this Readme though many output files were generated in the /collections folder
2. Using "python" instead of "python3" resulted in errors for files that use "printf" in a python3 manner. (For my environment, "python" points to python3)

[codecov]

Codecov Report

Merging #1507 (741e905) into master (45a5a21) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master    #1507   +/-   ##
=========================================
  Coverage     57.27%   57.27%           
  Complexity      978      978           
=========================================
  Files           166      166           
  Lines          9101     9101           
  Branches       1231     1231           
=========================================
  Hits           5213     5213           
  Misses         3498     3498           
  Partials        390      390           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 45a5a21...741e905. Read the comment docs.

[MXueguang]

@ArthurChen189 Thanks for the reproduction

(1) shouldn't be an issue since collections is in .gitignore

anserini/.gitignore

Line 48 in 45a5a21

collections/

(2) shouldn't be an issue too I think? We assume python is linked to python3 by default. If you get error by using python, I think it accidentally linked to python2 in your environment. Do you mind double-check? And paste the error you get?

besides, could you please copy/paste the reproduction results you get in this PR too? We'd like to have a record of if we get "exactly" the same result across different environments (e.g. OS).

[ArthurChen189]

@ArthurChen189 Thanks for the reproduction

(1) shouldn't be an issue since collections is in .gitignore

anserini/.gitignore

Line 48 in 45a5a21

collections/

(2) shouldn't be an issue too I think? We assume python is linked to python3 by default. If you get error by using python, I think it accidentally linked to python2 in your environment. Do you mind double-check? And paste the error you get?

besides, could you please copy/paste the reproduction results you get in this PR too? We'd like to have a record of if we get "exactly" the same result across different environments (e.g. OS).

1. if that, would I still able to upload files in the "collections" folder since no commit can be made.
2. what I got:

arthur@arthur-thinkpad:~/Workspace/arthur/anserini$ python tools/scripts/msmarco/convert_collection_to_jsonl.py \ --collection-path collections/msmarco-passage/collection.tsv \ --output-folder collections/msmarco-passage/collection_jsonl File "tools/scripts/msmarco/convert_collection_to_jsonl.py", line 39 print(f'Converted {i:,} docs, writing into file {file_index}') ^ SyntaxError: invalid syntax

I think my python points to python2.

[MXueguang]

if that, would I still able to upload files in the "collections" folder since no commit can be made.

you don't have to commit the things in collections. Those files are for local only

I think my python points to python2.

I see. Could you change the python3 in this PR back to python for now since it is not an issue?

[ArthurChen189]

Thanks. Will change it back and submit a new pr.
Reproduction results:

map                   	all	0.1957
recall_1000           	all	0.8573
 

[lintool]

Thanks... please change python3 back to python and we'll be ready for merging.

[lintool]

Closed by #1509