Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Synthetic Data Generation for Retriever Evaluation #338

Open
wants to merge 95 commits into
base: main
Choose a base branch
from
Open

Synthetic Data Generation for Retriever Evaluation #338

wants to merge 95 commits into from
+12,286 −665

Conversation

vinay-raman
Copy link

@vinay-raman vinay-raman commented Oct 30, 2024 •
edited
Loading

Description

Synthetic data generation for Retriever Evaluation

Usage

python tutorials/synthetic-retrieval-evaluation-customization/main.py \
  --api_key=<API Key> \
  --input_file=tutorials/synthetic-retrieval-evaluation-customization/data/sample_data_rawdoc.jsonl \
  --pipeline-config=tutorials/synthetic-retrieval-evaluation-customization/config/config.yaml\
  --input_format=rawdoc \
  --output_dir=tutorials/synthetic-retrieval-evaluation-customization/outputs/sample_data_rawdoc

Checklist

  • I am familiar with the Contributing Guide.
  • New or Existing tests cover these changes.
  • The documentation is up to date with these changes.

@vinay-raman
Copy link
Author

@ryantwolf I have added the tests and fixed the signoff checks as well. Thanks!

ryantwolf
ryantwolf requested changes Oct 30, 2024
View reviewed changes
Copy link
Collaborator

@ryantwolf ryantwolf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few more pieces of feedback, more about the configuration this time.

tutorials/synthetic_retrieval_evaluation_customization/filters.py Outdated Show resolved Hide resolved
tutorials/synthetic_retrieval_evaluation_customization/requirements.txt Outdated Show resolved Hide resolved
tutorials/synthetic_retrieval_evaluation_customization/retriever_evalset_generator.py Outdated Show resolved Hide resolved
tutorials/synthetic_retrieval_evaluation_customization/README.md Outdated Show resolved Hide resolved
tutorials/synthetic_retrieval_evaluation_customization/README.md Outdated Show resolved Hide resolved
tutorials/synthetic_retrieval_evaluation_customization/notebooks/quickstart.ipynb Outdated Show resolved Hide resolved
tutorials/synthetic_retrieval_evaluation_customization/filters.py Outdated Show resolved Hide resolved
nemo_curator/modules/config.py Outdated Show resolved Hide resolved
tutorials/synthetic_retrieval_evaluation_customization/retriever_evalset_generator.py Outdated Show resolved Hide resolved
@vinay-raman vinay-raman force-pushed the sdg_pipeline/retriever_evalset_generation_signoff_fixed branch from 062cac6 to d5dc0ae Compare November 12, 2024 22:52
ryantwolf and others added 27 commits November 12, 2024 16:17
@ryantwolf @vinay-raman
Fix id field (NVIDIA#250)
aa8a7cc 
Signed-off-by: Ryan Wolf <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@praateekmahajan @vinay-raman
Enabled nightly build using RAPIDS nightly (NVIDIA#237)
3a288ca 
* Sign and add

Signed-off-by: Praateek Mahajan <[email protected]>

* implement strtobool

Signed-off-by: Praateek Mahajan <[email protected]>

* pre-commit

Signed-off-by: Praateek Mahajan <[email protected]>

* Update readme

Signed-off-by: Praateek Mahajan <[email protected]>

---------

Signed-off-by: Praateek Mahajan <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@ryantwolf @vinay-raman
Add Image Curation Tutorial (NVIDIA#254)
1732e61 
* Make embedding column flexible for semdedup

Signed-off-by: Ryan Wolf <[email protected]>

* Fix embedding col in add_dist_to_cents

Signed-off-by: Ryan Wolf <[email protected]>

* Add image curation tutorial

Signed-off-by: Ryan Wolf <[email protected]>

* Address Sarah's feedback

Signed-off-by: Ryan Wolf <[email protected]>

* Add output to image curation tutorial

Signed-off-by: Ryan Wolf <[email protected]>

* Add punctuation

Signed-off-by: Ryan Wolf <[email protected]>

* Address Vibhu's comments

Signed-off-by: Ryan Wolf <[email protected]>

---------

Signed-off-by: Ryan Wolf <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@chrisalexiuk-nvidia @vinay-raman
Fixing Model Address (NVIDIA#263)
1c68e52 
* Fixing Model Address

Signed-off-by: Chris Alexiuk <[email protected]>

* Fixing Model Address v1

Signed-off-by: Chris Alexiuk <[email protected]>

---------

Signed-off-by: Chris Alexiuk <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@miguelusque @vinay-raman
Fix 264. Incorrect affinity parameter (NVIDIA#265)
5cb446d 
Fix NVIDIA#264. Scikit-learn is not expecting anymore the "affinity parameter"

Signed-off-by: Miguel Martínez <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@rjzamora @vinay-raman
Add dask query-planning support (NVIDIA#139)
48f8e33 
* start adding dask-expr support

Signed-off-by: rjzamora <[email protected]>

* add query_planning_enabled util

Signed-off-by: rjzamora <[email protected]>

* add global keyword

Signed-off-by: rjzamora <[email protected]>

* Forgot to remove top level query-planning check

Signed-off-by: rjzamora <[email protected]>

* fix other shuffle-arg problems that don't 'work' with dask-expr

Signed-off-by: rjzamora <[email protected]>

* remove name arg usage for now

Signed-off-by: rjzamora <[email protected]>

* fix bugs

Signed-off-by: rjzamora <[email protected]>

---------

Signed-off-by: rjzamora <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@praateekmahajan @vinay-raman
add enable_cudf_spill to LocalCudaCluster
98d1d33 
Signed-off-by: Praateek Mahajan <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@praateekmahajan @vinay-raman
remove typecheck
6355caa 
Signed-off-by: Praateek Mahajan <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@praateekmahajan @vinay-raman
remove unused imports
6885a1f 
Signed-off-by: Praateek Mahajan <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@praateekmahajan @vinay-raman
remove _enable_spilling
d7f883d 
Signed-off-by: Praateek Mahajan <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@praateekmahajan @vinay-raman
remove unused LCU obj
e158287 
Signed-off-by: Praateek Mahajan <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@praateekmahajan @vinay-raman
remove unused LCC obj
5ecf6bf 
Signed-off-by: Praateek Mahajan <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@sarahyurick @vinay-raman
oom best practices draft
a57ca9f 
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@sarahyurick @vinay-raman
remove vram section
b291729 
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@sarahyurick @vinay-raman
address ryan's review
d8f2ebd 
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@sarahyurick @vinay-raman
add other's suggestions
b8c1028 
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@sarahyurick @vinay-raman
fix indents
f124d0d 
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@sarahyurick @vinay-raman
revise dask dashboard paragraph
546cda7 
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@sarahyurick @vinay-raman
add ryan's comments
6d48128 
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@sarahyurick @vinay-raman
dash bullet points
9b611b6 
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@sarahyurick @vinay-raman
spacing?
1d2591b 
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@sarahyurick @vinay-raman
edit bullet pts
68e8952 
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@sarahyurick @vinay-raman
hyphens
cf7bd1c 
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@sarahyurick @vinay-raman
hyphens
9029288 
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@sarahyurick @vinay-raman
2 spaces
7544309 
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@miguelusque @vinay-raman
Fix 255 - Improve separate_by_metadata performance for jsonl files (N…
0ef899b 
…VIDIA#256)

* Improve performance in jsonl files

Signed-off-by: miguelusque <[email protected]>

* Improve performance in jsonl files

Signed-off-by: miguelusque <[email protected]>

* Shutdown Dask cluster at exit

Signed-off-by: miguelusque <[email protected]>

* Remove unneeded persist() and wait() operations

Signed-off-by: miguelusque <[email protected]>

* Display only Dask error messages or above

Signed-off-by: miguelusque <[email protected]>

* Cancel any remaining futures

Signed-off-by: miguelusque <[email protected]>

* Remove Dask warning message

Signed-off-by: miguelusque <[email protected]>

* Rename new arguments

Signed-off-by: miguelusque <[email protected]>

* Refactor separate_by_metadata

Signed-off-by: miguelusque <[email protected]>

---------

Signed-off-by: miguelusque <[email protected]>
Co-authored-by: miguelusque <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@ayushdg @vinay-raman
Pin to spacy<3.8 temporarily to unblock CI (NVIDIA#276)
88b8a80 
* Pin to spacy<3.8 temporarily to unblock CI

Signed-off-by: Ayush Dattagupta <[email protected]>

* Update pin in rapids nightly dep as well

Signed-off-by: Ayush Dattagupta <[email protected]>

---------

Signed-off-by: Ayush Dattagupta <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@vinay-raman
minor changes, main.py WIP
c21d7c3 
Signed-off-by: viraman <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@vinay-raman
minor changes
1fd3cb5 
Signed-off-by: viraman <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@vinay-raman
added write to beir functionality
ee32779 
Signed-off-by: viraman <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@vinay-raman
added RetreiverEvalSDGConfig dataclass to config.py
a52de31 
Signed-off-by: Vinay Raman <[email protected]>
@vinay-raman
minor changes
bb35e65 
Signed-off-by: viraman <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@vinay-raman
removed unused imports
5cfc448 
Signed-off-by: viraman <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@vinay-raman
added README
ec699ba 
Signed-off-by: viraman <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@vinay-raman
file name change for config.yaml
49e5f90 
Signed-off-by: Vinay Raman <[email protected]>
@vinay-raman
minor changes to README
91d5cde 
Signed-off-by: viraman <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@vinay-raman
progress bar for generations
87b1408 
Signed-off-by: viraman <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@vinay-raman
added additional configs
c05d866 
Signed-off-by: viraman <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@vinay-raman
minor changes
3c044fd 
Signed-off-by: viraman <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@vinay-raman
added quickstart notebook
d77867b 
Signed-off-by: viraman <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@vinay-raman
minor changes
fd123a4 
Signed-off-by: viraman <[email protected]>
Signed-off-by: Vinay Raman <[email protected]>
@vinay-raman
fixed TokenizationError
5d6fbd2 
Signed-off-by: Vinay Raman <[email protected]>
@vinay-raman
added tests
e300078 
Signed-off-by: Vinay Raman <[email protected]>
@vinay-raman
removed deleted files
8103bcc 
Signed-off-by: Vinay Raman <[email protected]>
@vinay-raman
removed api key from notebook
6d9faf6 
Signed-off-by: Vinay Raman <[email protected]>
@vinay-raman
changed tutorials name
231cb75 
Signed-off-by: Vinay Raman <[email protected]>
@vinay-raman
put filters in nemo_curator.filters module
1aaa6f9 
Signed-off-by: Vinay Raman <[email protected]>
@vinay-raman
fixed PR comments pertaining to config
01e3ad0 
Signed-off-by: Vinay Raman <[email protected]>
@vinay-raman
modified requirements.txt
9004fb7 
Signed-off-by: Vinay Raman <[email protected]>
@vinay-raman
added to tutorials/README.md
b2dda07 
Signed-off-by: Vinay Raman <[email protected]>
@vinay-raman
changed tutorials name to have hypens instead of _
410ee56 
Signed-off-by: Vinay Raman <[email protected]>
@vinay-raman
removed retrieverconfig from modules.config and added individual para…
018658e 
…meters to filters constructor

Signed-off-by: Vinay Raman <[email protected]>
@vinay-raman
made changes to README
7632777 
Signed-off-by: Vinay Raman <[email protected]>
@vinay-raman
docs WIP
864218b 
Signed-off-by: Vinay Raman <[email protected]>
@vinay-raman
removed docs, will be in separate PR
092e21f 
Signed-off-by: Vinay Raman <[email protected]>
@vinay-raman
added table of contents to notebook, tested main.py
052044c 
Signed-off-by: Vinay Raman <[email protected]>
@vinay-raman vinay-raman force-pushed the sdg_pipeline/retriever_evalset_generation_signoff_fixed branch from d5dc0ae to 052044c Compare November 13, 2024 00:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ryantwolf ryantwolf ryantwolf requested changes

Requested changes must be addressed to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.