Make GPU dependencies optional #27

ayushdg · 2024-04-09T17:30:43Z

Moves some of the GPU imports into optional import blocks.
Split up the install into an extra cu12 style install for GPU components.
Update README's
Update installation in CI testing.

ayushdg · 2024-04-22T19:37:24Z

I'm seeing two new errors pop up probably because of newer pandas/dask versions (cudf was probably setting the upper pin to something more restrictive. I need to look into this a bit more but pinging @ryantwolf in case something immediately comes to mind.

ryantwolf · 2024-04-22T21:17:31Z

Nothing immediately comes to mind unfortunately. I haven't seen those errors before. Happy to help debug more though if you want help.

Signed-off-by: Ayush Dattagupta <[email protected]>

ryantwolf

LGTM. I only had one question for my own curiousity. Just one final check before I approve, have you run the RAPIDS dedup CI tests with this change?

nemo_curator/datasets/doc_dataset.py

ayushdg · 2024-04-23T20:35:52Z

LGTM. I only had one question for my own curiousity. Just one final check before I approve, have you run the RAPIDS dedup CI tests with this change?

Yup I've manually run the pipeline and verified that it works as expected

* Move GPU imports and make them optional Signed-off-by: Ayush Dattagupta <[email protected]> * Move gpu dependencies to a seperate install Signed-off-by: Ayush Dattagupta <[email protected]> * Remove unused import Signed-off-by: Ayush Dattagupta <[email protected]> * Switch to placeholder import that raises on usage Signed-off-by: Ayush Dattagupta <[email protected]> * Remove deprecated utils usage Signed-off-by: Ayush Dattagupta <[email protected]> * Add cuML attribution Signed-off-by: Ayush Dattagupta <[email protected]> * Safe import tests, improve install instruction, update gha workflow Signed-off-by: Ayush Dattagupta <[email protected]> * Fix pytests due to loc bug Signed-off-by: Ayush Dattagupta <[email protected]> * update install instructions Signed-off-by: Ayush Dattagupta <[email protected]> * Raise on non module-not-found errors, update logging Signed-off-by: Ayush Dattagupta <[email protected]> * Update logging to not change root logger Signed-off-by: Ayush Dattagupta <[email protected]> --------- Signed-off-by: Ayush Dattagupta <[email protected]> Signed-off-by: Nicole Luo <[email protected]>

* Init commit for tutorial notebook Signed-off-by: Nicole Luo <[email protected]> * Fix metadata inference with pandas and dask (#35) * Fix metadata inference with pandas and dask Signed-off-by: Ryan Wolf <[email protected]> * Fix datatypes for task decontamination Signed-off-by: Ryan Wolf <[email protected]> * Use targetted import Signed-off-by: Ryan Wolf <[email protected]> --------- Signed-off-by: Ryan Wolf <[email protected]> Signed-off-by: Nicole Luo <[email protected]> * Disable PyTorch Compile Multiprocessing (#34) * Move tokenizer import Signed-off-by: Ryan Wolf <[email protected]> * Reduce inductor threads Signed-off-by: Ryan Wolf <[email protected]> * Change env int to string Signed-off-by: Ryan Wolf <[email protected]> * Change location of env var Signed-off-by: Ryan Wolf <[email protected]> * Add comment linking issue Signed-off-by: Ryan Wolf <[email protected]> --------- Signed-off-by: Ryan Wolf <[email protected]> Signed-off-by: Nicole Luo <[email protected]> * Improve speed of AddId module (#36) * Add fast id method Signed-off-by: Ryan Wolf <[email protected]> * Add type conversion Signed-off-by: Ryan Wolf <[email protected]> * Fix off by one errors in tests Signed-off-by: Ryan Wolf <[email protected]> --------- Signed-off-by: Ryan Wolf <[email protected]> Signed-off-by: Nicole Luo <[email protected]> * Make GPU dependencies optional (#27) * Move GPU imports and make them optional Signed-off-by: Ayush Dattagupta <[email protected]> * Move gpu dependencies to a seperate install Signed-off-by: Ayush Dattagupta <[email protected]> * Remove unused import Signed-off-by: Ayush Dattagupta <[email protected]> * Switch to placeholder import that raises on usage Signed-off-by: Ayush Dattagupta <[email protected]> * Remove deprecated utils usage Signed-off-by: Ayush Dattagupta <[email protected]> * Add cuML attribution Signed-off-by: Ayush Dattagupta <[email protected]> * Safe import tests, improve install instruction, update gha workflow Signed-off-by: Ayush Dattagupta <[email protected]> * Fix pytests due to loc bug Signed-off-by: Ayush Dattagupta <[email protected]> * update install instructions Signed-off-by: Ayush Dattagupta <[email protected]> * Raise on non module-not-found errors, update logging Signed-off-by: Ayush Dattagupta <[email protected]> * Update logging to not change root logger Signed-off-by: Ayush Dattagupta <[email protected]> --------- Signed-off-by: Ayush Dattagupta <[email protected]> Signed-off-by: Nicole Luo <[email protected]> * Fix failing GPU tests with latest pandas bump (#41) Signed-off-by: Ayush Dattagupta <[email protected]> Signed-off-by: Nicole Luo <[email protected]> * Adds Nemo Curator K8s example (#40) * [K8s]: Adds a helper script to create a dask cluster on k8s and includes instructions for how to a Curator workload on k8s Signed-off-by: Terry Kong <[email protected]> * black formatting Signed-off-by: Terry Kong <[email protected]> * big_english -> my_dataset Signed-off-by: Terry Kong <[email protected]> * 24.01 -> 24.03 default container Signed-off-by: Terry Kong <[email protected]> * Add help kwarg to all flags Signed-off-by: Terry Kong <[email protected]> * Clarify why venv is needed Signed-off-by: Terry Kong <[email protected]> * fix precommit failures Signed-off-by: Terry Kong <[email protected]> --------- Signed-off-by: Terry Kong <[email protected]> Signed-off-by: Nicole Luo <[email protected]> * Move common dedup utils and remove unused code (#42) * Refactor common utils and remove unused code Signed-off-by: Ayush Dattagupta <[email protected]> * More cleanup Signed-off-by: Ayush Dattagupta <[email protected]> * More updates/shuffling Signed-off-by: Ayush Dattagupta <[email protected]> * Move gpu_dedup scripts into subfolder Signed-off-by: Ayush Dattagupta <[email protected]> * Remove gpu_deduplication subfolder Signed-off-by: Ayush Dattagupta <[email protected]> * Add readme to fuzzy dedup scripts section Signed-off-by: Ayush Dattagupta <[email protected]> * Fix typo and relative links Signed-off-by: Ayush Dattagupta <[email protected]> * Remove legacy script entrypoints Signed-off-by: Ayush Dattagupta <[email protected]> * Remove legacy scripts and add init file Signed-off-by: Ayush Dattagupta <[email protected]> * Update GpuDeduplication.rst Signed-off-by: Ayush Dattagupta <[email protected]> --------- Signed-off-by: Ayush Dattagupta <[email protected]> Signed-off-by: Nicole Luo <[email protected]> * Fix lang id example (#37) * Fix lang id example Signed-off-by: Ryan Wolf <[email protected]> * Add classifier unit tests Signed-off-by: Ryan Wolf <[email protected]> * Add test for failure Signed-off-by: Ryan Wolf <[email protected]> * Remove failure test Signed-off-by: Ryan Wolf <[email protected]> --------- Signed-off-by: Ryan Wolf <[email protected]> Signed-off-by: Nicole Luo <[email protected]> * Add dataset blending tool (#32) * Add initial dataset blending function Signed-off-by: Ryan Wolf <[email protected]> * Add blend unit tests Signed-off-by: Ryan Wolf <[email protected]> * Add self parameter Signed-off-by: Ryan Wolf <[email protected]> * Fix return type of blend dataset Signed-off-by: Ryan Wolf <[email protected]> * Fix blending tests Signed-off-by: Ryan Wolf <[email protected]> * Change assert statement for very uneven blend Signed-off-by: Ryan Wolf <[email protected]> * Fix key error Signed-off-by: Ryan Wolf <[email protected]> * Add proper proportion blending test Signed-off-by: Ryan Wolf <[email protected]> * Add four dataset blend and clarify docs Signed-off-by: Ryan Wolf <[email protected]> * Add shuffle module Signed-off-by: Ryan Wolf <[email protected]> * Add blend example and tests Signed-off-by: Ryan Wolf <[email protected]> * Fix random method name Signed-off-by: Ryan Wolf <[email protected]> * Wrap return type in DocumentDataset Signed-off-by: Ryan Wolf <[email protected]> * Save result of column drop Signed-off-by: Ryan Wolf <[email protected]> * Change equality check for shuffle tests Signed-off-by: Ryan Wolf <[email protected]> * Fix expected order after shuffle Signed-off-by: Ryan Wolf <[email protected]> * Add more documents to shuffle test Signed-off-by: Ryan Wolf <[email protected]> * Add assert statement Signed-off-by: Ryan Wolf <[email protected]> * Add within partition shuffle Signed-off-by: Ryan Wolf <[email protected]> * Refactor add rand column for shuffle Signed-off-by: Ryan Wolf <[email protected]> * Fix filename tests Signed-off-by: Ryan Wolf <[email protected]> * Add determinism handling for shuffle Signed-off-by: Ryan Wolf <[email protected]> * Change numpy random function Signed-off-by: Ryan Wolf <[email protected]> * Fix tests with new random method Signed-off-by: Ryan Wolf <[email protected]> * Remove length call from blending Signed-off-by: Ryan Wolf <[email protected]> * Improve scaling of blending function Signed-off-by: Ryan Wolf <[email protected]> * Fix blend tests Signed-off-by: Ryan Wolf <[email protected]> * Add blending script Signed-off-by: Ryan Wolf <[email protected]> * Add additional file paths call Signed-off-by: Ryan Wolf <[email protected]> * Add documentation Signed-off-by: Ryan Wolf <[email protected]> * Reformat docs Signed-off-by: Ryan Wolf <[email protected]> * Remove backticks Signed-off-by: Ryan Wolf <[email protected]> * Add context manager for shuffle tests Signed-off-by: Ryan Wolf <[email protected]> * Add better deterministic shuffle path Signed-off-by: Ryan Wolf <[email protected]> * Update documentation and reset index Signed-off-by: Ryan Wolf <[email protected]> --------- Signed-off-by: Ryan Wolf <[email protected]> Signed-off-by: Nicole Luo <[email protected]> * High level fuzzy duplicates module (#46) * Initial pass at fuzzy dedup api Signed-off-by: Ayush Dattagupta <[email protected]> * Update deprecated shuffle arg Signed-off-by: Ayush Dattagupta <[email protected]> * dask_cuda gpu only import Signed-off-by: Ayush Dattagupta <[email protected]> * Move fuzzy_dedup imports to optional Signed-off-by: Ayush Dattagupta <[email protected]> * more tests Signed-off-by: Ayush Dattagupta <[email protected]> * Move FuzzyDeDupConfig to it's own class Signed-off-by: Ayush Dattagupta <[email protected]> * Add example script and config file, fix typo Signed-off-by: Ayush Dattagupta <[email protected]> * Remove slurm examples for gpu dedup Signed-off-by: Ayush Dattagupta <[email protected]> * Add config module Signed-off-by: Ayush Dattagupta <[email protected]> * Rename FuzzyDeDupConfig and minhash_length to FuzzyDuplicatesConfig, num_hashes Signed-off-by: Ayush Dattagupta <[email protected]> * Add comments and update example Signed-off-by: Ayush Dattagupta <[email protected]> * Write to same format as input in fuzzy dedup example Signed-off-by: Ayush Dattagupta <[email protected]> --------- Signed-off-by: Ayush Dattagupta <[email protected]> Signed-off-by: Nicole Luo <[email protected]> * Fix indexing in PII Modifier (#55) * Fix pii index issue Signed-off-by: Ryan Wolf <[email protected]> * Add sequential wrapper Signed-off-by: Ryan Wolf <[email protected]> * Fix pii tests Signed-off-by: Ryan Wolf <[email protected]> --------- Signed-off-by: Ryan Wolf <[email protected]> Signed-off-by: Nicole Luo <[email protected]> * Disable string conversion globally (#56) Signed-off-by: Ryan Wolf <[email protected]> Signed-off-by: Nicole Luo <[email protected]> * Fix issue #43 (empty files creation) and improve reading/writing speed (#57) This commit fixes issue #43 (empty files created when invoking reshard_jsonl method at nemo_curator.utils.file_utils.py) by double-checking the files size after being generated, and deleting them with size zero. In addition to that, I have noticed there is no need to parse to JSON object the content of the different lines, which should be already in json format. By removing that extra-parsing, there is a significant speed up in the execution of this method. Signed-off-by: Miguel Martínez <[email protected]> Signed-off-by: Nicole Luo <[email protected]> * [Tutorials] Add a tutorial for PEFT data curation (#45) This PR adds a new tutorial to demonstrate data curation for PEFT use-cases. Signed-off-by: Mehran Maghoumi <[email protected]> Signed-off-by: Nicole Luo <[email protected]> * Only import PII constants during Curator import (#61) * Move PII constants to a seperate file that does not import presidio/spacy and other GPU dependencies Signed-off-by: Ayush Dattagupta <[email protected]> * Add comment around import, move constant import to global scope Signed-off-by: Ayush Dattagupta <[email protected]> --------- Signed-off-by: Ayush Dattagupta <[email protected]> Signed-off-by: Nicole Luo <[email protected]> * Deleting links Signed-off-by: Nicoel Luo <[email protected]> Signed-off-by: Nicole Luo <[email protected]> * Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb Co-authored-by: Ryan Wolf <[email protected]> Signed-off-by: nicoleeeluo <[email protected]> Signed-off-by: Nicole Luo <[email protected]> * Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb Co-authored-by: Ryan Wolf <[email protected]> Signed-off-by: nicoleeeluo <[email protected]> Signed-off-by: Nicole Luo <[email protected]> * Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb Co-authored-by: Ryan Wolf <[email protected]> Signed-off-by: nicoleeeluo <[email protected]> Signed-off-by: Nicole Luo <[email protected]> * Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb Co-authored-by: Ryan Wolf <[email protected]> Signed-off-by: nicoleeeluo <[email protected]> Signed-off-by: Nicole Luo <[email protected]> * Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb Co-authored-by: Ryan Wolf <[email protected]> Signed-off-by: nicoleeeluo <[email protected]> Signed-off-by: Nicole Luo <[email protected]> * Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb Co-authored-by: Ryan Wolf <[email protected]> Signed-off-by: nicoleeeluo <[email protected]> Signed-off-by: Nicole Luo <[email protected]> * Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb Co-authored-by: Ryan Wolf <[email protected]> Signed-off-by: nicoleeeluo <[email protected]> Signed-off-by: Nicole Luo <[email protected]> * Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb Co-authored-by: Ryan Wolf <[email protected]> Signed-off-by: nicoleeeluo <[email protected]> Signed-off-by: Nicole Luo <[email protected]> * Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb Co-authored-by: Ryan Wolf <[email protected]> Signed-off-by: nicoleeeluo <[email protected]> Signed-off-by: Nicole Luo <[email protected]> * Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb Co-authored-by: Ryan Wolf <[email protected]> Signed-off-by: nicoleeeluo <[email protected]> Signed-off-by: Nicole Luo <[email protected]> * Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb Co-authored-by: Ryan Wolf <[email protected]> Signed-off-by: nicoleeeluo <[email protected]> Signed-off-by: Nicole Luo <[email protected]> * Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb Co-authored-by: Ryan Wolf <[email protected]> Signed-off-by: nicoleeeluo <[email protected]> Signed-off-by: Nicole Luo <[email protected]> * Fixed typo. Update content to lastest NeMo Curator version. Added fuzzy deduplication wrapper example Signed-off-by: Nicole Luo <[email protected]> * Fixing Style Signed-off-by: Nicole Luo <[email protected]> * Updating container version Signed-off-by: Nicole Luo <[email protected]> * Fixing style Signed-off-by: Nicole Luo <[email protected]> * Update get_client() according to latest version; Update log path for map_bucket section Signed-off-by: Nicole Luo <[email protected]> --------- Signed-off-by: Nicole Luo <[email protected]> Signed-off-by: Ryan Wolf <[email protected]> Signed-off-by: Ayush Dattagupta <[email protected]> Signed-off-by: Terry Kong <[email protected]> Signed-off-by: Miguel Martínez <[email protected]> Signed-off-by: Mehran Maghoumi <[email protected]> Signed-off-by: Nicoel Luo <[email protected]> Signed-off-by: nicoleeeluo <[email protected]> Co-authored-by: Ryan Wolf <[email protected]> Co-authored-by: Ayush Dattagupta <[email protected]> Co-authored-by: Terry Kong <[email protected]> Co-authored-by: Miguel Martínez <[email protected]> Co-authored-by: Mehran Maghoumi <[email protected]> Co-authored-by: Ryan Wolf <[email protected]>

ayushdg force-pushed the optional-gpu-imports branch from d175da6 to d0307f8 Compare April 9, 2024 17:34

ayushdg force-pushed the optional-gpu-imports branch from 9255b04 to 8b77262 Compare April 22, 2024 19:27

ayushdg force-pushed the optional-gpu-imports branch from 8b77262 to 9f38aa1 Compare April 23, 2024 01:55

ayushdg marked this pull request as ready for review April 23, 2024 03:04

ayushdg requested a review from ryantwolf April 23, 2024 18:33

ayushdg added 11 commits April 23, 2024 11:35

Move GPU imports and make them optional

5068f04

Signed-off-by: Ayush Dattagupta <[email protected]>

Move gpu dependencies to a seperate install

6c56352

Signed-off-by: Ayush Dattagupta <[email protected]>

Remove unused import

48a4da2

Signed-off-by: Ayush Dattagupta <[email protected]>

Switch to placeholder import that raises on usage

edd926b

Signed-off-by: Ayush Dattagupta <[email protected]>

Remove deprecated utils usage

3bed23d

Signed-off-by: Ayush Dattagupta <[email protected]>

Add cuML attribution

754f3e4

Signed-off-by: Ayush Dattagupta <[email protected]>

Safe import tests, improve install instruction, update gha workflow

d080e6e

Signed-off-by: Ayush Dattagupta <[email protected]>

Fix pytests due to loc bug

7d68763

Signed-off-by: Ayush Dattagupta <[email protected]>

update install instructions

f520fb9

Signed-off-by: Ayush Dattagupta <[email protected]>

Raise on non module-not-found errors, update logging

f18444f

Signed-off-by: Ayush Dattagupta <[email protected]>

Update logging to not change root logger

f508c30

Signed-off-by: Ayush Dattagupta <[email protected]>

ayushdg force-pushed the optional-gpu-imports branch from 7bc1536 to f508c30 Compare April 23, 2024 18:37

ryantwolf reviewed Apr 23, 2024

View reviewed changes

nemo_curator/datasets/doc_dataset.py Show resolved Hide resolved

ryantwolf self-requested a review April 23, 2024 20:39

ryantwolf approved these changes Apr 23, 2024

View reviewed changes

ryantwolf merged commit 17e0d5f into NVIDIA:main Apr 23, 2024
3 checks passed

ayushdg mentioned this pull request Apr 23, 2024

Make NeMo-Curator installable in non GPU environments #4

Closed

ayushdg deleted the optional-gpu-imports branch June 3, 2024 17:18

ayushdg mentioned this pull request Jun 18, 2024

Fix nemo_curator import in CPU only environment when GPU packages are installed. #123

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make GPU dependencies optional #27

Make GPU dependencies optional #27

ayushdg commented Apr 9, 2024 •

edited

Loading

ayushdg commented Apr 22, 2024

ryantwolf commented Apr 22, 2024

ryantwolf left a comment

ayushdg commented Apr 23, 2024

Make GPU dependencies optional #27

Make GPU dependencies optional #27

Conversation

ayushdg commented Apr 9, 2024 • edited Loading

ayushdg commented Apr 22, 2024

ryantwolf commented Apr 22, 2024

ryantwolf left a comment

Choose a reason for hiding this comment

ayushdg commented Apr 23, 2024

ayushdg commented Apr 9, 2024 •

edited

Loading