Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add datasets API to import graph data from configuration/metadata files #2367

Merged
merged 84 commits into from
Jul 25, 2022
Merged

Add datasets API to import graph data from configuration/metadata files #2367

merged 84 commits into from
Jul 25, 2022

Conversation

betochimas
Copy link
Contributor

@betochimas betochimas commented Jun 22, 2022 •
edited by rlratzel
Loading

Addresses issue #1348. A working version of the datasets API has been added under the "experimental" module of cuGraph. This API comes with the ability to import a handful of built-in datasets to create graphs and edge lists. Each dataset comes with its own metadata file in the format of a YAML file. These files contain general information about the dataset, as well as formatting information about their columns and datatypes.

betochimas and others added 30 commits June 7, 2022 13:54
@betochimas
Created initial datasets classes and first json metadata
37e111e
test comment
5abac2e
init method for metadata can read json
a4929df
create dolphin. add refs field
ec6333d
add author field
5a25495
@betochimas
Add initial small datasets to namespace
9235652
minor detalis to json
5fc06d4
add temp url to karate. read metadata
d074c25
@betochimas
Initial testing, plus comments
06f4d04
convert json to yaml
ee34f87
change extention to yaml
7432db4
remove unneeded test file
9b671ee
remove properties
717dbbe
remove metadata. fleshed out get_edgelist()
6e32c66
implement __download_csv
608517e
@betochimas
Created datasets_config yaml file plus other edits
acc3a4a
@betochimas
Passing initial test, generalized some dataset variables
8251f97
update yaml datatypes. prevent yaml sorting. reading config and stori…
57e8a3a 
…ng default dowload dir
@betochimas
Yaml edits plus adding new metadata files
ac0ca61
Removed a bug where graph was always set to directed=False
c65a522
writing method to load_all datasets from metadata/ dir
a927b96
@betochimas
Merge branch 'rapidsai:branch-22.08' into datasets-api-docs
023aae5
Created a general load_all() method to fetch all available datasets
1bcbc42
@betochimas
Merge branch 'rapidsai:branch-22.08' into datasets-api
b0df1ab
@betochimas
Merge branch 'rapidsai:branch-22.08' into datasets-api-docs
89d2752
@betochimas
Merge pull request #86 from betochimas/datasets-api-docs
c22a3ab 
Datasets api docs
@betochimas
Update docs
8d95be7
@betochimas
Metadata files are read-only, load_all works as long as target dir ex…
6fc469f 
…ists, update metadata yamls
@betochimas
Modify setup.py to include config yaml files, wip
c8505d3
Re-designed 'path' usage. Removed from Metadata files and stored loca…
9e8cae4 
…l ONLY
@rlratzel rlratzel marked this pull request as ready for review July 11, 2022 20:47
@rlratzel rlratzel requested a review from a team as a code owner July 11, 2022 20:47
rlratzel
rlratzel reviewed Jul 18, 2022
View reviewed changes
Copy link
Contributor

@rlratzel rlratzel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would also remove the metadata/__init__.py file if that directory does not contain any .py files.

python/cugraph/cugraph/experimental/datasets/dataset.py Outdated Show resolved Hide resolved
python/cugraph/cugraph/experimental/datasets/dataset.py Outdated Show resolved Hide resolved
python/cugraph/cugraph/experimental/datasets/dataset.py Show resolved Hide resolved
python/cugraph/cugraph/experimental/datasets/dataset.py Outdated Show resolved Hide resolved
python/cugraph/cugraph/experimental/datasets/dataset.py Outdated Show resolved Hide resolved
python/cugraph/cugraph/experimental/datasets/dataset.py Outdated Show resolved Hide resolved
python/cugraph/cugraph/experimental/datasets/dataset.py Outdated Show resolved Hide resolved
python/cugraph/cugraph/experimental/datasets/dataset.py Outdated Show resolved Hide resolved
python/cugraph/cugraph/experimental/datasets/dataset.py Outdated Show resolved Hide resolved
python/cugraph/cugraph/tests/test_dataset.py Outdated Show resolved Hide resolved
rlratzel
rlratzel requested changes Jul 21, 2022
View reviewed changes
Copy link
Contributor

@rlratzel rlratzel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few more minor requests

python/cugraph/cugraph/experimental/datasets/dataset.py Outdated Show resolved Hide resolved
python/cugraph/cugraph/experimental/datasets/dataset.py Show resolved Hide resolved
python/cugraph/cugraph/experimental/datasets/dataset.py Outdated Show resolved Hide resolved
python/cugraph/cugraph/experimental/datasets/dataset.py Outdated Show resolved Hide resolved
python/cugraph/cugraph/experimental/datasets/dataset.py Outdated Show resolved Hide resolved
python/cugraph/cugraph/experimental/datasets/dataset.py Outdated Show resolved Hide resolved
jnke2016
jnke2016 approved these changes Jul 25, 2022
View reviewed changes
Copy link
Contributor

@jnke2016 jnke2016 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me

@rlratzel
Copy link
Contributor

@gpucibot merge

@rapids-bot rapids-bot bot merged commit 6644813 into rapidsai:branch-22.08 Jul 25, 2022
@rlratzel
Copy link
Contributor

closes #1348

@rlratzel rlratzel mentioned this pull request Jul 25, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jnke2016 jnke2016 jnke2016 approved these changes

@rlratzel rlratzel rlratzel approved these changes

Assignees

@oorliu oorliu

Labels
improvement Improvement / enhancement to an existing function non-breaking Non-breaking change
Projects
None yet
Milestone
22.08
Development

Successfully merging this pull request may close these issues.
8 participants
@betochimas @codecov-commenter @raydouglass @rlratzel @jnke2016 @BradReesWork @oorliu @nv-rliu