Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

G-retriever API updates (NVTX, Remote Backend, Large Graph Indexer, Examples) #9666

Merged
merged 1,012 commits into from
Nov 26, 2024
Merged

G-retriever API updates (NVTX, Remote Backend, Large Graph Indexer, Examples) #9666

merged 1,012 commits into from
Nov 26, 2024

Conversation

zaristei
Copy link
Contributor

@zaristei zaristei commented Sep 16, 2024 •
edited by puririshi98
Loading

Follow up to PR 9597. Includes multiple changes related to LLM+GNN experiments and scaling up to a remote backend. Including:

  • LargeGraphIndexer for building a large knowledge graph locally from multiple samples in an arbitrary dataset
  • Remote Backend Loader and examples for deploying a Retrieval algorithm to a third party backend FeatureStore or GraphStore
  • NVTX profiling tools for nsys users
  • Quality of Life improvements and benchmarking scripts for G-Retriever.

Updates using these for WebQSP will be moved to a seperate PR

UPDATE:
PR is being broken up into smaller PRs. These can be previewed here:

Zack Aristei and others added 30 commits August 28, 2024 15:03
@zaristei
finished dataloader
b912652
@zaristei
finished dataloader 2
ce5370b
@zaristei
add to data transform
dc3f9f4
@zaristei
init commit of dataset
4488d70
@zaristei
debug pcst part 1
eb9b761
@zaristei
push what i have for now
85da0e7
@zaristei
fixes after rebase
168a37a
@zaristei
more fixes and mocking
2cf727f
@zaristei
more mocking
9ab9b22
@zaristei
main mocking working
6a1ee0f
@zaristei
pr fixes
03f6f4e
@zaristei
migrate large graph indexer
0b4ee3d
@zaristei
fix save overrider
81a1b8a
@zaristei
formatting 1
dd2ec56
@zaristei
formatting 2
917fe54
@zaristei
start unittests
ae8d057
@zaristei
tests done for largegraphindexer
7fdae85
@zaristei
tests done for largegraphindexer 2
5f67a84
@zaristei
tests done for largegraphindexer 3
b0ab66f
@zaristei
tests done for largegraphindexer 4
bfd1a2f
@zaristei
migrate updated qsp dataset
bcee67a
@zaristei
formatting
2117605
@zaristei
add dataset
866866c
@zaristei
add option for new dataloader
70f0380
@zaristei
fix formatting
db94397
@zaristei
instantiate ds in train func
a03028a
@zaristei
Restore mapping attrs
0ec7aac
@zaristei
Test edited to cover restoration of data info
c1e5cc8
@zaristei
begin trying profiling
5a09881
@zaristei
speedup retrieval to avoid timeouts
9d1093c
zaristei and others added 15 commits November 23, 2024 09:16
@zaristei
make webqsp example use current webqsp implementation
ec61ec3
@zaristei
col limits on prints
a7f7f25
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
23dd66c 
for more information, see https://pre-commit.ci
@zaristei
col limits on prints 2
14ae2fb
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
c1380ba 
for more information, see https://pre-commit.ci
@zaristei
col limits on prints 3
14e0376
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
204ddcb 
for more information, see https://pre-commit.ci
@zaristei
col limits on prints 4
0fa20ad
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
8f63e66 
for more information, see https://pre-commit.ci
@zaristei
col limits on prints 5
51129d0
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
79c43b1 
for more information, see https://pre-commit.ci
@zaristei
col limits on prints 6
278cada
@zaristei
pre-commit fix
581e184
@puririshi98
Merge branch 'master' into zaristei/g_retriever_experiments
257319a
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
a7cfc61 
for more information, see https://pre-commit.ci
zaristei
zaristei commented Nov 25, 2024
View reviewed changes
docs/source/advanced/rag.rst Outdated Show resolved Hide resolved
docs/source/advanced/rag.rst Outdated Show resolved Hide resolved
docs/source/advanced/rag.rst Outdated Show resolved Hide resolved
docs/source/advanced/rag.rst Outdated Show resolved Hide resolved
test/data/test_large_graph_indexer.py Outdated Show resolved Hide resolved
@puririshi98 puririshi98 changed the title G Retriever Experiments and Improvements (full) G-retriever API updates (NVTX, Remote Backend, Large Graph Indexer, Examples) Nov 25, 2024
puririshi98
puririshi98 requested changes Nov 25, 2024
View reviewed changes
Copy link
Contributor

@puririshi98 puririshi98 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just make new PR w/ webqsp changes and move advanced/rag.rst to that. and apply the fix for skipping pytest if not correct versioning.

test/data/test_large_graph_indexer.py Outdated Show resolved Hide resolved
zaristei and others added 8 commits November 25, 2024 19:14
@zaristei
large graph indexer test skip with wrong version
0e69cd0
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
794e487 
for more information, see https://pre-commit.ci
@zaristei
Remove outdated docs 1
f0ffc14 
Will be re-added with an updated version of WebQSP
@zaristei
Remove outdated docs 2
147fc84 
Will be re-added with an updated version of WebQSP
@zaristei
Remove outdated docs 3
2140250 
Will be re-added with an updated version of WebQSP
@zaristei
Remove outdated docs 4
32ad70a 
Will be re-added with an updated version of WebQSP
@zaristei
Remove outdated docs 5
13301f0 
Will be re-added with an updated version of WebQSP
@zaristei
lint fix
b5069e2
@zaristei zaristei mentioned this pull request Nov 25, 2024
Draft
@zaristei
Copy link
Contributor Author

zaristei commented Nov 25, 2024 •
edited
Loading

just make new PR w/ webqsp changes and move advanced/rag.rst to that. and apply the fix for skipping pytest if not correct versioning.

New PR to be found here for WebQSP and doc changes

puririshi98
puririshi98 approved these changes Nov 26, 2024
View reviewed changes
Copy link
Contributor

@puririshi98 puririshi98 left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Has been reviewed 10+ times over the last few months. Code tested. @zaristei has addressed all reviews. Most of the reviews are much above

@puririshi98 puririshi98 merged commit 742f790 into pyg-team:master Nov 26, 2024
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@puririshi98 puririshi98 puririshi98 approved these changes

@wsad1 wsad1 Awaiting requested review from wsad1 wsad1 is a code owner

@rusty1s rusty1s Awaiting requested review from rusty1s rusty1s is a code owner

@mananshah99 mananshah99 Awaiting requested review from mananshah99 mananshah99 is a code owner

@EdisonLeeeee EdisonLeeeee Awaiting requested review from EdisonLeeeee EdisonLeeeee is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@zaristei @puririshi98 @zacool64