Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Refactor][Graph] Merge DGLGraph and DGLHeteroGraph #1862

Merged
merged 144 commits into from
Jul 28, 2020
Merged

[Refactor][Graph] Merge DGLGraph and DGLHeteroGraph #1862

merged 144 commits into from
Jul 28, 2020

Conversation

jermainewang
Copy link
Member

@jermainewang jermainewang commented Jul 25, 2020 •
edited
Loading

Description

Merge DGLGraph and DGLHeteroGraph by rename DGLHeteroGraph with DGLGraph. The old DGLGraph is renamed to DGLGraphStale since some components are still not ready. Sweep all unittests, examples and tutorials for best-effort compatibility.

Breaking changes:

  • Graph and feature tensors now must have the same device context. Must call DGLGraph.to(device).
  • Removed APIs (with proper error messages to show users how to migrate)
    • DGLGraph.send, DGLGraph.recv
    • DGLGraph.register_apply_node_func, DGLGraph.register_apply_edge_func, DGLGraph.register_message_func, DGLGraph.register_reduce_func
    • init_ndata, init_edata, set_n_repr, get_n_repr, pop_n_repr, set_e_repr, get_e_repr, pop_e_repr
    • g.ndata and g.edata do not accept numpy arrays anymore.
    • DGLGraph.from_networkx -> dgl.from_networkx
    • DGLGraph.from_scipy_sparse_matrix -> dgl.from_scipy
  • BatchedHeteroGraph is deleted and merged to DGLGraph. DGLGraph.batch_num_nodes and DGLGraph.batch_num_edges are no longer properties but member functions to support heterogeneity.
  • [Tentative] When copying a graph to GPU, DGL automatically casts it to int32 ID type. This breaks several examples (also TF GPU unittests) and should be fixed. Need to add full int64 GPU support.

TODO:

  • Pass Tensorflow GPU tests.
  • JTNN example

Checklist

Please feel free to remove inapplicable items for your PR.

  • The PR title starts with [$CATEGORY] (such as [NN], [Model], [Doc], [Feature]])
  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage
  • Code is well-documented
  • To the my best knowledge, examples are either not affected by this change,
    or have been fixed to be compatible with this change
  • Related issue is referred in this PR

jermainewang and others added 30 commits July 13, 2020 16:22
@jermainewang
Merge
ec016e4
@jermainewang
Merge branch 'master' of github.com:dmlc/dgl into graph_refactor
b7ef313
@jermainewang
[Graph][CUDA] Graph on GPU and many refactoring (#1791)
a8b4b9c 
* change edge_ids behavior and C++ impl

* fix unittests; remove utils.Index in edge_id

* pass mx and th tests

* pass tf test

* add aten::Scatter_

* Add nonzero; impl CSRGetDataAndIndices/CSRSliceMatrix

* CSRGetData and CSRGetDataAndIndices passed tests

* CSRSliceMatrix basic tests

* fix bug in empty slice

* CUDA CSRHasDuplicate

* has_node; has_edge_between

* predecessors, successors

* deprecate send/recv; fix send_and_recv

* deprecate send/recv; fix send_and_recv

* in_edges; out_edges; all_edges; apply_edges

* in deg/out deg

* subgraph/edge_subgraph

* adj

* in_subgraph/out_subgraph

* sample neighbors

* set/get_n/e_repr

* wip: working on refactoring all idtypes

* pass ndata/edata tests on gpu

* fix

* stash

* workaround nonzero issue

* stash

* nx conversion

* test_hetero_basics except update routines

* test_update_routines

* test_hetero_basics for pytorch

* more fixes

* WIP: flatten graph

* wip: flatten

* test_flatten

* test_to_device

* fix bug in to_homo

* fix bug in CSRSliceMatrix

* pass subgraph test

* fix send_and_recv

* fix filter

* test_heterograph

* passed all pytorch tests

* fix mx unittest

* fix pytorch test_nn

* fix all unittests for PyTorch

* passed all mxnet tests

* lint

* fix tf nn test

* pass all tf tests

* lint

* lint

* change deprecation

* try fix compile

* lint

* update METIDS

* fix utest

* fix

* fix utests

* try debug

* revert

* small fix

* fix utests
@yzh119
upd
04dc998
@yzh119
upd
37a6807
@yzh119
upd
4f8732f
@yzh119
fix
6c99ff9
@yzh119
upd
9d61466
@yzh119
upd
5426e4d
@yzh119
upd
d06170a
@yzh119
upd
680d221
@yzh119
upd
414545f
@yzh119
trigger
251b36e
@yzh119
+1s
0a877b6
@yzh119
[kernel] Use heterograph index instead of unitgraph index (#1813)
40855ba 
* upd

* upd

* upd

* fix

* upd

* upd

* upd

* upd

* upd

* trigger

* +1s
@classicsong
[Graph] Mutation for Heterograph (#1818)
6f29ce2 
* mutation add_nodes and add_edges

* Add support for remove_edges, remove_nodes, add_selfloop, remove_selfloop

* Fix

Co-authored-by: Ubuntu <[email protected]>
@yzh119
upd
3645ace
@yzh119
Merge commit '3645ace0' into graph_refactor
86c02fc
@yzh119
Merge branch 'graph_refactor' of https://github.com/yzh119/dgl into g…
735d067 
…raph_refactor
@yzh119
upd
4248073
@yzh119
upd
a84a7d4
@yzh119
upd
1dbf282
@yzh119
fix
2ef52a8
@classicsong
[Transfom] Mutable transform (#1833)
8ffbbf6 
* add nodesy

* All three

* Fix

* lint

* Add some test case

* Fix

* Fix

* Fix

* Fix

* Fix

* Fix

* fix

* triger

* Fix

* fix

Co-authored-by: Ubuntu <[email protected]>
@jermainewang
[Graph] Migrate Batch & Readout module to heterograph (#1836)
e711a5f 
* dgl.batch

* unbatch

* fix to device

* reduce readout; segment reduce

* change batch_num_nodes|edges to function

* reduce readout/ softmax

* broadcast

* topk

* fix

* fix tf and mx

* fix some ci

* fix batch but unbatch differently

* new checkk
@yzh119
upd
48de21a
@yzh119
upd
ee35726
@yzh119
upd
3e11057
@jermainewang
idtype behavior; code reorg
d8c345e
@jermainewang
idtype behavior; code reorg
c7bf6db
jermainewang and others added 27 commits July 25, 2020 16:46
@jermainewang
Merge branch 'master' of github.com:dmlc/dgl into graph_refactor
158455d
Fix cluster gcn
695c714
Merge branch 'graph_refactor' of https://github.com/dmlc/dgl into gra…
1b89f65 
…ph_refactor
cluster-gcn is ready
dbaa1ab
@jermainewang
on fixing block related codes
3983584
@jermainewang
Merge branch 'graph_refactor' of github.com:dmlc/dgl into graph_refactor
e3f58e4
@yzh119
2nd order derivative
523bf6c
@yzh119
Revert "2nd order derivative"
4c8985f 
This reverts commit 523bf6c.
@jermainewang
passed torch utests again
87f7ddb
@jermainewang
fix all mxnet unittests
4b97e91
@jermainewang
delete some useless tests
6cba53d
@jermainewang
pass all tf cpu tests
707f048
@jermainewang
disable
bc29283
@jermainewang
Merge branch 'graph_refactor' of github.com:dmlc/dgl into graph_refactor
6186e8a
@jermainewang
disable distributed unittest
c3487b8
@jermainewang
Merge branch 'master' into graph_refactor
ad3b8e5
@jermainewang
fix
ef8ee7f
@jermainewang
Merge branch 'graph_refactor' of github.com:dmlc/dgl into graph_refactor
71bc080
@jermainewang
fix
411f64f
@jermainewang
lint
44a0584
@jermainewang
fix
a4bc16c
@jermainewang
fix
43fc4e7
@jermainewang
fix script
d3cc5d5
@jermainewang
fix tutorial
d9f9ccc
@jermainewang
fix apply edges bug
c281e2f
@jermainewang
fix 2 basics
9ecb313
@jermainewang
fix tutorial
4db7772
@jermainewang jermainewang merged commit 44089c8 into master Jul 28, 2020
@jermainewang jermainewang deleted the graph_refactor branch July 28, 2020 06:30
kingmbc pushed a commit to kingmbc/dgl that referenced this pull request Aug 2, 2020
@jermainewang @yzh119
@classicsong @kingmbc
[Refactor][Graph] Merge DGLGraph and DGLHeteroGraph (dmlc#1862)
01d4069 
* Merge

* [Graph][CUDA] Graph on GPU and many refactoring (dmlc#1791)

* change edge_ids behavior and C++ impl

* fix unittests; remove utils.Index in edge_id

* pass mx and th tests

* pass tf test

* add aten::Scatter_

* Add nonzero; impl CSRGetDataAndIndices/CSRSliceMatrix

* CSRGetData and CSRGetDataAndIndices passed tests

* CSRSliceMatrix basic tests

* fix bug in empty slice

* CUDA CSRHasDuplicate

* has_node; has_edge_between

* predecessors, successors

* deprecate send/recv; fix send_and_recv

* deprecate send/recv; fix send_and_recv

* in_edges; out_edges; all_edges; apply_edges

* in deg/out deg

* subgraph/edge_subgraph

* adj

* in_subgraph/out_subgraph

* sample neighbors

* set/get_n/e_repr

* wip: working on refactoring all idtypes

* pass ndata/edata tests on gpu

* fix

* stash

* workaround nonzero issue

* stash

* nx conversion

* test_hetero_basics except update routines

* test_update_routines

* test_hetero_basics for pytorch

* more fixes

* WIP: flatten graph

* wip: flatten

* test_flatten

* test_to_device

* fix bug in to_homo

* fix bug in CSRSliceMatrix

* pass subgraph test

* fix send_and_recv

* fix filter

* test_heterograph

* passed all pytorch tests

* fix mx unittest

* fix pytorch test_nn

* fix all unittests for PyTorch

* passed all mxnet tests

* lint

* fix tf nn test

* pass all tf tests

* lint

* lint

* change deprecation

* try fix compile

* lint

* update METIDS

* fix utest

* fix

* fix utests

* try debug

* revert

* small fix

* fix utests

* upd

* upd

* upd

* fix

* upd

* upd

* upd

* upd

* upd

* trigger

* +1s

* [kernel] Use heterograph index instead of unitgraph index (dmlc#1813)

* upd

* upd

* upd

* fix

* upd

* upd

* upd

* upd

* upd

* trigger

* +1s

* [Graph] Mutation for Heterograph (dmlc#1818)

* mutation add_nodes and add_edges

* Add support for remove_edges, remove_nodes, add_selfloop, remove_selfloop

* Fix

Co-authored-by: Ubuntu <[email protected]>

* upd

* upd

* upd

* fix

* [Transfom] Mutable transform (dmlc#1833)

* add nodesy

* All three

* Fix

* lint

* Add some test case

* Fix

* Fix

* Fix

* Fix

* Fix

* Fix

* fix

* triger

* Fix

* fix

Co-authored-by: Ubuntu <[email protected]>

* [Graph] Migrate Batch & Readout module to heterograph (dmlc#1836)

* dgl.batch

* unbatch

* fix to device

* reduce readout; segment reduce

* change batch_num_nodes|edges to function

* reduce readout/ softmax

* broadcast

* topk

* fix

* fix tf and mx

* fix some ci

* fix batch but unbatch differently

* new checkk

* upd

* upd

* upd

* idtype behavior; code reorg

* idtype behavior; code reorg

* wip: test_basics

* pass test_basics

* WIP: from nx/ to nx

* missing files

* upd

* pass test_basics:test_nx_conversion

* Fix test

* Fix inplace update

* WIP: fixing tests

* upd

* pass test_transform cpu

* pass gpu test_transform

* pass test_batched_graph

* GPU graph auto cast to int32

* missing file

* stash

* WIP: rgcn-hetero

* Fix two datasety

* upd

* weird

* Fix capsuley

* fuck you

* fuck matthias

* Fix dgmg

* fix bug in block degrees; pass rgcn-hetero

* rgcn

* gat and diffpool fix
also fix ppi and tu dataset

* Tree LSTM

* pointcloud

* rrn; wip: sgc

* resolve conflicts

* upd

* sgc and reddit dataset

* upd

* Fix deepwalk, gindt and gcn

* fix datasets and sign

* optimization

* optimization

* upd

* upd

* Fix GIN

* fix bug in add_nodes add_edges; tagcn

* adaptive sampling and gcmc

* upd

* upd

* fix geometric

* fix

* metapath2vec

* fix agnn

* fix pickling problem of block

* fix utests

* miss file

* linegraph

* upd

* upd

* upd

* graphsage

* stgcn_wave

* fix hgt

* on unittests

* Fix transformer

* Fix HAN

* passed pytorch unittests

* lint

* fix

* Fix cluster gcn

* cluster-gcn is ready

* on fixing block related codes

* 2nd order derivative

* Revert "2nd order derivative"

This reverts commit 523bf6c.

* passed torch utests again

* fix all mxnet unittests

* delete some useless tests

* pass all tf cpu tests

* disable

* disable distributed unittest

* fix

* fix

* lint

* fix

* fix

* fix script

* fix tutorial

* fix apply edges bug

* fix 2 basics

* fix tutorial

Co-authored-by: yzh119 <[email protected]>
Co-authored-by: xiang song(charlie.song) <[email protected]>
Co-authored-by: Ubuntu <[email protected]>
Co-authored-by: Ubuntu <[email protected]>
Co-authored-by: Ubuntu <[email protected]>
Co-authored-by: Ubuntu <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jermainewang @yzh119 @classicsong