Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Branch 23.06 Fix Forward Merge #3462

Merged
merged 29 commits into from
Apr 10, 2023

Commits on Mar 31, 2023

  1. Refactor betweenness centrality (rapidsai#2971)

    A CAPI implementation of betweenness centrality is available and this PR:
    1. Implement PLC betweenness centrality by leveraging the CAPI
    2. Refactor the python SG implementation of betweenness centrality by leveraging the PLC implementation
    3. Add a python MG implementation of betweenness centrality with tests
    
    closes rapidsai#3145 
    closes rapidsai#2605
    closes rapidsai#2648 
    closes rapidsai#2649
    closes rapidsai#2650
    
    Authors:
      - Joseph Nke (https://github.com/jnke2016)
      - Chuck Hastings (https://github.com/ChuckHastings)
      - Rick Ratzel (https://github.com/rlratzel)
      - Alex Barghi (https://github.com/alexbarghi-nv)
      - Vyas Ramasubramani (https://github.com/vyasr)
      - Vibhu Jawa (https://github.com/VibhuJawa)
    
    Approvers:
      - Rick Ratzel (https://github.com/rlratzel)
      - Chuck Hastings (https://github.com/ChuckHastings)
      - Alex Barghi (https://github.com/alexbarghi-nv)
      - Brad Rees (https://github.com/BradReesWork)
    
    URL: rapidsai#2971
    jnke2016 authored Mar 31, 2023
    Configuration menu
    Copy the full SHA
    9df94ee View commit details
    Browse the repository at this point in the history
  2. resolve merge conflict

    alexbarghi-nv committed Mar 31, 2023
    Configuration menu
    Copy the full SHA
    c7e35e7 View commit details
    Browse the repository at this point in the history

Commits on Apr 2, 2023

  1. Multi-trainers cugraph-DGL examples (rapidsai#3212)

    This PR adds a working Multi-GPU Graph (on 2 dask workers)  being trained/loaded on multiple pytorch trainers.  (3)
    
    Todo: 
    - [x] Verify works on multiple trainers and multiple dask workers
    - [x] Show scaling as you increase training GPUs 
    
    At 1 second we become bottlenecked by sampling dask cluster, but we see perf improvement by going from `1 GPU`->`2GPU`.   
    **On OBGN-Products**
    ```md
    | Number of Training GPUs | Time per epoch |
    |-------------------------|----------------|
    | 1                       | 2.3 s          |
    | 2                       | 0.582 s        |
    | 4                       | 0.792 s        |
    ```
    
    This PR depends upon:  rapidsai#3393
    CC: @rlratzel , @alexbarghi-nv , @BradReesWork
    
    Authors:
      - Vibhu Jawa (https://github.com/VibhuJawa)
      - Alex Barghi (https://github.com/alexbarghi-nv)
    
    Approvers:
      - Alex Barghi (https://github.com/alexbarghi-nv)
    
    URL: rapidsai#3212
    VibhuJawa authored Apr 2, 2023
    Configuration menu
    Copy the full SHA
    f4a0778 View commit details
    Browse the repository at this point in the history
  2. New cugraph site structure (rapidsai#3343)

    resolves rapidsai#3297 
    
    new website structure with some migrations.
    
    Authors:
      - Don Acosta (https://github.com/acostadon)
      - Brad Rees (https://github.com/BradReesWork)
    
    Approvers:
      - Brad Rees (https://github.com/BradReesWork)
      - Alex Barghi (https://github.com/alexbarghi-nv)
    
    URL: rapidsai#3343
    acostadon authored Apr 2, 2023
    Configuration menu
    Copy the full SHA
    b10978c View commit details
    Browse the repository at this point in the history
  3. Update cuGraph-PyG Tests and Support Loading Saved Bulk Samples (rapi…

    …dsai#3289)
    
    Updates tests to support presence of PyTorch and PyG in the environment.
    
    Updates cugraph-pyg to include support for loading from samples on disk. Makes CuGraphStore serializable and allows creating CuGraphStore and BulkSampleLoader instances without the graph structure.
    
    Merge after rapidsai#3288 
    
    Closes rapidsai#3287 
    Closes rapidsai#3176
    
    Authors:
      - Alex Barghi (https://github.com/alexbarghi-nv)
    
    Approvers:
      - Vibhu Jawa (https://github.com/VibhuJawa)
      - Rick Ratzel (https://github.com/rlratzel)
    
    URL: rapidsai#3289
    alexbarghi-nv authored Apr 2, 2023
    Configuration menu
    Copy the full SHA
    b34e19a View commit details
    Browse the repository at this point in the history

Commits on Apr 3, 2023

  1. Generate pyproject dependencies using dfg (rapidsai#3355)

    This PR uses dependencies.yaml to generate the dependency lists in pyproject.toml
    
    Authors:
      - Vyas Ramasubramani (https://github.com/vyasr)
    
    Approvers:
      - AJ Schmidt (https://github.com/ajschmidt8)
      - Brad Rees (https://github.com/BradReesWork)
    
    URL: rapidsai#3355
    vyasr authored Apr 3, 2023
    Configuration menu
    Copy the full SHA
    de73484 View commit details
    Browse the repository at this point in the history
  2. Add back deleted version attribute (rapidsai#3411)

    The `__version__` attribute appears to have been accidentally deleted in rapidsai#2971.
    
    Authors:
      - Vyas Ramasubramani (https://github.com/vyasr)
    
    Approvers:
      - Alex Barghi (https://github.com/alexbarghi-nv)
    
    URL: rapidsai#3411
    vyasr authored Apr 3, 2023
    Configuration menu
    Copy the full SHA
    e36201e View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    386eeb7 View commit details
    Browse the repository at this point in the history
  4. Use Correct Searchsorted Function and Drop cupy from CuGraphStore in …

    …cugraph-pyg (rapidsai#3382)
    
    Resolves an issue where the wrong version of `searchsorted` caused a device to host copy.  Also removes the backend option from `CuGraphStore` entirely to prevent similar bugs from happening in the future and better align cugraph-pyg with the pyg/pytorch ecosystem.
    
    Merge after rapidsai#3289 
    Closes rapidsai#2995
    
    Authors:
      - Alex Barghi (https://github.com/alexbarghi-nv)
    
    Approvers:
      - Rick Ratzel (https://github.com/rlratzel)
      - Vibhu Jawa (https://github.com/VibhuJawa)
    
    URL: rapidsai#3382
    alexbarghi-nv authored Apr 3, 2023
    Configuration menu
    Copy the full SHA
    7b28710 View commit details
    Browse the repository at this point in the history
  5. Reindex Start Vertices and Batch Ids Prior to Sampling Call (rapidsai…

    …#3393)
    
    This PR fixes a bug where output sample batch ids do not match those expected when using the bulk sampler, causing subgraphs that are larger than expected and incorrect.  Without reindexing, the wrong batch ids are assigned to the start vertices.  Reindexing ensures that the same order is preserved for batch ids and start vertices.
    
    This PR also changes the empty dataframe passed to dask in `uniform_neighbor_sample` to match the correct ordering of batch_id and hop_id.  This ensures that the columns are named correctly and are not inadvertently renamed due to them being created in a different order.
    
    This PR is non-breaking because it restores the original behavior of bulk sampling and reverses a bug that was inadvertently introduced with the dask updates.
    
    Resolves rapidsai#3390
    
    Authors:
      - Alex Barghi (https://github.com/alexbarghi-nv)
    
    Approvers:
      - Rick Ratzel (https://github.com/rlratzel)
      - Vibhu Jawa (https://github.com/VibhuJawa)
      - Joseph Nke (https://github.com/jnke2016)
    
    URL: rapidsai#3393
    alexbarghi-nv authored Apr 3, 2023
    Configuration menu
    Copy the full SHA
    1281bb8 View commit details
    Browse the repository at this point in the history
  6. Add MG implementation of induced subgraph (rapidsai#3391)

    This PR adds an MG implementation of induced subgraph by leveraging the CAPI
    
    closes rapidsai#2535 
    closes rapidsai#2536
    
    Authors:
      - Joseph Nke (https://github.com/jnke2016)
      - Alex Barghi (https://github.com/alexbarghi-nv)
    
    Approvers:
      - Chuck Hastings (https://github.com/ChuckHastings)
      - Rick Ratzel (https://github.com/rlratzel)
    
    URL: rapidsai#3391
    jnke2016 authored Apr 3, 2023
    Configuration menu
    Copy the full SHA
    ee98cea View commit details
    Browse the repository at this point in the history
  7. SG tested Leiden (rapidsai#2980)

    ### Leiden algorithm 
    This PR adds Leiden implementation using cugraph primitives.  It also add an implementation of `maximal independent set ` using cugraph primitives that is used by the Leiden implementation. 
    
    - It reuses code from Louvain implementation to find locally optimal moves for vertices. 
    - Locally optimal movement creates  clusters of `Louvain communities` 
    - In refinement phase, which is the main feature of Leiden algorithm, vertices are moved only within a `Louvain community` to subdivide a `Louvain community` in multiple `sub communities` or `Leiden communities`
    - In graph contraction phase, vertices belonging to a `Leiden community` are merged to become a node in aggregated graph.
    
    This code has been tested to work on SG. We aim to test MG version in next release.
    Future improvement:  
    
    - Move vertices randomly instead of greedily refinement phase
    - Test for MNMG
    
    Authors:
      - Naim (https://github.com/naimnv)
      - Chuck Hastings (https://github.com/ChuckHastings)
    
    Approvers:
      - Chuck Hastings (https://github.com/ChuckHastings)
      - Seunghwa Kang (https://github.com/seunghwak)
    
    URL: rapidsai#2980
    naimnv authored Apr 3, 2023
    Configuration menu
    Copy the full SHA
    d0cc407 View commit details
    Browse the repository at this point in the history

Commits on Apr 4, 2023

  1. New PyG End-to-End Examples (rapidsai#3326)

    - Adds new examples for cugraph-pyg.
    - Removes outdated examples.
    - Moves MG scripts to top-level directory.
    - Makes the input to `_get_vertex_groups_from_sample` a tensor instead of Series
    - Adds `is_sorted` arg to `_get_vertex_groups_from_sample` to skip sorting if tensor already sorted
    - Some fixes to `CuGraphStore` for running multi-GPU workflows
    
    Merge after rapidsai#3288  - merged
    Merge after rapidsai#3289  - merged
    Merge after rapidsai#3382  - merged
    
    Closes rapidsai#3316 
    Closes rapidsai#3226 
    Closes rapidsai#3072
    
    Authors:
      - Alex Barghi (https://github.com/alexbarghi-nv)
    
    Approvers:
      - Vibhu Jawa (https://github.com/VibhuJawa)
      - Rick Ratzel (https://github.com/rlratzel)
    
    URL: rapidsai#3326
    alexbarghi-nv authored Apr 4, 2023
    Configuration menu
    Copy the full SHA
    c61ef05 View commit details
    Browse the repository at this point in the history

Commits on Apr 5, 2023

  1. Adapt to rapidsai/rmm#1221 which moves allocator callbacks (rapidsai#…

    …3300)
    
    The allocator callbacks now live in their own submodules (so that RMM does not, for example, import pytorch unless required) and so must be explicitly imported.
    
    Authors:
      - Lawrence Mitchell (https://github.com/wence-)
    
    Approvers:
      - Alex Barghi (https://github.com/alexbarghi-nv)
      - Rick Ratzel (https://github.com/rlratzel)
    
    URL: rapidsai#3300
    wence- authored Apr 5, 2023
    Configuration menu
    Copy the full SHA
    0225e92 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2381977 View commit details
    Browse the repository at this point in the history
  3. Critical: Force cudf.concat when passing in a cudf Series to MG Unifo…

    …rm Neighbor Sample (rapidsai#3416)
    
    Currently, cudf does not merge series properly when they already share an index.  I'm not sure if this is a bug in cudf, or intentional behavior.  This issue does not occur with dask_cudf.  The resolution is to use `cudf.concat` when passing a `cudf.Series` for start vertices and batch ids, and `df.to_frame().merge` when passing in a `dask_cudf.Series` for start vertices and batch ids.
    
    This PR also adds an additional test which tests both cudf and dask_cudf inputs to catch these sort of problems in the future.
    
    Authors:
      - Alex Barghi (https://github.com/alexbarghi-nv)
    
    Approvers:
      - Rick Ratzel (https://github.com/rlratzel)
      - Vibhu Jawa (https://github.com/VibhuJawa)
      - Joseph Nke (https://github.com/jnke2016)
    
    URL: rapidsai#3416
    alexbarghi-nv authored Apr 5, 2023
    Configuration menu
    Copy the full SHA
    e76406d View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    e1c44b7 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    00d0b35 View commit details
    Browse the repository at this point in the history
  6. Support Minor Releases of PyG (rapidsai#3422)

    Required to support the upcoming minor release of PyG, which will allow full compatibility with pylibcugraphops.
    
    Authors:
      - Alex Barghi (https://github.com/alexbarghi-nv)
    
    Approvers:
      - Vibhu Jawa (https://github.com/VibhuJawa)
      - Ray Douglass (https://github.com/raydouglass)
    
    URL: rapidsai#3422
    alexbarghi-nv authored Apr 5, 2023
    Configuration menu
    Copy the full SHA
    7e396e3 View commit details
    Browse the repository at this point in the history
  7. Doc Updates (rapidsai#3418)

    Trying to fix doc build issues
    
    Authors:
      - Brad Rees (https://github.com/BradReesWork)
    
    Approvers:
      - Don Acosta (https://github.com/acostadon)
      - Rick Ratzel (https://github.com/rlratzel)
    
    URL: rapidsai#3418
    BradReesWork authored Apr 5, 2023
    Configuration menu
    Copy the full SHA
    81f1937 View commit details
    Browse the repository at this point in the history
  8. update conv layers in cugraph-dgl for pylibcugraphops 23.04 (rapidsai…

    …#3360)
    
    Update cugraph-ops models to use pylibcugraphops 23.04. This PR also supersedes rapidsai#3264 
    
    CC: @MatthiasKohl
    
    Authors:
      - Tingyu Wang (https://github.com/tingyu66)
      - Alex Barghi (https://github.com/alexbarghi-nv)
    
    Approvers:
      - AJ Schmidt (https://github.com/ajschmidt8)
      - Vibhu Jawa (https://github.com/VibhuJawa)
      - Rick Ratzel (https://github.com/rlratzel)
    
    URL: rapidsai#3360
    tingyu66 authored Apr 5, 2023
    Configuration menu
    Copy the full SHA
    a4c97b5 View commit details
    Browse the repository at this point in the history

Commits on Apr 6, 2023

  1. Pin dask and distributed for release (rapidsai#3427)

    This PR pins `dask` and `distributed` to `2023.3.2` and `2023.3.2.1` respectively for `23.04` release.
    
    xref: rapidsai/cudf#13070
    
    Authors:
      - GALI PREM SAGAR (https://github.com/galipremsagar)
    
    Approvers:
      - Ray Douglass (https://github.com/raydouglass)
      - Peter Andreas Entschev (https://github.com/pentschev)
      - Rick Ratzel (https://github.com/rlratzel)
      - Joseph (https://github.com/jolorunyomi)
    
    URL: rapidsai#3427
    galipremsagar authored Apr 6, 2023
    Configuration menu
    Copy the full SHA
    72cffc1 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e262b21 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    0ad2722 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    0061424 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    c72a875 View commit details
    Browse the repository at this point in the history

Commits on Apr 7, 2023

  1. update rmm version

    alexbarghi-nv committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    353e894 View commit details
    Browse the repository at this point in the history
  2. bump ucx-py to 0.32.*

    alexbarghi-nv committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    16b3f0c View commit details
    Browse the repository at this point in the history

Commits on Apr 10, 2023

  1. Configuration menu
    Copy the full SHA
    a625c46 View commit details
    Browse the repository at this point in the history