A todo list for the sparse feature (GPU)

[eric-haibin-lin]

#8168

Operator (forward pass only unless specified):

• elemwise_mul(csr, dense) = csr: both forward and backward
• sum(csr)
• mean(csr)
• concat(csr, csr)
• stack(csr, csr)
• unary operators for csr (arcsin, arcsinh, arctan, arctanh, ceil, expm1, floor, log1p, power, rint, sign, sin, sinh, sqrt, tan, tanh, trunc) - most of them can just reuse the template code implemented for row_sparse

Iterator:

• NDArrayIter(csr) with last_batch_handle = pad / roll_over

[eric-haibin-lin]

cc @ZiyueHuang

[leezu]

• Standalone pruning operator as in https://github.com/apache/incubator-mxnet/blob/274fae82d66b1cfed48f78e69538631c57c5b6b6/src/operator/tensor/cast_storage-inl.cuh#L116-L153

[eric-haibin-lin]

LogisticLoss with csr labels (requires reshape and reshape_like)

[ghost]

Has there been any progress on this?

[eric-haibin-lin]

Not that I'm aware of. @Codewithsk are you looking for any of these?

[ghost]

I'm looking to contribute and this looks interesting. I can maybe submit a proposal on the dev list?

[ghost]

@eric-haibin-lin Any pointers on where I can start?
I see this: A Guide To Implementing Sparse Operators in MXNet Backend

[jermainewang]

@eric-haibin-lin Do we have gradient support of sparse matrix for the dot operator right now?

[eric-haibin-lin]

@jermainewang No. For operator: dot(csr, dense) = dense, in the backward pass gradient calculation for left-hand side is not yet supported. Do you know if anyone is interested in contributing?

[jermainewang]

It's actually quite important for implementing Graph Attention Network. Pytorch 1.0 has enabled this support. There is also a workaround here (https://github.com/Diego999/pyGAT). It would be great if this could be prioritized.