A Todo List for the Sparse Feature (CPU)

[eric-haibin-lin]

Operator:

• sum(csr) @anirudh2290
• mean(csr) @anirudh2290
• sparse embedding(row_sparse gradient) @eric-haibin-lin
• concat(csr, csr)
• split_axis(csr) @ZiyueHuang
• eye(csr) @ZiyueHuang
• support custom operators with scipy @anirudh2290
• unary operators for csr (arcsin, arcsinh, arctan, arctanh, ceil, expm1, floor, log1p, power, rint, sign ,sin, sinh, sqrt, tan, tanh, trunc) - most of them can just reuse the template code implemented for row_sparse
• performance optimization for dot(csr.T, dense) using parallel scan
• _div_scalar(csr) @ZiyueHuang
• NCE loss @eric-haibin-lin
• dot(csr.T, csr) = row_sparse
• dot(dense, csr) = csr @anirudh2290
• elemwise_add(csr, dense)
• broadcast_add(csr) & broadcast_sub(csr)
• contrib.sparse_broadcast_mul(csr) / contrib.sparse_broadcast_div(csr)
• _set_value(row_sparse) & setitem(int) for row_sparse
• norm(row_sparse)

NDArray:

• convenient constructor (csr) with (data, row_idx, col_idx) @anirudh2290
• check_format(csr & row_sparse): checks the validity of a sparse ndarray @ZiyueHuang
• as_scipy(csr) @anirudh2290
• getitem & setitem for csr[int]
• getitem & setitem for csr[py_slice]

Iterator:

• NDArrayIter(csr) with shuffle = True @ZiyueHuang
• NDArrayIter(csr) with last_batch_handle = pad / roll_over
• NDArrayIter(csr) with scipy inputs
• LibSVMIter with shuffle and round_batch = False
• LibFMIter

Module API:

• expose group2ctx to Module API to enable part of the symbols trained on CPU while others are trained on GPU @ZiyueHuang

Example:

• factorization machine w/ criteo dataset @eric-haibin-lin
• recommendation system @eric-haibin-lin
• language modeling with sparse embedding
• wide & deep learning @ZiyueHuang
• field-aware factorization machine

GPU Support:

• local kvstore push/row_sparse_pull with device comm for sparse weight of size ~ 1GB @ZiyueHuang
• an example which uses group2ctx for one part of the model on CPU, the other on GPU
• sparse embedding (row_sparse) @ZiyueHuang

Gluon Support

• multi-device/machine support
• imperative/symbolic/hybrid support

[mdespriee]

Nice to have a visibility on your roadmap on this topic. I'm very interested some items (eg dot(csr.T, csr) = row_sparse) , as well as scala support of sparse arrays -> this might appear as a paragraph in this todo-list.