WIP: [CUDA] New CUDA version

[shiyu1994]

Current GPU and CUDA versions of LightGBM only calculate histograms on GPU. This is a new CUDA version for LightGBM which runs the whole training process on GPU.

[StrikerRUS]

@shiyu1994

One option would be to keep the old cuda version as cuda_hist, and the new cuda version as cuda in the device type parameter.

Does new version reuse current CUDA histograms building code?

[shiyu1994]

@StrikerRUS No, there's no dependence on the old CUDA histogram code.

[StrikerRUS]

@shiyu1994 Maybe then we can name new version as cuda_exp (from experimental, or any other better naming) and after some number of releases remove current cuda code and rename cuda_exp into normal cuda?

[shiyu1994]

@StrikerRUS I agree. Let's add the new CUDA version as cuda_exp for some versions, and replace the old CUDA version with it until it stabilizes.

[jameslamb]

Just leaving a "request changes" review to be sure this isn't accidentally merged until we've agreed that enough time has passed since 3.3.0.

Relevant comment: #4633 (comment)

[GeorgePearse]

How can I build this branch?

[shiyu1994]

@GeorgePearse Thanks for using LightGBM. To build this branch, you may clone from https://github.com/shiyu1994/LightGBM/tree/cuda-new-refactor-2 and follow the build instruction of LightGBM CUDA version here. https://lightgbm.readthedocs.io/en/latest/Installation-Guide.html#build-cuda-version-experimental
Note that this is still a in-progress PR and there may be some bugs remain to be fixed.

[GeorgePearse]

@shiyu1994 Thanks! Just need to try to find the fastest training model possible. Stability not a worry. Is it useful for me to add any bugs I encounter here? Or is there good enough test coverage that I'll just be restating what's already known.

[shiyu1994]

@GeorgePearse Yes, if you find any bug you may post it here. There are some features that CUDA version of this branch does not support, including training with categorical features. Other additional features during training, like monotonic constraints, extremely random trees are not supported.

I'm now working with #4630, which only dispatches the tree learner part to CUDA. That branch is relatively more complete. I'll finish #4630 first, and then merge additional contents in this PR.

Thanks for your support.

[jameslamb]

I'm now working with #4630, which only dispatches the tree learner part to CUDA. That branch is relatively more complete. I'll finish #4630 first, and then merge additional contents in this PR.

@shiyu1994 is it still your plan to merge master into this PR branch? Or should this be closed and new PRs opened for additional CUDA changes.

[shiyu1994]

Or should this be closed and new PRs opened for additional CUDA changes.

@jameslamb I believe we should close this. I'll open additional PRs for the rest CUDA work. Thanks!

[github-actions]

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.