lc0-cudnn : add support for fp16 network eval

[ankan-ban]

slightly more than 2x speedup (for large batch sizes) on supported hardware, without much loss of precision.

• currently fp16 is only supported on Volta based GPUs (TitanV, tesla V100, etc)

[jjoshua2]

How can there be more than 2x improvement. Shouldn't that be max. Is it loss of precision causing higher avg batchsize?

[mooskagh]

Would it be possible instead of converting all float* to void* to have two template variants, for float and __fp16, similarly to what I did here: https://github.com/LeelaChessZero/lc0/blob/release/src/neural/network_tf.cc#L299 for tensorflow.
and then use ValueType* instead? (maybe some better name for ValueType though.. Float?)

So, it would be

template <typename ValueType>
class CudnnNetwork ...

and all helper classes/function would also become templates, and registered like:

REGISTER_NETWORK("cudnn", CudnnNetwork<float>, 110)
REGISTER_NETWORK("cudnn-fp16", CudnnNetwork<__fp16>, 105)

The drawback though is that best configuration won't be automatically chosen.
But on other hand, as cudnn-fp32 has better precision, everyone should be able to select it.

So, with this approach, if there is a function which needs completely different implementation, then template specialization is useful. If however there is difference in small details, this is possible:

if (std::is_same<float, ValueType>::value) {  // In C++17 (or 14?)  ::value is not necessary
   // Do something float-specific.
}

Also this file becomes bit, maybe at that point it deserves a separate src/neural/cudnn/ directory and it is worth splitting into several files?

[ankan-ban]

Volta Tensor cores (for FP16 math) have much higher throughput than fp32.
E.g: see table in https://www.anandtech.com/show/12170/nvidia-titan-v-preview-titanomachy

Just slightly more than 2X (instead of 8X is because):

1. we aren't getting full utilization yet (need even bigger batch size).
2. cudnn can't use winograd convolution algorithm with tensor cores. (edit: actually I think it can, but it needs really big channel counts/batch sizes.. maybe I will try it out with 256 filter networks)

[jnewlin12345]

I'm looking at the anandtech link and see:

Single Precision: 13.8 TFLOPS
Half Precision: 27.6 TFLOPS

Where is the 8x coming from?

[ankan-ban]

Tensor Performance(Deep Learning) | 110 TFLOPS
Tensor cores are basically special units for small matrix multiplies.
see this blog for more details: https://devblogs.nvidia.com/programming-tensor-cores-cuda-9/

[ankan-ban]

@mooskagh , Sorry for long delay. I was playing with int8 but I don't have it working yet. I have made the changes to have two template variants for half and float. Will add int8 in seperate PR when it's ready.

[Error323]

Hi @ankan-ban are you using TensorRT? Could you push what you have? I'd like to dive in :) (in a separate branch that is)

[ankan-ban]

Haven't tried tensorRT yet. It's linux only and I am on Windows right now :(. I think it's time for me to setup a linux partition and compare performance of TensorRT with cudnn.

I was trying out int8 with cudnn where the adjustment/calibration of weights has to be done manually. (I believe TensorRT does it for you?)

[Error323]

That's correct, tensorrt allows you to calibrate automatically given a dataset. I did some preliminary testing here glinscott/leela-chess#52 (comment)

[mooskagh]

const

[mooskagh]

Sorry, for nitpicking, but consider changing all comments to proper English sentences. (starting with capital letter, ending with a period).

Also consider running clang-format -I -style=google network_cudnn.cu but I can do that myself if you don't have clang installed.

[ankan-ban]

Done. I don't have clang but I got just the clang-format binary for windows.

[mooskagh]

also const

[mooskagh]

not needed?

Looks good, but remove clang-format.exe from the PR. :)