Investigate moving value/policy head to GPU

[glinscott]

There was a prototype here: https://github.com/ihavnoid/leela-zero/tree/dualhead_conv

It was not a win for Leela Zero, but with us increasing from 1/2 to 32 or 64 channels, it could make a big difference now.

[Error323]

I'm looking into this. It's not as trivial as I had hoped.

[gcp]

leela-zero/leela-zero@f4a36c1

This doesn't move the entire head (i.e. not the FC/innerproduct layers), just the final 1x1 convolution. It's useful because it saves a lot of data transfer (64-128 fold!) over the PCIe bus, even if the computational gain is little.

[Error323]

edit - never mind I found the issue. I do wonder though, what is a good way to debug OpenCL code? I'm a CUDA guy.

[glinscott]

Resolved by #64. Thanks @Error323!

[Ttl]

#64 only moved the convolutions to GPU. Innerproducts are still on CPU and according to the profiler take more time than convolutions on GPU.

I made earlier a GPU innerproduct for leelaz, but it was slower than doing it on CPU. It should be faster for lczero since innerproducts are much bigger. I'll see if I can port it over.

[Error323]

Hey @glinscott sorry it's not fully resolved yet, we should still move the innerproducts to the GPU, atleast the policy head 32x8x8 -> 1924 and the 32x8x8 -> 128 from the valuehead.

[glinscott]

Ah, yes, good call @Error323. After some quick work by @Ttl the innerproducts are now moved to GPU as well! Thanks @Ttl!