Verify all backends support biases.

[Tilps]

BN beta enabling in training (LeelaChessZero/lczero-training#4) will produce weight files that have non-zero biases. (As an alternative to requiring a weights file versioning. This is the same strategy that leela-zero go used.)

I've taken a look at the code and can see cuda backend merges biases into the BN means. Open Cl and Blas backends also appear to do the same thing. Cuda backend also sometimes folds the BN means back in to biases after that, but the transform appears sound at first glance.

I am however having a bit of difficulty finding where biases are handled in the tf backend.

[Tilps]

http://webphactory.net/lczero/bootstrap-192x15-50000.txt.gz is an example net which has non-zero biases. It could be used for cross validation of backends across a variety of positions, but I don't have lc0 built with support for all backends.

[mooskagh]

Indeed the bias values are not used in tf backend.
I copied model from the training python code https://github.com/LeelaChessZero/lczero-training/blob/master/tf/tfprocess.py#L328 and biases are not trained there either.

Is it something to fix in training?

[Tilps]

BN beta PR enables 'betas' during training, and further removes the biases from the training process. It then 'converts' the betas back in to biases by mathematically pushing them back through the batch normalization layer in order to export them to the weights file format without requiring a new version of the weights format which has a row for betas for each batch normalization layer.
So no training doesn't need to change to support biases - just tf backend (since all other backends already support it)

[killerducky]

Status: Crem's commit should have fixed this but no one had time to verify it. There isn't really a use case for TF backend except as yet another check things are being done correctly, so this is low priority.

[mooskagh]

Should we close this?