[WIP] Implement the RENTS algorithm for tree exploration

[kiudee]

Description

This is a WIP branch where I will implement the RENTS algorithm as proposed by Tuan et al. 2020 [1]. Currently, it is not working yet. I will remove this note as soon as you can run it.

Policies are continually recomputed using the following weighted softmax:

To ensure enough exploration, the uniform distribution is mixed in:

with the following exploration term:

The initialization of the Q values is crucial. It is possible to use the existing policy head to come up with a semi-decent initialization, but preliminary experiments in a0lite have shown that initializing using the value head is much better. For that we would need a value head which outputs Q values for every move.

Pros and Cons

Pros:

• Faster convergence than PUCT
• Potentially better scaling behaviour
• Potentially easier collection of batches

Cons:

• More computational overhead during backpropagation (may need to come up with a clever updating scheme, or update only rarely)
• Unclear yet, if it is better for chess (the variant TENTS could work better)

Ablations

Here I will record SPRT tests of different design decisions.

To do

Implementation:

• Implement weighted softmax function
• Initialize Q values
• Update Q values and policies during backpropagation
• Select moves based on policies
• Additional parameters to control exploration, softmax temperature etc
• Ensure correct handling of Q values
• Check how it interacts with cache

Test:

• Performance using fixed nodes
• ...

References

[1] Dam, T., D'Eramo, C., Peters, J., and Pajarinen, J., “Convex Regularization in Monte-Carlo Tree Search”, arXiv e-prints, 2020.