Merge pull request LeelaChessZero#12 from LeelaChessZero/master

merge master

.gitignore

@@ -2,7 +2,9 @@
 *.swp
 .clang_complete
 .DS_Store
+.clangd/
 build/
+__pycache__/
 compile_commands.json
 CUDA_NN/
 lc0.xcodeproj/

CONTRIBUTING.md

@@ -2,7 +2,7 @@
 
 These are the guidelines and standards followed by this codebase.
 
-The language is C++, specifically C++14. As such, manual `new` and `delete` memory mangement is strongly discouraged; use the standard library tools for managing memory (such as `unique_ptr`, `shared_ptr` etc.). When compiler support is more widespread, the project may upgrade to the C++17 standard in the future.
+The language is C++, specifically C++17. As such, manual `new` and `delete` memory mangement is strongly discouraged; use the standard library tools for managing memory (such as `unique_ptr`, `shared_ptr` etc.).
 
 This codebase uses semantic versioning. A release is the final commit for that version number, and all subsequent commits are development for the next version. `master` is the default branch, and the active development branch (as such, all Pull Requests go here); it always targets a minor (or major) version which succeeds the current relase. `release` is always equivalent to the latest tag.
 
@@ -56,21 +56,11 @@ The internal code dependency structure looks like this:
 
 ### Git history
 
-Pull Requests are squahsed when merged. This means all commits in the branch will be squashed into one commit applied onto master, so branches and their PRs should stick to *one* topic only. If you think changes deserve separate commits, make separate PRs for each commit.
+Pull Requests are squashed when merged. This means all commits in the branch will be squashed into one commit applied onto master, so branches and their PRs should stick to *one* topic only. If you think changes deserve separate commits, make separate PRs for each commit.
 
 This also means it's not possible to reuse one branch for multiple PRs; new PRs must either use entirely new branches, or else you could use `git reset --hard` on the current branch.
 
 
-### Command line/UCI options
-
-The options code handles both UCI options and command line options at the same time; in fact they are one and the same. Each option has a "flag name" and a "description". The flag name is used as the command line `--flag-name`. The description serves a dual purpose: it is the text printed by `./lc0 --help`, but it also serves as the *name* of the UCI option as well. Therefore the description should:
-
- * Not end with a period (per the UCI specification)
- * Be clear and succinct, to serve as both a help message and standalone UCI option name
- * Be short (to fit as a UCI option in chess GUIs)
- * Be different from the flag name (since it's a help message)
-
-
 ### Allowed features
 
 Lc0 is still in early stages of development, and has not yet reached the point where we are ready to add small tweaks to add few points of a rating. Large code changes still happen, and having lots of small optimizations adds overhead to larger changes, slowing development.

FLAGS.md

@@ -52,7 +52,7 @@ List of command line flags:
 | --fpu-reduction=NUM | First Play Urgency reduction | Default: `0.2` |
 | --cache-history-length=NUM | The length of history to include in the cache | Default: `7` |
 | --extra-virtual-loss=NUM | Extra virtual loss | Default: `0` |
-| -l,<br>--debuglog=FILENAME | Do debug logging into a file | Default is off (empty string) |
+| -l,<br>--logfile=FILENAME | Do debug logging into a file | Default is off (empty string) |
 
 
 ## Configuration Files
@@ -65,7 +65,7 @@ List of command line flags:
 # The -- is optional.  The following flags will work as well:
 weights=10445.txt.gz
 syzygy-paths=syzygy
-debuglog=lc0.log
+logfile=lc0.log
 ```
 You can tell `lc0` to ignore the default configuration file by passing `--config=` on the command line.  Command line arguments will override any arguments that also exist in the configuration file.
 

README.md

@@ -9,17 +9,25 @@ Lc0 is a UCI-compliant chess engine designed to play chess via neural network, s
 
 Lc0 can be acquired either via a git clone or an archive download from GitHub. Be aware that there is a required submodule which isn't included in source archives.
 
-For essentially all purposes, including selfplay game generation and match play, we highly recommend using the latest `release/version` branch (for example `release/0.24`), which is equivalent to using the latest version tag.
+For essentially all purposes, including selfplay game generation and match play, we highly recommend using the latest `release/version` branch (for example `release/0.25`), which is equivalent to using the latest version tag.
 
 Versioning follows the Semantic Versioning guidelines, with major, minor and patch sections. The training server enforces game quality using the versions output by the client and engine.
 
 
 Download using git:
 
 ```
-git clone -b release/0.24 --recurse-submodules https://github.com/LeelaChessZero/lc0.git
+git clone -b release/0.25 --recurse-submodules https://github.com/LeelaChessZero/lc0.git
 ```
 
+If you have cloned already an old version, fetch, view and checkout a new branch:
+```
+git fetch --all
+git branch --all
+git checkout -t remotes/origin/release/0.25
+```
+
+
 If you prefer to download an archive, you need to also download and place the submodule:
  * Download the [.zip](https://api.github.com/repos/LeelaChessZero/lc0/zipball/release/0.24) file ([.tar.gz](https://api.github.com/repos/LeelaChessZero/lc0/tarball/release/0.24) archive is also available)
  * Extract
@@ -40,6 +48,8 @@ Backend support includes (in theory) any CBLAS-compatible library for CPU usage,
 
 Finally, lc0 requires a compiler supporting C++17. Minimal versions seem to be g++ v8.0, clang v5.0 (with C++17 stdlib) or Visual Studio 2017.
 
+*Note* that cuda checks the compiler version and stops even with newer compilers, and to work around this we have added the `nvcc_ccbin` build option. This is more of an issue with new Linux versions, where we recommend to install `g++-7` and add `-Dnvcc_ccbin=g++-7` to the `build.sh` command.
+
 Given those basics, the OS and backend specific instructions are below.
 
 ### Linux
@@ -160,6 +170,8 @@ Now download the lc0 source, if you haven't already done so, following the instr
 
 ### Raspberry Pi
 
+You'll need to be running the latest raspbian "buster".
+
 1. Install OpenBLAS
 
 ```
@@ -177,17 +189,10 @@ pip3 install meson
 pip3 install ninja
 ```
 
-3. Install clang
+3. Install compiler and standard libraries
 
 ```
-wget http://releases.llvm.org/6.0.0/clang+llvm-6.0.0-armv7a-linux-gnueabihf.tar.xz
-tar -xf clang+llvm-6.0.0-armv7a-linux-gnueabihf.tar.xz
-rm clang+llvm-6.0.0-armv7a-linux-gnueabihf.tar.xz
-mv clang+llvm-6.0.0-armv7a-linux-gnueabihf clang_6.0.0
-sudo mv clang_6.0.0 /usr/local
-echo 'export PATH=/usr/local/clang_6.0.0/bin:~/.local/bin:$PATH' >> .bashrc
-echo 'export LD_LIBRARY_PATH=/usr/local/clang_6.0.0/lib:$LD_LIBRARY_PATH' >> .bashrc
-source .bashrc
+sudo apt install clang-6.0 libstdc++-8-dev
 ```
 
 4. Clone lc0 and compile
@@ -196,7 +201,7 @@ source .bashrc
 git clone https://github.com/LeelaChessZero/lc0.git
 cd lc0
 git submodule update --init --recursive
-CC=clang CXX=clang++ ./build.sh -Ddefault_library=static
+CC=clang-6.0 CXX=clang++-6.0 ./build.sh -Ddefault_library=static
 ```
 
 5. The resulting binary will be in build/release

appveyor.yml

@@ -11,6 +11,13 @@ environment:
   - NAME: cpu-dnnl
   - NAME: cpu-openblas
   - NAME: android
+for:
+-
+  matrix:
+    only:
+    - NAME: gpu-opencl
+    - NAME: cpu-dnnl
+  skip_non_tags: true
 clone_folder: c:\projects\lc0
 install:
 - cmd: set CUDA=false
@@ -31,9 +38,11 @@ install:
 - cmd: IF %NAME%==cpu-openblas IF NOT EXIST C:\cache\OpenBLAS appveyor DownloadFile https://sjeng.org/ftp/OpenBLAS-0.3.3-win-oldthread.zip
 - cmd: IF %NAME%==cpu-openblas IF NOT EXIST C:\cache\OpenBLAS 7z x OpenBLAS-0.3.3-win-oldthread.zip -oC:\cache\OpenBLAS
 - cmd: IF %OPENCL%==true nuget install opencl-nug -Version 0.777.77 -OutputDirectory C:\cache
-- cmd: IF %BLAS%==true IF NOT EXIST C:\cache\ispc-v1.9.2-windows appveyor DownloadFile https://sourceforge.net/projects/ispcmirror/files/v1.9.2/ispc-v1.9.2-windows.zip
-- cmd: IF %BLAS%==true IF NOT EXIST C:\cache\ispc-v1.9.2-windows 7z x ispc-v1.9.2-windows.zip -oC:\cache
-- cmd: IF %BLAS%==true set PATH=C:\cache\ispc-v1.9.2-windows;%PATH%
+- cmd: set ISPC=%BLAS%
+- cmd: IF %NAME%==android set ISPC=true
+- cmd: IF %ISPC%==true IF NOT EXIST C:\cache\ispc-v1.13.0-windows appveyor DownloadFile https://github.com/ispc/ispc/releases/download/v1.13.0/ispc-v1.13.0-windows.zip
+- cmd: IF %ISPC%==true IF NOT EXIST C:\cache\ispc-v1.13.0-windows 7z x ispc-v1.13.0-windows.zip -oC:\cache\ispc-v1.13.0-windows
+- cmd: IF %ISPC%==true set PATH=C:\cache\ispc-v1.13.0-windows\bin;%PATH%
 - cmd: set "CUDA_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0"
 - cmd: IF %CUDA%==true IF NOT EXIST "%CUDA_PATH%" set CUDA_INSTALL=1
 - cmd: IF DEFINED CUDA_INSTALL appveyor DownloadFile https://developer.nvidia.com/compute/cuda/10.0/Prod/network_installers/cuda_10.0.130_win10_network
@@ -64,7 +73,7 @@ install:
 - cmd: IF %GTEST%==true IF NOT EXIST KQvKQ.rtbz curl --remote-name-all https://tablebase.lichess.ovh/tables/standard/3-4-5/K{P,N,R,B,Q}vK{P,N,R,B,Q}.rtb{w,z}
 - cmd: cd C:\projects\lc0
 cache:
-  - C:\cache -> appveyor.yml
+  - C:\cache
   - 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0'
   - C:\projects\lc0\subprojects\packagecache
   - C:\ndk\android-ndk-r19c\toolchains\llvm\prebuilt\windows-x86_64
@@ -77,7 +86,7 @@ before_build:
 - cmd: IF %APPVEYOR_REPO_TAG%==true IF %ANDROID%==true SET EMBED=true
 - cmd: IF %ANDROID%==false meson build --backend vs2017 --buildtype release -Dgtest=%GTEST% -Dopencl=%OPENCL% -Dblas=%BUILD_BLAS% -Ddnnl=true -Ddx=%DX% -Dcudnn=%CUDA% -Dispc_native_only=false -Dpopcnt=false -Dcudnn_include="%CUDA_PATH%\include","%PKG_FOLDER%\cuda\include" -Dcudnn_libdirs="%CUDA_PATH%\lib\x64","%PKG_FOLDER%\cuda\lib\x64" -Dopenblas_include="%PKG_FOLDER%\OpenBLAS\dist64\include" -Dopenblas_libdirs="%PKG_FOLDER%\OpenBLAS\dist64\lib" -Ddnnl_dir="%PKG_FOLDER%\dnnl_win_1.1.1_cpu_vcomp" -Dopencl_include="%PKG_FOLDER%\opencl-nug.0.777.77\build\native\include" -Dopencl_libdirs="%PKG_FOLDER%\opencl-nug.0.777.77\build\native\lib\x64" -Ddefault_library=static
 - cmd: IF %ANDROID%==true meson arm64-v8a --buildtype release -Dgtest=false -Dopenblas_include="%PKG_FOLDER%\OpenBLAS\android-aarch64\include" -Dopenblas_libdirs="%PKG_FOLDER%\OpenBLAS\android-aarch64\lib" -Dembed=%EMBED% -Ddefault_library=static --cross-file crossfile-aarch64
-- cmd: IF %ANDROID%==true meson armeabi-v7a --buildtype release -Dgtest=false -Dopenblas_include="%PKG_FOLDER%\OpenBLAS\android-armv7a\include" -Dopenblas_libdirs="%PKG_FOLDER%\OpenBLAS\android-armv7a\lib" -Dembed=%EMBED% -Ddefault_library=static --cross-file crossfile-armv7a
+- cmd: IF %ANDROID%==true meson armeabi-v7a --buildtype release -Dgtest=false -Dopenblas_include="%PKG_FOLDER%\OpenBLAS\android-armv7a\include" -Dopenblas_libdirs="%PKG_FOLDER%\OpenBLAS\android-armv7a\lib" -Dembed=%EMBED% -Ddefault_library=static --cross-file crossfile-armv7a -Dispc=false
 build_script:
 - cmd: IF %ANDROID%==false call scripts\appveyor_win_build.cmd
 - cmd: IF %ANDROID%==true call scripts\appveyor_android_build.cmd

build.cmd

@@ -25,12 +25,12 @@ echo Deleting build directory:
 rd /s build
 
 if exist "C:\Program Files (x86)\Microsoft Visual Studio\2019" (
-  call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvarsall.bat" amd64
-  set MSBuild=msbuild
+  where /q cl
+  if errorlevel 1 call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvarsall.bat" amd64
   set backend=vs2019
 ) else (
-  call "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvarsall.bat" amd64
-  set MSBuild="C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\MSBuild.exe"
+  where /q cl
+  if errorlevel 1 call "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvarsall.bat" amd64
   set backend=vs2017
 )
 
@@ -58,5 +58,5 @@ pause
 
 cd build
 
-%MSBuild% /m /p:Configuration=Release /p:Platform=x64 /p:WholeProgramOptimization=true ^
-/p:PreferredToolArchitecture=x64 lc0.sln /filelogger
+msbuild /m /p:Configuration=Release /p:Platform=x64 /p:WholeProgramOptimization=true ^
+/p:PreferredToolArchitecture=x64 lc0.sln /filelogger

build.sh

@@ -16,6 +16,11 @@ esac
 
 BUILDDIR=build/${BUILDTYPE}
 
+if ! hash meson 2>/dev/null && [ -x ${HOME}/.local/bin/meson ]
+then
+  export PATH=${PATH}:${HOME}/.local/bin
+fi
+
 if [ -f ${BUILDDIR}/build.ninja ]
 then
   meson configure ${BUILDDIR} -Dbuildtype=${BUILDTYPE} -Dprefix=${INSTALL_PREFIX:-/usr/local} "$@"

changelog.txt

@@ -1,6 +1,44 @@
-v0.25.0-rc1 (2020-04-09)
+v0.25.0 (2020-04-28)
 ~~~~~~~
 
+* Relax strictness for complete standard fens in uci and opening books. Fen
+  must still be standard, but default values will be substituted for sections
+  that are missing.
+* Restore some backwards compatibility in cudnn backends that was lost with
+  the addition of the new convolution implementation. It is also on by default
+  for more scenarios, although still off for fp16 on RTX gpus.
+* Small logic fix for nps smoothing in the new optional experimental time 
+  manager.
+
+v0.25.0-rc2 (2020-04-23)
+~~~~~~~~~~~
+
+* Increased upper limit for maximum collision events.
+* Allow negative values for some of the extended moves left head parameters.
+* Fix a critical bug in training data generation for input type 3.
+* Fix for switching between positions in uci mode that only differ by 50 move
+  rule in initial fen.
+* Some refinements of certainty propagation.
+* Better support for c++17 implementations that are missing charconv.
+* Option to more accurately apply time management for uci hosts using 
+  cuteseal or similar timing techniques.
+* Fix for selfplay mode to allow exactly book length total games.
+* Fix for selfplay opening books with castling moves starting from chess960 fens.
+* Add build option to override nvcc compiler.
+* Improved validity checking for some uci input parameters.
+* Updated the Q to CP conversion formula to better fit recent T60 net outputs to
+  expectations.
+* Add a new experimental time manager.
+* Bug fix for the Q+U in verbose move stats. It is now called S: and contains
+  the total score, including any moves left based effect if applicable.
+* New temperature decay option to allow to delay the start of decay.
+* All temperature options have been hidden by default.
+* New optional cuda backend convolution implementation. Off by default for 
+  cudnn-fp16 until an issue with cublas performance on some gpus is resolved.
+
+v0.25.0-rc1 (2020-04-09)
+~~~~~~~~~~~
+
 * Now requires a c++17 supporting compilation environment to build.
 * Support for Moves Left Head based networks. Includes options to adjust search
   to favour shorter/longer wins/losses based on the moves left head output.