Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Attention body support for blas / eigen / mkl #1852

Merged
merged 49 commits into from
Mar 29, 2023
Merged

Attention body support for blas / eigen / mkl #1852

merged 49 commits into from
Mar 29, 2023

Conversation

almaudoh
Copy link
Contributor

Implementation of attentionbody for blas / eigen / mkl backends.

borg323 and others added 30 commits February 20, 2023 17:44
@borg323
update some shared files for attention body
f2ba92a
@borg323
update default net (LeelaChessZero#1804)
f091f5e
@borg323
option to select onnx opset (LeelaChessZero#1803)
ad04529
@trre123 @borg323
Add 'simple' time manager (LeelaChessZero#1764)
2782b94
@zz4032 @borg323
Backendbench assistant with TC-dependent output. (LeelaChessZero#1546)
b218726
@mooskagh @borg323
In smooth TM, use piggybank when it seems like the bestmove can be ov…
402da93 
…ertaken. (LeelaChessZero#1762)

fixes LeelaChessZero#1582
@borg323
only use thread binding with blas backend (LeelaChessZero#1797)
8774cae
@Naphthalin @borg323
Fix apparent "mate blindness" in endgame due to instamove logic (Leel…
a3646ea 
…aChessZero#1742)
@borg323
initial dfrc support (LeelaChessZero#1684)
2feda7c
@almaudoh @borg323
Fix error in mha scaling in encoder layers. (LeelaChessZero#1808)
2efcd4f
@borg323
add probability for early termination of selfplay openings (LeelaChes…
eeeab14 
…sZero#1681)
@borg323
update some shared files for smolgen
bb23800
@borg323
add swish activation
c80a4b7
@ankan-ban @borg323
Persistent L2 cache opt for cuda backend (LeelaChessZero#1815)
45965dd 
* add persistent L2 cache opt

- goal is to fit activations in residual block in L2 cache.
 - around 6.7% improvement in T80 networks.

* add checks for cuda version

- prevent compile errors when compiled with old CUDA toolkit.

* fix typo with cudart_version
@borg323
fix undefined behavior (LeelaChessZero#1817)
a127abd
@borg323
set MADV_RANDOM for syzygy mmap (LeelaChessZero#1818)
f4a918a
@almaudoh @almaudoh-1 @borg323
Fix softmax cuda (LeelaChessZero#1822)
6a77459 
* Fix softmax function in cuda.

* Add fix for negative zero.

* Code style fixes.

Co-authored-by: Alma <[email protected]>
@borg323
set cache_opt default to false (LeelaChessZero#1823)
662648a
@borg323
fix onnx final mlh activation (LeelaChessZero#1825)
a5de019
@borg323
inline fp16 conversions (LeelaChessZero#1824)
c41ba16
@borg323
onnx squeeze cleanup (LeelaChessZero#1826)
9ff390b
@borg323
update submodule
ec887c5
@borg323
update for new net.proto
825d0d0
@borg323
ispc alternatives for attention policy parts
155c54a
@borg323
remove mha transpose
cab4c04
@borg323
ispc softmax
ce14608
@borg323
important typo fix
bd8aafe
@borg323
refactor blas encoder
8d67b1f
@borg323
blas performance fix
a96c4c0
@almaudoh @borg323
Attention body support to blas.
ffd6bba
@almaudoh almaudoh mentioned this pull request Feb 21, 2023
Closed
@borg323 borg323 requested a review from Tilps February 21, 2023 12:56
mooskagh
mooskagh reviewed Mar 28, 2023
View reviewed changes
src/neural/loader.cc Outdated Show resolved Hide resolved
mooskagh
mooskagh approved these changes Mar 29, 2023
View reviewed changes
Copy link
Member

@mooskagh mooskagh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me (although I didn't check for issues from NN perspective)

@borg323 borg323 merged commit 791cce4 into LeelaChessZero:master Mar 29, 2023
PikaCat-OuO pushed a commit to official-pikafish/px0 that referenced this pull request Apr 14, 2023
@almaudoh @PikaCat-OuO
Attention body support for blas / eigen / mkl (LeelaChessZero#1852)
5ab7895 
* remove mha transpose

* refactor blas encoder

* Attention body support to blas.

* Replace 64 with kSquares

* vectorize activation functions

---
Co-authored-by: borg323 <[email protected]>
Co-authored-by: Alma <[email protected]>

(cherry picked from commit 791cce4)
@almaudoh almaudoh mentioned this pull request Aug 3, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@borg323 borg323 borg323 left review comments

@mooskagh mooskagh mooskagh approved these changes

@Tilps Tilps Awaiting requested review from Tilps

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@almaudoh @mooskagh @borg323 @trre123 @zz4032 @Naphthalin @ankan-ban