summer-ospp 2022: 飞桨PaddlePaddle Sparse Conv开发和优化: gather-gemm-scatter fuse #46679

umiswing · 2022-09-30T08:40:43Z

PR types

Performance optimization

PR changes

APIs, Others

Describe

Include cutlass in Paddle, add cutlass.cmake to download cutlass v2.9.1 automatically.
Fuse gather, gemm and scatter in Conv3dCOOGpuKernel with cutlass, this fusion supports fp16, fp32 and fp64, but only works for channels cutlass supports. I remain the old code for other channels.
Add gather_gemm_scatter.h and gather_gemm_scatter.cu. These two files define the details of the fused kernel.

Following is a comparison of running 500 times forward with gather-gemm-grouped fusion and without this fusion on GeForce RTX 3060 Mobile GPU. This fusion can achieve 1.80 speedups at most.

time(seconds)\shape	0(subm)	1(subm)	2(conv)	3(subm)	4(conv)	5(subm)	6(conv)	7(subm)	8(subm)	9(conv)
fp16
fused	0.948059366	0.948617946	0.918128433	0.654550546	1.247602738	0.643608378	3.765412761	0.467916381	0.469078923	0.166027756
old	0.963316033	0.978123623	1.100062921	0.85829027	1.841123472	1.073228994	6.164761058	0.813147195	0.815460634	0.211744664
spconv 2.1.21	0.370279025	0.356675464	1.255860301	0.67564418	1.989892343	0.646101865	4.651363121	0.397059219	0.399020844	0.274251656
old/fused	1.016092523	1.031103857	1.198157993	1.311266602	1.475728945	1.667518682	1.637207246	1.737804505	1.738429492	1.275357019
spconv/fused	0.390565231	0.375994852	1.367848174	1.032226135	1.594972728	1.00387423	1.235286386	0.848568751	0.850647566	1.651842214
fp32
fused	1.074077862	1.076842442	1.068928335	0.834602646	1.765838748	1.058910525	5.879794197	0.851036824	0.851216344	0.211274696
old	1.017575127	1.055912888	1.444613468	1.186929089	2.804123872	1.722058823	10.58918662	1.401214913	1.40421477	0.327167335
spconv 2.1.21	0.367197311	0.379458345	1.375788394	0.765276265	2.479875851	1.204778016	7.822352023	0.972676926	0.973764882	0.298347718
old/fused	0.947394191	0.980563959	1.351459607	1.422148725	1.58798411	1.626255271	1.800945112	1.646479769	1.649656729	1.548540082
spconv/fused	0.341872153	0.352380562	1.287072621	0.916934866	1.404361442	1.137752423	1.330378541	1.142931655	1.143968732	1.412131806
fp64
fused	1.420543399	1.208440223	2.657113268	2.489402152	9.968613399	9.779180488	65.87212427	9.246852072	9.22867435	1.489004803
old	1.544003572	1.587914612	3.242279789	3.420265195	11.49426685	11.72775161	oom	11.08054339	11.07393647	1.80335558
spconv 2.1.21(not support)
old/fused	1.086910525	1.314019992	1.220226412	1.37393036	1.153045703	1.199257098		1.198304386	1.199948774	1.211114683

Shape details:

	(x, y, z)	in_channels	out_channels	kernel_size	strides	paddings	nnz
0	(41, 1600, 1408)	4	16	(3, 3, 3)	(1, 1, 1)	(0, 0, 0)	136000
1	(41, 1600, 1408)	16	16	(3, 3, 3)	(1, 1, 1)	(0, 0, 0)	136000
2	(41, 1600, 1408)	16	32	(3, 3, 3)	(2, 2, 2)	(1, 1, 1)	136000
3	(21, 800, 704)	32	32	(3, 3, 3)	(1, 1, 1)	(0, 0, 0)	220939
4	(21, 800, 704)	32	64	(3, 3, 3)	(2, 2, 2)	(1, 1, 1)	220939
5	(11, 400, 352)	64	64	(3, 3, 3)	(1, 1, 1)	(0, 0, 0)	146376
6	(11, 400, 352)	64	64	(3, 3, 3)	(1, 1, 1)	(0, 0, 0)	146376
7	(5, 200, 176)	64	64	(3, 3, 3)	(2, 2, 2)	(0, 1, 1)	65421
8	(5, 200, 176)	64	64	(3, 3, 3)	(1, 1, 1)	(0, 0, 0)	65421
9	(5, 200, 176)	64	64	(3, 1, 1)	(2, 1, 1)	(0, 0, 0)	65421

… develop

…tlass

… develop

binary.h.

* Convert slice+grad oneDNN fluid kernels to PHI * Change mutable_data to Alloc * Refactor licences

…t in dy2static (PaddlePaddle#46128) * add numpy api err msg * fix bug * fix unittest * add set_state_dict err * rewrite numpy_api_check * add define * change err msg * fix test * move import statement

… fuse

zkh2016 · 2022-10-31T02:47:03Z

paddle/phi/kernels/sparse/gpu/conv_kernel.cu

@@ -1,11 +1,8 @@
 /* Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
-


这里应该是有空行的？

zkh2016

LGTM。
辛苦后面补充下反向融合的实现。

… fuse

…r fuse (PaddlePaddle#46679)

cherry-pick some PR about optimize sparse kernel and fix some bug： #47736 #47703 #47604 #46679 #48439 #49009 #49734

umiswing and others added 30 commits June 30, 2022 15:34

add sparse conv test, compared with spconv

6752bba

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

b2d34f5

… develop

update test with spconv

39849db

some debug print

b01748e

add cutlass to sparse_conv, first version

d58a3fa

support input channel padding. remove some comments

1ba0050

Merge remote-tracking branch 'refs/remotes/origin/develop' into develop

0db6e6a

Merge branch 'cutlass' into develop

75fc91e

restructure align code

a6268d3

restructure align code again, only works for input channel align now

25d9f31

support output channel align. the code need to be restructure.

549b8d8

add option WITH_CUTLASS in CMakeLists.txt to choose whether to use cu…

2be352a

…tlass

restructure align code. add warmup to benchmark

168164b

add an option to use cutlass. only for fp16 now

cbc1350

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

40be4b8

… develop

use group gemm, but still slow

83346ae

track cutlass

7e2cd84

add gather-group-gemm-fusion, faster on fp32 now :)

93f3e19

add some comments

c76c351

restructure code, conv3d_coo can choose to use cutlass for fp32

d1181e5

add custom gather-gemm-grouped fusion file based on cutlass

0777605

Add cutlass.cmake to auto download cutlass 2.9

36f28e1

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

06b5394

… develop

fix header include path

94252fb

remove some debug function

b584cc4

update test

1e3b555

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

605f11f

… develop

restructure conv3dcoo. add 'bool cutlass' to interface in binary.cc and

38f2f83

binary.h.

[PHI] Migrate sgd and stack oneDNN kernels (PaddlePaddle#46374)

709fbe9

* Convert slice+grad oneDNN fluid kernels to PHI * Change mutable_data to Alloc * Refactor licences

[error msg] update error msg of user call numpy api and set_state_dic…

3d54a9b

…t in dy2static (PaddlePaddle#46128) * add numpy api err msg * fix bug * fix unittest * add set_state_dict err * rewrite numpy_api_check * add define * change err msg * fix test * move import statement

umiswing added 11 commits October 26, 2022 18:44

add tuned kernel for fp64

a2838d0

remove unused code

109c296

update tuned kernel for fp16

6674317

update tuned kernel for fp32

2bf8825

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

c31a319

… fuse

remove cutlass option in kernel. remove unused group fusion.

f2a0c2a

fix ci: don't build gather_gemm_scatter when WITH_CUTLASS off

4be3fbc

fix ci: fix build error when WITH_CUTLASS off

52b2834

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

1a890f2

… fuse

restructure code

ed55393

restructure code again

4ee2235

umiswing changed the title ~~summer-ospp 2022: 飞桨PaddlePaddle Sparse Conv开发和优化-group-gather-gemm fuse~~ summer-ospp 2022: 飞桨PaddlePaddle Sparse Conv开发和优化: gather-gemm-scatter fuse Oct 31, 2022

zkh2016 reviewed Oct 31, 2022

View reviewed changes

remove some useless changing

a34a41c

zkh2016 previously approved these changes Oct 31, 2022

View reviewed changes

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

f79f7b0

… fuse

umiswing dismissed zkh2016’s stale review via 12ace7d October 31, 2022 07:28

remove PADDLE_WITH_CUTLASS and WITH_CUTLASS

113ef38

umiswing force-pushed the fuse branch from 12ace7d to 113ef38 Compare October 31, 2022 07:38

umiswing added 5 commits October 31, 2022 16:40

fix ci

60ef88b

fix ci

7f3e05f

fix ci

fb0bb92

fix ci again

ab3fda2

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

ba201b1

… fuse

zkh2016 approved these changes Oct 31, 2022

View reviewed changes

zkh2016 merged commit 5158fa4 into PaddlePaddle:develop Oct 31, 2022

zkh2016 pushed a commit to zkh2016/Paddle that referenced this pull request Jan 31, 2023

summer-ospp 2022: 飞桨PaddlePaddle Sparse Conv开发和优化: gather-gemm-scatte…

e14c1fc

…r fuse (PaddlePaddle#46679)

zkh2016 mentioned this pull request Feb 2, 2023

[cherry-pick] Optimize sparse kernel and fix some bug #50118

Merged

lanxianghit pushed a commit that referenced this pull request Feb 2, 2023

[cherry-pick] Optimize sparse kernel and fix some bug (#50118)

8c5e432

cherry-pick some PR about optimize sparse kernel and fix some bug： #47736 #47703 #47604 #46679 #48439 #49009 #49734

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

summer-ospp 2022: 飞桨PaddlePaddle Sparse Conv开发和优化: gather-gemm-scatter fuse #46679

summer-ospp 2022: 飞桨PaddlePaddle Sparse Conv开发和优化: gather-gemm-scatter fuse #46679

umiswing commented Sep 30, 2022 •

edited

Loading

zkh2016 Oct 31, 2022

zkh2016 left a comment

		@@ -1,11 +1,8 @@
		/* Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.

summer-ospp 2022: 飞桨PaddlePaddle Sparse Conv开发和优化: gather-gemm-scatter fuse #46679

summer-ospp 2022: 飞桨PaddlePaddle Sparse Conv开发和优化: gather-gemm-scatter fuse #46679

Conversation

umiswing commented Sep 30, 2022 • edited Loading

PR types

PR changes

Describe

zkh2016 Oct 31, 2022

Choose a reason for hiding this comment

zkh2016 left a comment

Choose a reason for hiding this comment

umiswing commented Sep 30, 2022 •

edited

Loading