decouple activation function's type from model compression's process in SE_A, now tanh & gelu is both available. #1020

liangadam · 2021-08-23T16:24:44Z

I modified 3 pieces of code and wrote a brief unittest 'source/tests/test_tabulate.py' .

Now, when the 'activation_function' is designated as 'tanh' or 'gelu' in descriptor SE_A, model compression can produce a reasonable model.

Moreover, cause the type of activation function is fully decoupled from the process of model compression, now we can add any other function type we need in a very easy way.

…ssion's process in SE_A, now tanh & gelu is both available.

codecov-commenter · 2021-08-23T18:19:33Z

Codecov Report

Merging #1020 (b946719) into devel (fd1f371) will decrease coverage by 10.72%.
The diff coverage is n/a.

❗ Current head b946719 differs from pull request most recent head 1c71fee. Consider uploading reports for the commit 1c71fee to get more accurate results

@@             Coverage Diff             @@
##            devel    #1020       +/-   ##
===========================================
- Coverage   75.01%   64.28%   -10.73%     
===========================================
  Files          86        5       -81     
  Lines        6924       14     -6910     
===========================================
- Hits         5194        9     -5185     
+ Misses       1730        5     -1725

Impacted Files	Coverage Δ
source/op/_gelu.py
deepmd/utils/learning_rate.py
deepmd/infer/__init__.py
deepmd/cluster/slurm.py
deepmd/infer/model_devi.py
source/op/_tabulate_grad.py
deepmd/op/__init__.py
deepmd/infer/deep_polar.py
deepmd/infer/deep_dipole.py
deepmd/model/__init__.py
... and 60 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fd1f371...1c71fee. Read the comment docs.

njzjz

Also, please notice that tests failed.

deepmd/utils/tabulate.py

source/op/unaggregated_grad.cc

denghuilu

The code indentation is inconsistent in multiple places, please check it.

deepmd/utils/tabulate.py

source/op/unaggregated_grad.cc

njzjz · 2021-08-25T07:06:57Z

Please revert changes from this commit to tabulate.py... We do not accept style format without actual change for this reason.

amcadmus · 2021-08-25T09:33:00Z

source/op/unaggregated_grad.cc

-//     UnaggregatedDyDxOp<GPUDevice, T>);                         
+//     UnaggregatedDyDxOp<GPUDevice, T>);


Plz revert all such format changes! thank you

njzjz · 2021-08-25T21:00:49Z

source/op/unaggregated_grad.cc

-	    					     y.shape(),
-	    					     &dy_dx));
+                                 y.shape(),
+                                 &dy_dx));


Should be reverted.

njzjz · 2021-08-25T21:00:57Z

source/op/unaggregated_grad.cc

-	    					     y.shape(),
-	    					     &dy2_dx));
+                                 y.shape(),
+                                 &dy2_dx));



Should be reverted.

plz revert to exactly the same as before

njzjz · 2021-08-25T21:01:05Z

source/op/unaggregated_grad.cc

-	    					     z.shape(),
-	    					     &dz_dx));
+                                 z.shape(),
+                                 &dz_dx));


Should be reverted.

njzjz · 2021-08-25T21:01:11Z

source/op/unaggregated_grad.cc

-	    					     z.shape(),
-	    					     &dz2_dx));
+                                 z.shape(),
+                                 &dz2_dx));


Should be reverted.

The original code mixed tab and space, using them in the same line. I just have normalized them.

amcadmus · 2021-08-25T23:53:56Z

deepmd/utils/tabulate.py

+        # functype
+        if activation_fn == ACTIVATION_FN_DICT["tanh"]:
+            self.functype = 1
+        elif activation_fn == ACTIVATION_FN_DICT["gelu"]:
+            self.functype = 2
+        else:
+            raise RunTimeError("Unknown actication function type!")


we should support all activation functions in ACTIVATION_FN_DICT

Yes, it will be done in future work.

amcadmus · 2021-08-26T02:25:22Z

source/op/unaggregated_grad.cc

-	    					     y.shape(),
-	    					     &dy_dx));
+                                                         y.shape(),
+                                                         &dy_dx));


plz revert to exactly the same as before

amcadmus · 2021-08-26T02:25:30Z

source/op/unaggregated_grad.cc

-	    					     y.shape(),
-	    					     &dy2_dx));
+                                 y.shape(),
+                                 &dy2_dx));



plz revert to exactly the same as before

amcadmus · 2021-08-26T02:25:41Z

source/op/unaggregated_grad.cc

-	    					     z.shape(),
-	    					     &dz_dx));
+                                                         z.shape(),
+                                                         &dz_dx));


plz revert to exactly the same as before

amcadmus · 2021-08-26T02:26:01Z

source/op/unaggregated_grad.cc

-	    					     z.shape(),
-	    					     &dz2_dx));
+                                                         z.shape(),
+                                                         &dz2_dx));


plz revert to exactly the same as before

OK. Tab 4space Tab Tab Tab Tab Tab 4space

denghuilu

nice work @liangadam

…in SE_A, now tanh & gelu is both available. (deepmodeling#1020) * commit-message: decouple activation function's type from model compression's process in SE_A, now tanh & gelu is both available. * commit-message: modified code and passed unittest * commit-message: Format Document * commit-message :Format revert * commit-message: format change * commit-message: Format change Co-authored-by: HLA <[email protected]>

…pmodeling#1020) python3.10 support: collection.Mapping -> collection.abc.Mapping repair this: deepmodeling/dpgen#1019

HLA and others added 2 commits August 24, 2021 00:00

commit-message: decouple activation function's type from model compre…

4877203

…ssion's process in SE_A, now tanh & gelu is both available.

Merge branch 'deepmodeling:devel' into devel

888c73c

njzjz requested changes Aug 23, 2021

View reviewed changes

deepmd/utils/tabulate.py Outdated Show resolved Hide resolved

njzjz reviewed Aug 23, 2021

View reviewed changes

deepmd/utils/tabulate.py Outdated Show resolved Hide resolved

njzjz reviewed Aug 23, 2021

View reviewed changes

source/op/unaggregated_grad.cc Outdated Show resolved Hide resolved

njzjz requested a review from denghuilu August 23, 2021 20:58

denghuilu reviewed Aug 23, 2021

View reviewed changes

liangadam and others added 2 commits August 25, 2021 12:15

Merge branch 'deepmodeling:devel' into devel

c0311e2

commit-message: modified code and passed unittest

95c1ea5

njzjz requested changes Aug 25, 2021

View reviewed changes

deepmd/utils/tabulate.py Outdated Show resolved Hide resolved

source/op/unaggregated_grad.cc Outdated Show resolved Hide resolved

commit-message: Format Document

2d7e537

amcadmus requested changes Aug 25, 2021

View reviewed changes

commit-message :Format revert

de03176

njzjz reviewed Aug 25, 2021

View reviewed changes

amcadmus reviewed Aug 25, 2021

View reviewed changes

commit-message: format change

7a68bd7

amcadmus requested changes Aug 26, 2021

View reviewed changes

commit-message: Format change

1c71fee

amcadmus approved these changes Aug 26, 2021

View reviewed changes

amcadmus requested a review from njzjz August 26, 2021 12:57

njzjz approved these changes Aug 26, 2021

View reviewed changes

denghuilu approved these changes Aug 26, 2021

View reviewed changes

amcadmus merged commit 5ab5fa1 into deepmodeling:devel Aug 27, 2021

tuoping mentioned this pull request Sep 14, 2021

[BUG] Compress model command covers the original checkpoint. #1146

Closed

njzjz pushed a commit to njzjz/deepmd-kit that referenced this pull request Sep 21, 2023

python3.10 support: collection.Mapping -> collection.abc.Mapping (dee…

7e6dfd3

…pmodeling#1020) python3.10 support: collection.Mapping -> collection.abc.Mapping repair this: deepmodeling/dpgen#1019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

decouple activation function's type from model compression's process in SE_A, now tanh & gelu is both available. #1020

decouple activation function's type from model compression's process in SE_A, now tanh & gelu is both available. #1020

liangadam commented Aug 23, 2021

codecov-commenter commented Aug 23, 2021 •

edited

Loading

njzjz left a comment

denghuilu left a comment

njzjz commented Aug 25, 2021

amcadmus Aug 25, 2021

njzjz Aug 25, 2021

njzjz Aug 25, 2021

amcadmus Aug 26, 2021

njzjz Aug 25, 2021

njzjz Aug 25, 2021

liangadam Aug 26, 2021

amcadmus Aug 25, 2021

liangadam Aug 26, 2021

amcadmus Aug 26, 2021

amcadmus Aug 26, 2021

amcadmus Aug 26, 2021

amcadmus Aug 26, 2021

liangadam Aug 26, 2021

denghuilu left a comment

		// UnaggregatedDyDxOp<GPUDevice, T>);
		// UnaggregatedDyDxOp<GPUDevice, T>);

decouple activation function's type from model compression's process in SE_A, now tanh & gelu is both available. #1020

decouple activation function's type from model compression's process in SE_A, now tanh & gelu is both available. #1020

Conversation

liangadam commented Aug 23, 2021

codecov-commenter commented Aug 23, 2021 • edited Loading

Codecov Report

njzjz left a comment

Choose a reason for hiding this comment

denghuilu left a comment

Choose a reason for hiding this comment

njzjz commented Aug 25, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

denghuilu left a comment

Choose a reason for hiding this comment

codecov-commenter commented Aug 23, 2021 •

edited

Loading