Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor property fitting interface #4471

Merged
merged 97 commits into from
Dec 25, 2024
Merged
Show file tree
Hide file tree
Changes from 94 commits
Commits
Show all changes
97 commits
Select commit Hold shift + click to select a range
4a69e22
change property.npy to any name
Chengqian-Zhang Dec 8, 2024
157f70c
Init branch
Chengqian-Zhang Dec 13, 2024
6172dc4
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 13, 2024
0608e17
Merge branch 'devel' into refactor_property
Chengqian-Zhang Dec 13, 2024
4c2033f
change | to Union
Chengqian-Zhang Dec 13, 2024
ac5fa5c
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 13, 2024
69803d6
change sub_var_name default to []
Chengqian-Zhang Dec 13, 2024
1fcf82c
Solve pre-commit
Chengqian-Zhang Dec 13, 2024
5be3457
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 13, 2024
9637b6a
solve scanning github
Chengqian-Zhang Dec 13, 2024
5ce6d31
fix UT
Chengqian-Zhang Dec 13, 2024
5a3bf94
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 13, 2024
13e1911
delete useless file
Chengqian-Zhang Dec 13, 2024
ff4650e
Solve some UT
Chengqian-Zhang Dec 13, 2024
9637390
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 13, 2024
4edf26f
Solve precommit
Chengqian-Zhang Dec 13, 2024
4d35df2
Merge branch 'refactor_property' of github.com:Chengqian-Zhang/deepmd…
Chengqian-Zhang Dec 13, 2024
7f09038
slove pre
Chengqian-Zhang Dec 13, 2024
32e6deb
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 13, 2024
8fec403
Solve dptest UT, dpatomicmodel UT, code scannisang
Chengqian-Zhang Dec 14, 2024
ba54bcc
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 14, 2024
b52065c
delete param and
Chengqian-Zhang Dec 15, 2024
c0f09e6
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 15, 2024
2566b2e
Solve UT fail caused by task_dim and property_name
Chengqian-Zhang Dec 15, 2024
3af4970
Merge branch 'refactor_property' of github.com:Chengqian-Zhang/deepmd…
Chengqian-Zhang Dec 15, 2024
b1e834a
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 15, 2024
e33d6e6
Fix UT
Chengqian-Zhang Dec 15, 2024
5e1e892
Solve conflict
Chengqian-Zhang Dec 15, 2024
cdd4e18
Fix UT
Chengqian-Zhang Dec 15, 2024
86f6744
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 15, 2024
d75ea7a
Fix UT
Chengqian-Zhang Dec 15, 2024
0dbd2b4
Merge branch 'refactor_property' of github.com:Chengqian-Zhang/deepmd…
Chengqian-Zhang Dec 15, 2024
1cc5c37
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 15, 2024
44fa4d6
Fix permutation error
Chengqian-Zhang Dec 15, 2024
3249891
Merge branch 'refactor_property' of github.com:Chengqian-Zhang/deepmd…
Chengqian-Zhang Dec 15, 2024
8e9bbc5
Add property bias UT
Chengqian-Zhang Dec 15, 2024
1b8d92f
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 15, 2024
c83d91e
recover rcond doc
Chengqian-Zhang Dec 15, 2024
63cec86
recover blank
Chengqian-Zhang Dec 15, 2024
99df857
Change code according according to coderabbitai
Chengqian-Zhang Dec 15, 2024
4155294
solve pre-commit
Chengqian-Zhang Dec 15, 2024
a1a6583
Fix UT
Chengqian-Zhang Dec 15, 2024
2859f02
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 15, 2024
5d3c50b
change apply_bias doc
Chengqian-Zhang Dec 15, 2024
1094032
update the version compatibility
Chengqian-Zhang Dec 16, 2024
c9b7ab4
Merge branch 'devel' into refactor_property
Chengqian-Zhang Dec 18, 2024
15eb6d0
delete sub_var_name
Chengqian-Zhang Dec 18, 2024
dbf394c
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 18, 2024
fd42d53
recover to property key
Chengqian-Zhang Dec 18, 2024
e48eb8b
Solve conflict
Chengqian-Zhang Dec 18, 2024
5036545
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 18, 2024
a125d38
Fix conflict
Chengqian-Zhang Dec 18, 2024
bff3378
Merge branch 'refactor_property' of github.com:Chengqian-Zhang/deepmd…
Chengqian-Zhang Dec 18, 2024
85f9166
Merge branch 'devel' into refactor_property
Chengqian-Zhang Dec 18, 2024
d384f62
Fix UT
Chengqian-Zhang Dec 20, 2024
d43da75
Add document of property fitting
Chengqian-Zhang Dec 20, 2024
f4aeeaf
Delete checkpoint
Chengqian-Zhang Dec 20, 2024
4613425
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 20, 2024
f07f1b2
Add get_property_name to DeepEvalBackend
Chengqian-Zhang Dec 20, 2024
120d9d8
Merge branch 'refactor_property' of github.com:Chengqian-Zhang/deepmd…
Chengqian-Zhang Dec 20, 2024
2dc6486
Merge branch 'devel' into refactor_property
Chengqian-Zhang Dec 22, 2024
1d2d866
change doc to py
Chengqian-Zhang Dec 22, 2024
a50d363
Merge branch 'refactor_property' of github.com:Chengqian-Zhang/deepmd…
Chengqian-Zhang Dec 22, 2024
2e18007
Add out_bias out_std doc
Chengqian-Zhang Dec 22, 2024
4ff2a23
Merge branch 'devel' into refactor_property
Chengqian-Zhang Dec 23, 2024
eff7eb1
change bias method to compute_stats_do_not_distinguish_types
Chengqian-Zhang Dec 23, 2024
c9e47a3
Merge branch 'refactor_property' of github.com:Chengqian-Zhang/deepmd…
Chengqian-Zhang Dec 23, 2024
28ef352
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 23, 2024
e74a6ac
change var_name to property_name
Chengqian-Zhang Dec 23, 2024
6615837
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 23, 2024
a88a655
change logic of extensive bias
Chengqian-Zhang Dec 23, 2024
d10a4ba
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 23, 2024
9219ee1
add doc for neww added parameter
Chengqian-Zhang Dec 23, 2024
35bb6a7
Merge branch 'refactor_property' of github.com:Chengqian-Zhang/deepmd…
Chengqian-Zhang Dec 23, 2024
ec3369d
change doc for compute_stats_do_not_distinguish_types
Chengqian-Zhang Dec 23, 2024
ed59869
try to fix dptest
Chengqian-Zhang Dec 23, 2024
15846fd
change all property to property_name
Chengqian-Zhang Dec 23, 2024
34cc7ce
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 23, 2024
a1d2608
Fix UT
Chengqian-Zhang Dec 23, 2024
c70a887
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 23, 2024
c3f82aa
Delete key 'property' completely
Chengqian-Zhang Dec 23, 2024
a392718
Solve conflict
Chengqian-Zhang Dec 23, 2024
9bf75cb
Fix UT
Chengqian-Zhang Dec 23, 2024
6e428a7
Fix dptest UT
Chengqian-Zhang Dec 23, 2024
14a9a89
Merge branch 'devel' into refactor_property
Chengqian-Zhang Dec 24, 2024
f963383
Merge branch 'devel' into refactor_property
Chengqian-Zhang Dec 24, 2024
9e09892
Delete attribute
Chengqian-Zhang Dec 24, 2024
d097bfa
Merge branch 'refactor_property' of github.com:Chengqian-Zhang/deepmd…
Chengqian-Zhang Dec 24, 2024
f7cf9e1
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 24, 2024
f9d1b55
Solve comment
Chengqian-Zhang Dec 24, 2024
9ae4dfd
Merge branch 'refactor_property' of github.com:Chengqian-Zhang/deepmd…
Chengqian-Zhang Dec 24, 2024
ce24784
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 24, 2024
211ec00
Solve error
Chengqian-Zhang Dec 24, 2024
42cfe99
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 24, 2024
b473b6a
delete property_name in serialize
Chengqian-Zhang Dec 24, 2024
af8789f
Update train-fitting-property.md
Chengqian-Zhang Dec 25, 2024
6638ba6
Update train-fitting-property.md
Chengqian-Zhang Dec 25, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions deepmd/dpmodel/atomic_model/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,9 @@
from .polar_atomic_model import (
DPPolarAtomicModel,
)
from .property_atomic_model import (
DPPropertyAtomicModel,
)

__all__ = [
"BaseAtomicModel",
Expand All @@ -50,6 +53,7 @@
"DPDipoleAtomicModel",
"DPEnergyAtomicModel",
"DPPolarAtomicModel",
"DPPropertyAtomicModel",
"DPZBLLinearEnergyAtomicModel",
"LinearEnergyAtomicModel",
"PairTabAtomicModel",
Expand Down
24 changes: 24 additions & 0 deletions deepmd/dpmodel/atomic_model/property_atomic_model.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
# SPDX-License-Identifier: LGPL-3.0-or-later
import numpy as np

from deepmd.dpmodel.fitting.property_fitting import (
PropertyFittingNet,
)
Expand All @@ -15,3 +17,25 @@ def __init__(self, descriptor, fitting, type_map, **kwargs):
"fitting must be an instance of PropertyFittingNet for DPPropertyAtomicModel"
)
super().__init__(descriptor, fitting, type_map, **kwargs)

def apply_out_stat(
self,
ret: dict[str, np.ndarray],
atype: np.ndarray,
):
"""Apply the stat to each atomic output.
Chengqian-Zhang marked this conversation as resolved.
Show resolved Hide resolved

In property fitting, each output will be multiplied by label std and then plus the label average value.

Parameters
----------
ret
The returned dict by the forward_atomic method
atype
The atom types. nf x nloc. It is useless in property fitting.

"""
out_bias, out_std = self._fetch_out_stat(self.bias_keys)
for kk in self.bias_keys:
ret[kk] = ret[kk] * out_std[kk][0] + out_bias[kk][0]
Chengqian-Zhang marked this conversation as resolved.
Show resolved Hide resolved
return ret
17 changes: 9 additions & 8 deletions deepmd/dpmodel/fitting/property_fitting.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,10 +41,9 @@ class PropertyFittingNet(InvarFitting):
this list is of length :math:`N_l + 1`, specifying if the hidden layers and the output layer are trainable.
intensive
Whether the fitting property is intensive.
bias_method
The method of applying the bias to each atomic output, user can select 'normal' or 'no_bias'.
If 'normal' is used, the computed bias will be added to the atomic output.
If 'no_bias' is used, no bias will be added to the atomic output.
property_name:
The name of fitting property, which should be consistent with the property name in the dataset.
If the data file is named `humo.npy`, this parameter should be "humo".
resnet_dt
Time-step `dt` in the resnet construction:
:math:`y = x + dt * \phi (Wx + b)`
Expand Down Expand Up @@ -74,7 +73,7 @@ def __init__(
rcond: Optional[float] = None,
trainable: Union[bool, list[bool]] = True,
intensive: bool = False,
bias_method: str = "normal",
property_name: str = "property",
resnet_dt: bool = True,
numb_fparam: int = 0,
numb_aparam: int = 0,
Expand All @@ -89,9 +88,8 @@ def __init__(
) -> None:
self.task_dim = task_dim
self.intensive = intensive
self.bias_method = bias_method
super().__init__(
var_name="property",
var_name=property_name,
ntypes=ntypes,
dim_descrpt=dim_descrpt,
dim_out=task_dim,
Expand All @@ -113,7 +111,7 @@ def __init__(
@classmethod
def deserialize(cls, data: dict) -> "PropertyFittingNet":
data = data.copy()
check_version_compatibility(data.pop("@version"), 3, 1)
check_version_compatibility(data.pop("@version"), 4, 1)
data.pop("dim_out")
data.pop("var_name")
data.pop("tot_ener_zero")
Expand All @@ -131,6 +129,9 @@ def serialize(self) -> dict:
**InvarFitting.serialize(self),
"type": "property",
"task_dim": self.task_dim,
"intensive": self.intensive,
"property_name": self.var_name,
Chengqian-Zhang marked this conversation as resolved.
Show resolved Hide resolved
}
dd["@version"] = 4

return dd
6 changes: 3 additions & 3 deletions deepmd/dpmodel/model/property_model.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# SPDX-License-Identifier: LGPL-3.0-or-later
from deepmd.dpmodel.atomic_model.dp_atomic_model import (
DPAtomicModel,
from deepmd.dpmodel.atomic_model import (
DPPropertyAtomicModel,
)
from deepmd.dpmodel.model.base_model import (
BaseModel,
Expand All @@ -13,7 +13,7 @@
make_model,
)

DPPropertyModel_ = make_model(DPAtomicModel)
DPPropertyModel_ = make_model(DPPropertyAtomicModel)


@BaseModel.register("property")
Expand Down
20 changes: 14 additions & 6 deletions deepmd/entrypoints/test.py
Original file line number Diff line number Diff line change
Expand Up @@ -779,9 +779,17 @@ def test_property(
tuple[list[np.ndarray], list[int]]
arrays with results and their shapes
"""
data.add("property", dp.task_dim, atomic=False, must=True, high_prec=True)
var_name = dp.get_var_name()
assert isinstance(var_name, str)
data.add(var_name, dp.task_dim, atomic=False, must=True, high_prec=True)
if has_atom_property:
data.add("atom_property", dp.task_dim, atomic=True, must=False, high_prec=True)
data.add(
f"atom_{var_name}",
dp.task_dim,
atomic=True,
must=False,
high_prec=True,
)

if dp.get_dim_fparam() > 0:
data.add(
Expand Down Expand Up @@ -832,12 +840,12 @@ def test_property(
aproperty = ret[1]
aproperty = aproperty.reshape([numb_test, natoms * dp.task_dim])

diff_property = property - test_data["property"][:numb_test]
diff_property = property - test_data[var_name][:numb_test]
mae_property = mae(diff_property)
rmse_property = rmse(diff_property)

if has_atom_property:
diff_aproperty = aproperty - test_data["atom_property"][:numb_test]
diff_aproperty = aproperty - test_data[f"atom_{var_name}"][:numb_test]
mae_aproperty = mae(diff_aproperty)
rmse_aproperty = rmse(diff_aproperty)

Expand All @@ -854,7 +862,7 @@ def test_property(
detail_path = Path(detail_file)

for ii in range(numb_test):
test_out = test_data["property"][ii].reshape(-1, 1)
test_out = test_data[var_name][ii].reshape(-1, 1)
pred_out = property[ii].reshape(-1, 1)

frame_output = np.hstack((test_out, pred_out))
Expand All @@ -868,7 +876,7 @@ def test_property(

if has_atom_property:
for ii in range(numb_test):
test_out = test_data["atom_property"][ii].reshape(-1, 1)
test_out = test_data[f"atom_{var_name}"][ii].reshape(-1, 1)
pred_out = aproperty[ii].reshape(-1, 1)

frame_output = np.hstack((test_out, pred_out))
Expand Down
6 changes: 4 additions & 2 deletions deepmd/infer/deep_eval.py
Original file line number Diff line number Diff line change
Expand Up @@ -70,8 +70,6 @@ class DeepEvalBackend(ABC):
"dipole_derv_c_redu": "virial",
"dos": "atom_dos",
"dos_redu": "dos",
"property": "atom_property",
"property_redu": "property",
"mask_mag": "mask_mag",
"mask": "mask",
# old models in v1
Expand Down Expand Up @@ -276,6 +274,10 @@ def get_has_spin(self) -> bool:
"""Check if the model has spin atom types."""
return False

def get_var_name(self) -> str:
"""Get the name of the fitting property."""
raise NotImplementedError

@abstractmethod
def get_ntypes_spin(self) -> int:
"""Get the number of spin atom types of this model. Only used in old implement."""
Expand Down
44 changes: 33 additions & 11 deletions deepmd/infer/deep_property.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,25 +37,41 @@ class DeepProperty(DeepEval):
Keyword arguments.
"""

@property
def output_def(self) -> ModelOutputDef:
"""Get the output definition of this model."""
return ModelOutputDef(
"""
Get the output definition of this model.
But in property_fitting, the output definition is not known until the model is loaded.
So we need to rewrite the output definition after the model is loaded.
See detail in change_output_def.
"""
pass

def change_output_def(self) -> None:
"""
Change the output definition of this model.
In property_fitting, the output definition is known after the model is loaded.
We need to rewrite the output definition and related information.
"""
self.output_def = ModelOutputDef(
FittingOutputDef(
[
OutputVariableDef(
"property",
shape=[-1],
self.get_var_name(),
shape=[self.get_task_dim()],
reducible=True,
atomic=True,
intensive=self.get_intensive(),
),
]
)
)

def change_output_def(self) -> None:
self.output_def["property"].shape = self.task_dim
self.output_def["property"].intensive = self.get_intensive()
self.deep_eval.output_def = self.output_def
self.deep_eval._OUTDEF_DP2BACKEND[self.get_var_name()] = (
f"atom_{self.get_var_name()}"
)
self.deep_eval._OUTDEF_DP2BACKEND[f"{self.get_var_name()}_redu"] = (
self.get_var_name()
)

@property
def task_dim(self) -> int:
Expand Down Expand Up @@ -120,10 +136,12 @@ def eval(
aparam=aparam,
**kwargs,
)
atomic_property = results["property"].reshape(
atomic_property = results[self.get_var_name()].reshape(
nframes, natoms, self.get_task_dim()
)
property = results["property_redu"].reshape(nframes, self.get_task_dim())
property = results[f"{self.get_var_name()}_redu"].reshape(
nframes, self.get_task_dim()
)

if atomic:
return (
Expand All @@ -141,5 +159,9 @@ def get_intensive(self) -> bool:
"""Get whether the property is intensive."""
return self.deep_eval.get_intensive()

def get_var_name(self) -> str:
"""Get the name of the fitting property."""
return self.deep_eval.get_var_name()


__all__ = ["DeepProperty"]
11 changes: 10 additions & 1 deletion deepmd/pt/infer/deep_eval.py
Original file line number Diff line number Diff line change
Expand Up @@ -184,6 +184,15 @@ def get_dim_aparam(self) -> int:
def get_intensive(self) -> bool:
return self.dp.model["Default"].get_intensive()

def get_var_name(self) -> str:
"""Get the name of the property."""
if hasattr(self.dp.model["Default"], "get_var_name") and callable(
getattr(self.dp.model["Default"], "get_var_name")
):
return self.dp.model["Default"].get_var_name()
else:
raise NotImplementedError

@property
def model_type(self) -> type["DeepEvalWrapper"]:
"""The the evaluator of the model type."""
Expand All @@ -200,7 +209,7 @@ def model_type(self) -> type["DeepEvalWrapper"]:
return DeepGlobalPolar
elif "wfc" in model_output_type:
return DeepWFC
elif "property" in model_output_type:
elif self.get_var_name() in model_output_type:
return DeepProperty
else:
raise RuntimeError("Unknown model type")
Expand Down
Loading
Loading