Upload attention based model by iProzd · Pull Request #1866 · deepmodeling/deepmd-kit

iProzd · 2022-08-19T17:00:24Z

Here I upload the implementation of attention based model (DPA-1), see details in paper: https://arxiv.org/abs/2208.08236.
The changes can mainly be listed as following:

Add a new descriptor 'se_atten', using attention schemes and type embeddings (deepmd/descriptor/se_atten.py);
Add a new custom op 'ProdEnvMatAMix', mixing atoms with different types in the sorting, which is only suitable for descriptor using type embeddings (source/op/prod_env_mat_multi_device.cc and other sub files, here I try the best to reuse the previous codes);
Other changes to match the newly added descriptor, such as 'sel' supporting 'int' type as input (deepmd/entrypoints/train.py ) and type embedding with padding (deepmd/utils/type_embed.py);
Mixed_type: a new format of DeepmdData to put different systems sharing the same 'nloc' (instead of same fingerprints) together. (deepmd/utils/data.py & data_system.py) Note that this is only the training code on this new format of data. The generation code of this new format will soon be uploaded in dpdata.
An example of DPA-1 on water system (examples/water/se_atten/input.json);
The unittests on both C++&CUDA&ROCM and python interface of newly added apis(source/lib/tests & source/tests).

Unittests passed individually for these newly added features. Thanks for reviewing and disccusions are welcome.

Devel update

Devel update 2.0.0 with more features

Devel update

iProzd · 2022-08-19T17:08:11Z

@amcadmus @njzjz

codecov-commenter · 2022-08-19T17:27:04Z

Codecov Report

Merging #1866 (b5e31cb) into devel (a5522e2) will decrease coverage by 15.44%.
The diff coverage is n/a.

❗ Current head b5e31cb differs from pull request most recent head becf0cb. Consider uploading reports for the commit becf0cb to get more accurate results

@@             Coverage Diff             @@
##            devel    #1866       +/-   ##
===========================================
- Coverage   76.98%   61.53%   -15.45%     
===========================================
  Files         117        4      -113     
  Lines        9417       13     -9404     
===========================================
- Hits         7250        8     -7242     
+ Misses       2167        5     -2162

Impacted Files	Coverage Δ
source/lib/include/neighbor_list.h	`100.00% <ø> (ø)`
deepmd/common.py
deepmd/descriptor/__init__.py
deepmd/entrypoints/train.py
deepmd/fit/ener.py
deepmd/model/ener.py
deepmd/train/run_options.py
deepmd/train/trainer.py
deepmd/utils/argcheck.py
deepmd/utils/convert.py
... and 104 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

njzjz

This is a preliminary review.

Besides, there is no documentation to describe it.

source/lib/src/prod_env_mat.cc

deepmd/entrypoints/train.py

source/lib/src/neighbor_list.cc

examples/water/se_atten/input.json

deepmd/descriptor/se_atten.py

deepmd/fit/ener.py

deepmd/utils/data.py

wanghan-iapcm

The organization and the data format of the DPA model is different from the standard DP model. We should explain in the doc what is the difference and how to prepare data.

source/tests/common.py

deepmd/descriptor/se_atten.py

deepmd/utils/data_system.py

deepmd/utils/type_embed.py

source/lib/src/prod_env_mat.cc

source/op/prod_env_mat_multi_device.cc

njzjz · 2022-08-23T17:06:02Z

Depth of TOC is wrong:

deepmd/utils/network.py

source/op/prod_env_mat_multi_device.cc

deepmd/utils/argcheck.py

Add docs to ntype and nmask in ProdEnvMatAMix. Change nmask from FPTYPE to bool. Fix bugs in toc depth.

deepmd/utils/type_embed.py

deepmd/train/trainer.py

doc/model/train-se-atten.md

deepmd/utils/data.py

deepmd/utils/data_system.py

source/tests/common.py

Delete the need of 'real_atom_numbs.npy' from both code and doc. Add correct checks for each systems of mix_type format. Add none support in get_activation_func.

deepmd/common.py

deepmd/entrypoints/train.py

Change the default values of type_embedding_args in argcheck.py. Match args in TypeEmbedNet of se_atten with default ones in type_embedding_args.

deepmd/utils/argcheck.py

Change default activation_function back to 'tanh' in tebd. Use None in se_atten if not provided.

) Fixes #1400. Fixes #2009. 1. Uses cmake native module `CMakeDetermineHIPCompiler` to find the search path; 2. for ROCm>=3.5.1, `hip-hcc hiprtc` is replaced by `amd_comgr amdhip64`, per ROCm/ROCm#1200. (I am not sure about the situation of `amd_comgr`?) 3. Removes `-hc` from the flag for ROCm>=3.5.1. 4. Bumps from C++11 to C++14 as C++ 14 required by `amd_comgr`. 5. Removes `--amdgpu-target=gfx906`. I don't see the reason why it is in the flag. 6. Fixes a typo in #1866. Signed-off-by: Jinzhe Zeng <jinzhe.zeng@rutgers.edu>

Here I upload the implementation of attention based model (DPA-1), see details in paper: https://arxiv.org/abs/2208.08236. The changes can mainly be listed as following: 1. Add a new descriptor '**se_atten**', using attention schemes and type embeddings (deepmd/descriptor/se_atten.py); 2. Add a new custom op '**ProdEnvMatAMix**', mixing atoms with different types in the sorting, which is only suitable for descriptor using type embeddings (source/op/prod_env_mat_multi_device.cc and other sub files, here I try the best to reuse the previous codes); 3. Other changes to match the newly added descriptor, such as 'sel' supporting 'int' type as input (deepmd/entrypoints/train.py ) and type embedding with padding (deepmd/utils/type_embed.py); 4. **Mixed_type**: a new format of DeepmdData to put different systems sharing the same 'nloc' (instead of same fingerprints) together. (deepmd/utils/data.py & data_system.py) Note that this is only the training code on this new format of data. The generation code of this new format will soon be uploaded in _dpdata_. 5. An example of DPA-1 on water system (examples/water/se_atten/input.json); 6. The unittests on both C++&CUDA&ROCM and python interface of newly added apis(source/lib/tests & source/tests). Unittests passed individually for these newly added features. Thanks for reviewing and disccusions are welcome.

…epmodeling#2012) Fixes deepmodeling#1400. Fixes deepmodeling#2009. 1. Uses cmake native module `CMakeDetermineHIPCompiler` to find the search path; 2. for ROCm>=3.5.1, `hip-hcc hiprtc` is replaced by `amd_comgr amdhip64`, per ROCm/ROCm#1200. (I am not sure about the situation of `amd_comgr`?) 3. Removes `-hc` from the flag for ROCm>=3.5.1. 4. Bumps from C++11 to C++14 as C++ 14 required by `amd_comgr`. 5. Removes `--amdgpu-target=gfx906`. I don't see the reason why it is in the flag. 6. Fixes a typo in deepmodeling#1866. Signed-off-by: Jinzhe Zeng <jinzhe.zeng@rutgers.edu>

iProzd and others added 16 commits June 1, 2021 23:42

Merge pull request #11 from deepmodeling/devel

a6ecd4b

Devel update

add gpu op unittest in source/tests

aab1931

fix bug in deepmodeling#713

d5dca99

Merge pull request #12 from deepmodeling/devel

91b68f5

Devel update

Fix bug of empty input in gelu.cu

9f9fdae

Merge pull request #13 from deepmodeling/devel

a958bcb

Devel update

Merge pull request #14 from deepmodeling/devel

5144a53

Devel update

Merge pull request #15 from deepmodeling/devel

46ab4e8

Devel update

Merge pull request #16 from deepmodeling/devel

7372db6

Devel update

Merge pull request #17 from deepmodeling/devel

7072344

Devel update

Merge pull request #18 from deepmodeling/devel

978b37c

Devel update

Merge pull request #19 from deepmodeling/devel

1cdf4c1

Devel update 2.0.0 with more features

Merge pull request #21 from deepmodeling/devel

42ce5b3

Devel update

Merge pull request #22 from deepmodeling/devel

b3ad9a5

Devel update

Merge pull request #23 from deepmodeling/devel

d807db9

Devel update

Upload attention based model

cb035d4

github-actions bot added Core CUDA Examples OP Python ROCM labels Aug 19, 2022

Fix bugs in DPA-1 when trying to compress other model

c31b283

njzjz requested changes Aug 19, 2022

View reviewed changes

wanghan-iapcm requested changes Aug 20, 2022

View reviewed changes

iProzd added 2 commits August 20, 2022 12:45

Add layernorm support for tf early version

c5d9d68

Upload the model image

b66c2ff

github-actions bot added the Docs label Aug 20, 2022

Add docs in toc

a54524f

njzjz reviewed Aug 23, 2022

View reviewed changes

deepmd/utils/network.py Outdated Show resolved Hide resolved

source/op/prod_env_mat_multi_device.cc Outdated Show resolved Hide resolved

deepmd/utils/argcheck.py Outdated Show resolved Hide resolved

Add docs to ntype and nmask

56e0386

Add docs to ntype and nmask in ProdEnvMatAMix. Change nmask from FPTYPE to bool. Fix bugs in toc depth.

iProzd requested review from njzjz and wanghan-iapcm and removed request for njzjz August 24, 2022 07:40

Fix duplicated period in each doc_activation_function

6a77184

iProzd requested review from njzjz and wanghan-iapcm and removed request for njzjz and wanghan-iapcm August 24, 2022 10:02

wanghan-iapcm requested a review from njzjz August 24, 2022 10:50

wanghan-iapcm requested changes Aug 24, 2022

View reviewed changes

iProzd added 3 commits August 25, 2022 00:02

Optimized mixed_type format

4bbccfe

Delete the need of 'real_atom_numbs.npy' from both code and doc. Add correct checks for each systems of mix_type format. Add none support in get_activation_func.

Update common.py

798d864

Git reset common.py of format editing.

3912df5

njzjz reviewed Aug 25, 2022

View reviewed changes

deepmd/common.py Show resolved Hide resolved

deepmd/entrypoints/train.py Show resolved Hide resolved

Match default args of tebd in se_atten

a6f8258

Change the default values of type_embedding_args in argcheck.py. Match args in TypeEmbedNet of se_atten with default ones in type_embedding_args.

njzjz reviewed Aug 25, 2022

View reviewed changes

deepmd/utils/argcheck.py Outdated Show resolved Hide resolved

Change default activation_function back to 'tanh' in tebd.

becf0cb

Change default activation_function back to 'tanh' in tebd. Use None in se_atten if not provided.

njzjz approved these changes Aug 25, 2022

View reviewed changes

njzjz requested a review from wanghan-iapcm August 25, 2022 17:09

wanghan-iapcm approved these changes Aug 26, 2022

View reviewed changes

wanghan-iapcm merged commit 3b00375 into deepmodeling:devel Aug 26, 2022

njzjz mentioned this pull request Oct 19, 2022

Fix compatibility with ROCm>=3.5.1; fix typo in hip neighbor_list #2012

Merged

njzjz mentioned this pull request Dec 12, 2022

store energy bias with interface precision #2174

Merged

Conversation

iProzd commented Aug 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iProzd commented Aug 19, 2022

Uh oh!

codecov-commenter commented Aug 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

njzjz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wanghan-iapcm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

njzjz commented Aug 23, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

iProzd commented Aug 19, 2022 •

edited

Loading

codecov-commenter commented Aug 19, 2022 •

edited

Loading