set allow_growth in default_tf_session_config

[njzjz]

Currently, TF only initializes GPUs once, and never releases memory. As a result, we must set allow_growth in the first Session (or every session?), otherwise it doesn't work.

Please confirm the allow_growth config in trainer.py is expected, otherwise we should remove it instead.

deepmd-kit/deepmd/train/trainer.py

Line 400 in 1a25414

config.gpu_options.allow_growth = True

[njzjz]

ping @shishaochen

[codecov-commenter]

Codecov Report

Merging #1067 (8930772) into devel (8dbc211) will decrease coverage by 10.88%.
The diff coverage is n/a.

❗ Current head 8930772 differs from pull request most recent head 30ae6e0. Consider uploading reports for the commit 30ae6e0 to get more accurate results

@@             Coverage Diff             @@
##            devel    #1067       +/-   ##
===========================================
- Coverage   75.16%   64.28%   -10.89%     
===========================================
  Files          87        5       -82     
  Lines        6959       14     -6945     
===========================================
- Hits         5231        9     -5222     
+ Misses       1728        5     -1723     

Impacted Files	Coverage Δ
deepmd/utils/learning_rate.py
deepmd/descriptor/hybrid.py
deepmd/infer/deep_polar.py
deepmd/model/model_stat.py
source/op/_prod_force_se_r_grad.py
deepmd/entrypoints/freeze.py
deepmd/infer/__init__.py
source/op/_tabulate_grad.py
deepmd/utils/tabulate.py
deepmd/calculator.py
... and 71 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8dbc211...30ae6e0. Read the comment docs.