Skip to content

add support for hdf5#1163

Merged
amcadmus merged 11 commits intodeepmodeling:develfrom
njzjz:hdf5
Sep 21, 2021
Merged

add support for hdf5#1163
amcadmus merged 11 commits intodeepmodeling:develfrom
njzjz:hdf5

Conversation

@njzjz
Copy link
Member

@njzjz njzjz commented Sep 21, 2021

dpdata format will be provided later.

The advantage of hdf5 is that it only has one file for multi-systems containing many systems, so it will be easier to transfer between servers or share online.

File size

About the file size, take an example of methane_v1:

  • Original file: 320 MB
  • Compressed file (.tar.bz2): 182 MB
  • HDF5 file: 258 MB
  • Compressed HDF5 file (.tar.bz2): 183 MB

We can see HDF5 file will have no advantage if fully compressed, but it is smaller if not compressed. Also, compressing and decompressing a single file should be faster than compressing a lot of small files.

Usage

One can add a HDF5 file in the systems list in input JSON file.

@codecov-commenter
Copy link

codecov-commenter commented Sep 21, 2021

Codecov Report

Merging #1163 (000c768) into devel (077df3d) will decrease coverage by 11.38%.
The diff coverage is n/a.

❗ Current head 000c768 differs from pull request most recent head dc44906. Consider uploading reports for the commit dc44906 to get more accurate results
Impacted file tree graph

@@             Coverage Diff             @@
##            devel    #1163       +/-   ##
===========================================
- Coverage   75.66%   64.28%   -11.39%     
===========================================
  Files          89        5       -84     
  Lines        7048       14     -7034     
===========================================
- Hits         5333        9     -5324     
+ Misses       1715        5     -1710     
Impacted Files Coverage Δ
deepmd/entrypoints/config.py
deepmd/descriptor/__init__.py
deepmd/infer/deep_polar.py
deepmd/entrypoints/__init__.py
deepmd/descriptor/se_a.py
deepmd/infer/deep_eval.py
deepmd/entrypoints/doc.py
deepmd/utils/learning_rate.py
deepmd/descriptor/se.py
deepmd/__main__.py
... and 71 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 077df3d...dc44906. Read the comment docs.

@njzjz njzjz requested a review from amcadmus September 21, 2021 20:19
@amcadmus amcadmus merged commit 53f1567 into deepmodeling:devel Sep 21, 2021
@njzjz njzjz deleted the hdf5 branch September 22, 2021 04:39
njzjz added a commit to njzjz/dpdata that referenced this pull request Oct 8, 2021
amcadmus pushed a commit to deepmodeling/dpdata that referenced this pull request Oct 10, 2021
* add deepmd/hdf5 format

To support deepmodeling/deepmd-kit#1163.

* bugfix

* add an example
njzjz pushed a commit to njzjz/deepmd-kit that referenced this pull request Sep 21, 2023
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants