Skip to content

add an option to merge data to one H5 file#1119

Merged
wanghan-iapcm merged 1 commit intodeepmodeling:develfrom
njzjz:one_h5
Jan 19, 2023
Merged

add an option to merge data to one H5 file#1119
wanghan-iapcm merged 1 commit intodeepmodeling:develfrom
njzjz:one_h5

Conversation

@njzjz
Copy link
Member

@njzjz njzjz commented Jan 17, 2023 •
edited
Loading

Fix #617.

@njzjz
add an option to merge data to one H5 file
e870cda 
Signed-off-by: Jinzhe Zeng <jinzhe.zeng@rutgers.edu>
@njzjz njzjz linked an issue Jan 17, 2023 that may be closed by this pull request
[Feature Request] Use HDF5 instead of numpy files #617
Closed
@njzjz njzjz marked this pull request as ready for review January 17, 2023 21:16
@njzjz njzjz mentioned this pull request Jan 17, 2023
Closed
@njzjz
Copy link
Member Author

njzjz commented Jan 18, 2023

Below is the test on my systems:

INFO:dpgen:Combining 45929 training systems to iter.000007/00.train/data.hdf5...
INFO:dpgen:-------------------------iter.000007 task 01--------------------------
2023-01-17 18:01:36,925 - INFO : info:check_all_finished: False
2023-01-17 18:01:36,925 - INFO : remote path: /scratch/jz748/dpgen_workdir/5351930e3552f5d9c883518f7e9e839f81cce2aa
2023-01-17 18:04:23,468 - INFO : job: 82764aa11f3b7d0d57c49a263df5405f75da2720 submit; job_id is 24121180

A 1.7GB file is generated:

-rw-r--r--  1 jz748 york 1.7G Jan 17 17:59 data.hdf5

@wanghan-iapcm wanghan-iapcm merged commit fb8668b into deepmodeling:devel Jan 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@Vibsteamer Vibsteamer Vibsteamer approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Feature Request] Use HDF5 instead of numpy files

3 participants

@njzjz @Vibsteamer @wanghan-iapcm