Skip to content

Only apply HDF5 shuffle filter when compression is enabled #189

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
The9Cat merged 7 commits into from
Dec 11, 2025
Merged

Only apply HDF5 shuffle filter when compression is enabled #189

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
e1625e5
Initial plan
Copilot Dec 11, 2025
63e71d5
Apply shuffle filter only when compression is actually enabled
Copilot Dec 11, 2025
447734e
Add explicit compression check for shuffle filter in sample stats sec…
Copilot Dec 11, 2025
1afbfdc
Use explicit H5compress > 0 checks for consistency and clarity
Copilot Dec 11, 2025
a02348f
Simplify shuffle filter conditions and improve consistency
Copilot Dec 11, 2025
6b7271e
Clarify comment for Deflate-only compression section
Copilot Dec 11, 2025
ae5b00f
Final update to PR description with complete summary
Copilot Dec 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions _codeql_detected_source_root
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
.
8 changes: 6 additions & 2 deletions expui/BiorthBasis.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5166,9 +5166,11 @@ namespace BasisClasses
auto dcpl3 = HighFive::DataSetCreateProps{}; // covariance

// Properties for sample stats
if (H5compress) {
// Note: Sample stats only use Deflate compression, not Szip
if (H5compress > 0) {
unsigned int csz = sampleCounts.size();
dcpl1.add(HighFive::Chunking({csz, 1}));
// Only apply shuffle filter when Deflate compression is enabled (guaranteed by outer if)
if (H5shuffle) dcpl1.add(HighFive::Shuffle());
dcpl1.add(HighFive::Deflate(H5compress));
}
Expand Down Expand Up @@ -5207,7 +5209,7 @@ namespace BasisClasses
stanza.createAttribute<unsigned>
("fullCovar", HighFive::DataSpace::From(icov)).write(icov);

if (H5compress) {
if (H5compress > 0 || H5szip) {
// Szip parameters
const int options_mask = H5_SZIP_NN_OPTION_MASK;
const int pixels_per_block = 8;
Expand All @@ -5218,6 +5220,7 @@ namespace BasisClasses
HighFive::Chunking data_dims2{std::min<unsigned>(csz2, H5chunk), 1};

dcpl2.add(data_dims2);
// Only apply shuffle filter when compression is enabled (guaranteed by outer if)
if (H5shuffle) dcpl2.add(HighFive::Shuffle());
if (H5szip) {
dcpl2.add(HighFive::Szip(options_mask, pixels_per_block));
Expand All @@ -5231,6 +5234,7 @@ namespace BasisClasses
HighFive::Chunking data_dims3{std::min<unsigned>(csz3, H5chunk), 1};

dcpl3.add(data_dims3);
// Only apply shuffle filter when compression is enabled (guaranteed by outer if)
if (H5shuffle) dcpl3.add(HighFive::Shuffle());
if (H5szip) {
dcpl3.add(HighFive::Szip(options_mask, pixels_per_block));
Expand Down
3 changes: 2 additions & 1 deletion exputil/EmpCylSL.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7381,10 +7381,11 @@ void EmpCylSL::WriteH5Cache()
//
auto dcpl = HighFive::DataSetCreateProps{};

if (H5compress) {
if (H5compress > 0 || H5szip) {
// Chunking
unsigned long nx = NUMX + 1, ny = NUMY + 1;
dcpl.add(HighFive::Chunking({nx, ny}));
// Only apply shuffle filter when compression is enabled (guaranteed by outer if)
if (H5shuffle) dcpl.add(HighFive::Shuffle());
if (H5szip) {
const int options_mask = H5_SZIP_NN_OPTION_MASK;
Expand Down