Skip to content

Optional debugging output for AbstractIOHandlerImpl::flush()#1495

Merged
ax3l merged 2 commits intoopenPMD:devfrom
franzpoeschel:easier-backend-debugging
Aug 10, 2023
Merged

Optional debugging output for AbstractIOHandlerImpl::flush()#1495
ax3l merged 2 commits intoopenPMD:devfrom
franzpoeschel:easier-backend-debugging

Conversation

@franzpoeschel
Copy link
Contributor

@franzpoeschel franzpoeschel commented Aug 7, 2023

I've used something like this probably a hundred times already, this PR puts debugging output (disabled by default) for the backend IO tasks into AbstractIOHandlerImpl::flush().

To be enabled by manually putting a #define OPENPMD_DEBUG 1 in the corresponding .cpp file.
This can be seen as a first step toward proper debug logging that users can enable e.g. via an environment variable.

Sample output:

Filters: write_test                                                                                 [1185/1896]
[0->0xe1e938] CREATE_FILE: serial_write
[0xe1e938->0xe1eb08] CREATE_PATH: /data/
[0xe1eb08->0xe32088] CREATE_PATH: 1
[0->0xe1e938] WRITE_ATT: (STRING) particlesPath
[0xe32088->0xe31fb8] CREATE_PATH: particles/
[0xe31fb8->0xe16f88] CREATE_PATH: e
[0xe16f88->0xe2e768] CREATE_PATH: position
[0xe2e768->0xe2d598] CREATE_DATASET: x
[0xe2e768->0xe2d598] WRITE_DATASET
[0xe2e768->0xe2d598] WRITE_DATASET
[0xe2e768->0xe2d598] WRITE_DATASET
[0xe2e768->0xe2d598] WRITE_DATASET
[0xe2e768->0xe2d598] WRITE_ATT: (DOUBLE) unitSI
[0xe16f88->0xe2e768] WRITE_ATT: (FLOAT) timeOffset
[0xe16f88->0xe2e768] WRITE_ATT: (ARR_DBL_7) unitDimension
[0xe16f88->0xe2df78] CREATE_PATH: positionOffset
[0xe2df78->0xe2fa18] CREATE_DATASET: x
[0xe2df78->0xe2fa18] WRITE_DATASET
[0xe2df78->0xe2fa18] WRITE_DATASET
[0xe2df78->0xe2fa18] WRITE_DATASET
[0xe2df78->0xe2fa18] WRITE_DATASET
[0xe2df78->0xe2fa18] WRITE_ATT: (DOUBLE) unitSI
[0xe16f88->0xe2df78] WRITE_ATT: (FLOAT) timeOffset
[0xe16f88->0xe2df78] WRITE_ATT: (ARR_DBL_7) unitDimension
[0xe1eb08->0xe32088] WRITE_ATT: (DOUBLE) dt
[0xe1eb08->0xe32088] WRITE_ATT: (DOUBLE) time
[0xe1eb08->0xe32088] WRITE_ATT: (DOUBLE) timeUnitSI
[0xe1eb08->0xe1b8a8] CREATE_PATH: 2
[0xe1b8a8->0xe30498] CREATE_PATH: particles/
[0xe30498->0xe1a6b8] CREATE_PATH: e
[0xe1a6b8->0xe2f0d8] CREATE_PATH: position
[0xe2f0d8->0xe2f3a8] CREATE_DATASET: x
[0xe2f0d8->0xe2f3a8] WRITE_DATASET
[0xe2f0d8->0xe2f3a8] WRITE_DATASET
[0xe2f0d8->0xe2f3a8] WRITE_DATASET
[0xe2f0d8->0xe2f3a8] WRITE_DATASET
[0xe2f0d8->0xe2f3a8] WRITE_ATT: (DOUBLE) unitSI
[0xe1a6b8->0xe2f0d8] WRITE_ATT: (FLOAT) timeOffset
[0xe1a6b8->0xe2f0d8] WRITE_ATT: (ARR_DBL_7) unitDimension
[0xe1a6b8->0xe32498] CREATE_PATH: positionOffset
[0xe32498->0xe32778] CREATE_DATASET: x
[0xe32498->0xe32778] WRITE_DATASET
[0xe32498->0xe32778] WRITE_DATASET
[0xe32498->0xe32778] WRITE_DATASET
[0xe32498->0xe32778] WRITE_DATASET
[0xe32498->0xe32778] WRITE_ATT: (DOUBLE) unitSI
[0xe1a6b8->0xe32498] WRITE_ATT: (FLOAT) timeOffset
[0xe1a6b8->0xe32498] WRITE_ATT: (ARR_DBL_7) unitDimension

@franzpoeschel franzpoeschel force-pushed the easier-backend-debugging branch from 35e8534 to 338e922 Compare August 7, 2023 13:54
franzpoeschel
franzpoeschel commented Aug 7, 2023
View reviewed changes
github-advanced-security[bot]
github-advanced-security bot found potential problems Aug 7, 2023
View reviewed changes
src/IO/AbstractIOHandlerImpl.cpp
#endif
}

std::future<void> AbstractIOHandlerImpl::flush()

Check warning

Code scanning / CodeQL

Poorly documented large function

Poorly documented function: fewer than 2% comments for a function of 328 lines.
@ax3l ax3l self-requested a review August 9, 2023 22:10
@ax3l ax3l self-assigned this Aug 9, 2023
ax3l
ax3l approved these changes Aug 9, 2023
View reviewed changes
Copy link
Member

@ax3l ax3l left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great!

Since it is not performance critical, one could also consider controlling this with an environment option (verbosity).

That way, no recompile needs to be done.

@franzpoeschel franzpoeschel force-pushed the easier-backend-debugging branch from 338e922 to 4e1535b Compare August 10, 2023 09:40
@franzpoeschel
Copy link
Contributor Author

franzpoeschel commented Aug 10, 2023

This is great!

Since it is not performance critical, one could also consider controlling this with an environment option (verbosity).

That way, no recompile needs to be done.

I had not added that because of our plans to introduce proper logging at some point, but adding an environment variable is probably a good idea. This way, we can easily ask users to give us more details in bug reports without recompiling anything.
In that case, I would maybe even include this in 0.15.2?

@franzpoeschel franzpoeschel added this to the 0.15.2 milestone Aug 10, 2023
ax3l
ax3l approved these changes Aug 10, 2023
View reviewed changes
Copy link
Member

@ax3l ax3l left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent, thank you!

@ax3l ax3l merged commit 0387b6f into openPMD:dev Aug 10, 2023
eschnett added a commit to eschnett/openPMD-api that referenced this pull request Sep 5, 2023
@eschnett
Merge branch 'dev' into eschnett/julia-bindings
724806e 
* dev:
  Fix CMake: HDF5 Libs are PUBLIC (openPMD#1520)
  Fix `chmod` in `download_samples.sh` (openPMD#1518)
  CI: Old CTest (openPMD#1519)
  Python: Fix ODR Violation (openPMD#1521)
  replace extent in weighting and displacement (openPMD#1510)
  CMake: Warn and Continue on Empty HDF5_VERSION (openPMD#1512)
  Replace openPMD_Datatypes global with function (openPMD#1509)
  Streaming examples: Set WAN as default transport (openPMD#1511)
  TOML Backend (openPMD#1436)
  make it possible to manually set chunks when loading dask arrays (openPMD#1477)
  [pre-commit.ci] pre-commit autoupdate (openPMD#1504)
  Optional debugging output for AbstractIOHandlerImpl::flush() (openPMD#1495)
  Python: 3.8+ (openPMD#1502)

# Conflicts:
#	.github/workflows/linux.yml
#	src/binding/python/Series.cpp
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ax3l ax3l ax3l approved these changes

Assignees

@ax3l ax3l

Labels

internal

Projects

None yet

Milestone

0.15.2

Development

Successfully merging this pull request may close these issues.

2 participants

@franzpoeschel @ax3l