Skip to content

feat: changed datacrumbs generation based logic to dynamic probe attach.#179

Open
hariharan-devarajan wants to merge 1 commit intodevelopfrom
checkpoint/04-13-26
Open

feat: changed datacrumbs generation based logic to dynamic probe attach.#179
hariharan-devarajan wants to merge 1 commit intodevelopfrom
checkpoint/04-13-26

Conversation

@hariharan-devarajan
Copy link
Copy Markdown
Member

@hariharan-devarajan hariharan-devarajan commented Apr 13, 2026

  1. Reduces the footprint of sudo review code.
  2. Removes unneccessary scripts only support systemd in sudo.
  3. Creates a system configuration and secrete to sign user probes.
  4. No predefined probes, everything is a probe submitted by user.
  5. Validation of probe is on runtime and we maintain invalid list.
  6. Updated documentation.
  7. No MPI dependency.
  8. All other features moved to utils.
  9. Composability replaced with user input of probes.
  10. Include utils part of build for ease of deployment no direct dependency.
  11. Updated the RuntimeConfigurationManager to utilize an SQLite database for managing runtime probe states, replacing the previous JSON-based approach.
  12. Introduced new methods for loading and persisting runtime probe states, including successful and invalid probes.
  13. Modified the attach_runtime_probes function to record successful and invalid probes in the new database.
  14. Enhanced error handling and logging for database operations.
  15. Removed JSON dependencies in favor of SQLite for system configuration management in SystemConfigurator.
  16. Added new utility functions for executing SQLite commands and writing key-value pairs to the database.
  17. Updated related header files to reflect changes in data structures and method signatures.

The flow for the new changes are as follows:

  • The system configuration is done by system admin in user-space and they create an install folder for datacrumbs.
  • The administrator will deploy datacrumbs
    • deploy signing service
    • deploy datacrumbs service as a part of prolog and epilog
    • add flux args plugin for allocation
  • User will clone datacrumbs-utils and create probe lists and sign them using the signing service
  • Then during allocation like flux alloc users can enable datacrumsbs and pass the probes file to the allocation.
  • Datacrumbs on load will verify the probes using the signing key and then dynamically attach every probe in the system using the bpf APIs directly.
  • The user use datacrumsb_client_so to allow datacrumbs to track which processes should be traced/profiled.
  • On epilog, it detaches the probes , creates the performance events, and exits.

@hariharan-devarajan hariharan-devarajan requested a review from a team as a code owner April 13, 2026 23:32
1. Reduces the footprint of sudo review code.
2. Removes unneccessary scripts only support systemd in sudo.
3. Creates a system configuration and secrete to sign user probes.
4. No predefined probes, everything is a probe submitted by user.
5. Validation of probe is on runtime and we maintain invalid list.
6. Updated documentation.
7. No MPI dependency.
8. All other features moved to utils.
9. Composability replaced with user input of probes.
10. Include utils part of build for ease of deployment no direct
    dependency.
11. Updated the RuntimeConfigurationManager to utilize an SQLite database for managing runtime probe states, replacing the previous JSON-based approach.
12. Introduced new methods for loading and persisting runtime probe states, including successful and invalid probes.
13. Modified the attach_runtime_probes function to record successful and invalid probes in the new database.
14. Enhanced error handling and logging for database operations.
15. Removed JSON dependencies in favor of SQLite for system configuration management in SystemConfigurator.
16. Added new utility functions for executing SQLite commands and writing key-value pairs to the database.
17. Updated related header files to reflect changes in data structures and method signatures.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant