DLPX-97123 Spec 0003 Phase 1: bootstrap merge of sdimitro/sdb + dh-virtualenv overlay#360
Merged
Merged
Conversation
github: remove pull request template
Signed-off-by: Serapheim Dimitropoulos <sdimitropoulos@coreweave.com>
Signed-off-by: Serapheim Dimitropoulos <sdimitropoulos@coreweave.com>
Signed-off-by: Serapheim Dimitropoulos <sdimitropoulos@coreweave.com>
Signed-off-by: Serapheim Dimitropoulos <sdimitropoulos@coreweave.com>
Signed-off-by: Serapheim Dimitropoulos <sdimitropoulos@coreweave.com>
Signed-off-by: Serapheim Dimitropoulos <sdimitropoulos@coreweave.com>
Signed-off-by: Serapheim Dimitropoulos <sdimitropoulos@coreweave.com>
This commit introduces an MDB-style syntax compatibility layer that allows users familiar with MDB (the Modular Debugger) to use the familiar `symbol::command` syntax in SDB. The preprocessor transforms this syntax into the equivalent SDB form `addr symbol | command`. Features: - Transform `symbol::command` to `addr symbol | command` - Support for command arguments: `spa::print -nr` - Support for piped commands: `spa::walk | head 5` - Support for `::command` syntax (leading ::) which becomes just `command` - Subsequent pipes can optionally use `::cmd` which becomes `cmd` - Properly handle quoted strings containing `::` (no transformation) - Properly handle shell commands after `!` (no transformation) - New `--no-mdb-compat` CLI flag to disable the preprocessor Implementation: - New module `sdb/mdb_compat.py` with the preprocessing logic - Integration into `sdb/pipeline.py` at the start of `invoke()` - CLI argument in `sdb/internal/cli.py` to control the feature Command name validation: - Added validation in `sdb/command.py` to ensure command names follow C identifier rules (start with letter/underscore, contain only alphanumeric and underscores) - This prevents future commands from conflicting with the `::` syntax Testing: - Unit tests for the preprocessor in `tests/unit/test_mdb_compat.py` - Unit tests for command validation in `tests/unit/test_command_validation.py` - Integration tests with regression dumps in `tests/integration/test_mdb_compat_generic.py` - Positive test cases: basic syntax, with arguments, with pipes, kernel variables - Negative test cases: multiple `::`, triple `:::`, invalid symbols, empty commands Example usage: # MDB-style (new) spa_namespace_avl::walk | head 5 jiffies::deref 0xffff1234::print -nr # Equivalent SDB-style (existing) addr spa_namespace_avl | walk | head 5 addr jiffies | deref addr 0xffff1234 | print -nr Note: This module only emulates the `symbol::command` syntax. It does NOT emulate other MDB-specific features such as number format conversions, expression evaluation, variable assignment, or macros. Signed-off-by: Serapheim Dimitropoulos <sdimitropoulos@coreweave.com> Signed-off-by: Claude (Anthropic AI Assistant)
Signed-off-by: Serapheim Dimitropoulos <sdimitropoulos@coreweave.com>
…le it Signed-off-by: Serapheim Dimitropoulos <sdimitropoulos@coreweave.com>
Signed-off-by: Serapheim Dimitropoulos <sdimitropoulos@coreweave.com>
The workflow triggered on push to '**' (all branches) AND on pull_request, causing duplicate CI runs for every push to a PR branch. Restrict push triggers to main and develop only; feature branch pushes are covered by the pull_request trigger. Made-with: Cursor
prakashsurya
force-pushed
the
projects/spec-0003-phase-1
branch
from
62f7bfd to
6265bbb
Compare
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #360 +/- ##
===========================================
- Coverage 85.12% 76.64% -8.48%
===========================================
Files 67 73 +6
Lines 3059 4556 +1497
Branches 0 728 +728
===========================================
+ Hits 2604 3492 +888
- Misses 455 878 +423
- Partials 0 186 +186 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
prakashsurya
force-pushed
the
projects/spec-0003-phase-1
branch
4 times, most recently
from
a3f65dc to
c778619
Compare
prakashsurya
changed the title
mmaybee
approved these changes
May 6, 2026
prakashsurya
force-pushed
the
projects/spec-0003-phase-1
branch
2 times, most recently
from
e7beaee to
9bc5711
Compare
The bootstrap merge of sdimitro/sdb wiped this file because it doesn't exist upstream — `.whitesource` is Delphix-internal infrastructure that configures Mend (formerly WhiteSource) license/SCA/IaC scanning policy for the repo. It needs to live on `develop` so the security pipeline keeps scanning the package after Phase 1 lands. Recovered verbatim from the previous `develop` tree (pre-bootstrap); no content changes. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The bootstrap merge of sdimitro/sdb wiped `.github/pull_request_template.md` because it doesn't exist upstream. The Delphix template is the standard "Problem / Solution / [Notes to Reviewers]" prompt used across every Delphix repo's PRs; restoring it keeps the PR-authoring experience consistent now that this repo lives in the Delphix GitHub namespace and is reviewed under Delphix's review conventions. Recovered verbatim from the previous `develop` tree (pre-bootstrap); no content changes. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
After Phase 1 of Spec 0003 lands, `delphix/sdb` is a packaging-only downstream of `sdimitro/sdb`: `develop` tracks upstream tree-identically plus a small overlay, and functional changes belong upstream. Anyone landing on this repo's GitHub page from a search result, a support ticket, or a stale link needs to know that without first reading the spec — surface the relationship in the README header so the upstream becomes the obvious contribution target. The notice specifically enumerates which paths make up the overlay (`.whitesource`, `.github/pull_request_template.md`, the notice itself, and `debian/`) so reviewers reading the diff understand its intentional bounds. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
prakashsurya
force-pushed
the
projects/spec-0003-phase-1
branch
2 times, most recently
from
613dc0b to
00e580f
Compare
This is the meat of Spec 0003 Phase 1: turn the `sdb` `.deb` into a
self-contained `dh-virtualenv` build that bundles drgn, kdumpling,
pyelftools, and all transitive Python deps inside a venv at
`/opt/venvs/sdb/`, instead of relying on system-installed drgn /
libkdumpfile (which previously required maintaining the `delphix/drgn`
and `delphix/libkdumpfile` Delphix forks just to keep
ABI-compatible versions pinned to whatever sdb expected).
The `/opt/venvs/<pkg>` install root is dh-virtualenv 1.2+'s default,
which the project deliberately moved to (from `/usr/share/python/<pkg>`)
to avoid clashes with Debian-shipped Python tooling living under
`/usr/share/python/`. It's also the FHS 3.0-correct home for an
add-on application package (FHS §3.13: `/opt` is "reserved for the
installation of add-on application software packages" with
self-contained per-package directory layout) — exactly matching what
a self-contained venv bundle is. The previous `delphix/sdb` `debian/`
referenced `/usr/share/python/sdb/` from a stale legacy convention
predating the dh-virtualenv 1.0 default change; this commit also
fixes that reference in `debian/control`'s package description.
Why the venv approach works at all: drgn's manylinux PyPI wheel is
fully self-contained — `drgn.libs/libkdumpfile.so.12` ships inside
the wheel and is loaded via auditwheel-injected
`RPATH=$ORIGIN/drgn.libs`. So there's nothing to link against on the
system; the venv carries its own ABI-pinned copy. After this lands,
`sdb` no longer consumes system drgn at all (verified during Phase 1
acceptance: `sudo dpkg -r --force-depends drgn` and `sdb stacks`
against a vmcore still works, identical output, exit 0).
Highlights of the diff:
- `debian/rules` switches to the `dh-virtualenv` sequencer
(`dh $@ --with python_virtualenv`) plus several hygiene overrides:
- `override_dh_dwz:` — drgn's manylinux .so files in `drgn.libs/`
arrive pre-stripped, with most having no `.debug_info` sections.
`dh_dwz` would fail with "Too few files for multifile
optimization." Skip dwz entirely for this venv-only package.
- `override_dh_strip:` — same reason; further stripping the
bundled libs produces no real savings and risks breaking
auditwheel's RPATH setup.
- `override_dh_shlibdeps:` — drgn's bundled libs use $ORIGIN
RPATH and aren't tracked by dpkg, so dpkg-shlibdeps can't
resolve them. ${python3:Depends} (filled by dh_python3) covers
the Python runtime requirement; the venv handles the rest.
- `override_dh_python3:` — dh_python3 doesn't understand the venv
layout dh_virtualenv produces and complains about unmanaged
files. dh-virtualenv already wires the right Python deps.
- `override_dh_md5sums: dh_md5sums -X.pyc -X.pyo` — Python rewrites
.pyc files at runtime when a bytecode header's recorded source
mtime/size doesn't match the .py on disk (unavoidable for
dh-virtualenv-built venvs, since build-time path/mtime rewrites
desync the .pyc headers). The first import after install
regenerates the affected .pyc, breaking the recorded md5 and
triggering false positives in dcenter's `debsums -c`. Excluding
bytecode keeps the integrity check meaningful — .py source and
native libs still get checksummed.
- `debian/control` Build-Depends adds `dh-virtualenv (>= 1.2.2)`;
runtime Depends carries no explicit `libkdumpfile` (drgn's wheel
handles it) — only ${python3:Depends} plus an empty
${shlibs:Depends}.
- `debian/links` creates `/usr/bin/sdb -> /opt/venvs/sdb/bin/sdb` so
`sdb` is on the standard path despite the venv living under
`/opt`.
- `debian/source/format` switches to `3.0 (native)`; `debian/compat`
is removed (the modern `debhelper-compat (= 13)` virtual-package
approach in Build-Depends is preferred).
- `debian/changelog` opens at version `0.6.0` (matching the upstream
tag we boot from); linux-pkg appends `-1delphix.<ts>` at build
time so successive builds are uniquely ordered for upgrades.
This indirectly unblocks Phase 3 of Spec 0003 (decommissioning
`delphix/drgn` and `delphix/libkdumpfile`), although Phase 3 is
deferred — other appliance consumers (`crash-python`, `savedump`,
`/usr/bin/drgn`, `delphix-platform-aws`'s diagnostics dep set) still
need the system packages today.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Phase 2 of Spec 0003 will wire `linux-pkg`'s `packages/sdb/config.sh` to derive `PACKAGE_VERSION` from `setuptools_scm` (running on the build host, before dh-virtualenv enters the venv) so the .deb metadata version tracks upstream sdb tags automatically and `debian/changelog`'s upstream-version prefix never goes stale across upstream syncs. That host-side `python3 -m setuptools_scm` invocation needs the package present on the build host. Stage the Build-Depends here so the Phase 2 linux-pkg change is a single-file flip — and so any manual local builds with the new config.sh don't fail with "setuptools_scm not found" after `install_build_deps_from_control_file`. The dependency is dormant in Phase 1 — nothing in `debian/rules` or the dh-virtualenv build path uses it; the inner sdb wheel build fetches its own copy via pyproject.toml's [build-system].requires. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
prakashsurya
force-pushed
the
projects/spec-0003-phase-1
branch
from
00e580f to
3a78e6b
Compare
sebroy
approved these changes
May 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
Phase 1 of Spec 0003. Pivot
delphix/sdbto a packaging-only downstream ofsdimitro/sdband rebuild the.debviadh-virtualenv.After this lands:
developis tree-identical tosdimitro/developplus a 4-path Delphix overlay (.whitesource,.github/pull_request_template.md,README.mdnotice,debian/)..debis a self-contained venv at/opt/venvs/sdb/— no runtime dependency on systemdrgnorlibkdumpfile.linux-pkg packages/sdb/config.shis unchanged. Phase 2 (separate PR, againstlinux-pkg) wires upstream auto-tracking.delphix/sdbcarries packaging deltas only.Status: All Phase 1 ACs green on a fresh
dlpx-developVM (2026-05-06, branch tip3a78e6b).debsums -c sdbclean. Detailed evidence in the Phase 1 verification log (currently in delphix/cd-aidlc#36).Appliance pipeline cross-check (F1): Jenkins build 13965.
Commits
Each commit is independently reviewable; commit-by-commit is fine. Commit messages carry the per-commit rationale — this list is just a map.
fe814fasdimitro/develop(tree-identical via-s ours+read-tree --reset)118a3de.whitesource(Mend SAST scanning config; downstream-only)cd6fe75a4f9323README.md42cfef8debian/with dh-virtualenv packaging — the meat. See commit body for the override-by-override rationale.3a78e6bpython3-setuptools-scmin Build-Depends so Phase 2 is a single-repo changeWhat we tested
A reviewer's job, beyond reading the diff, is to confirm the test strategy below is sufficient — i.e. that something would actually break a test if these design claims regressed. Raw build/install/runtime output for each item is in the verification log.
Tree-shape claims (deterministic; no VM needed; reviewer can re-run locally):
sdimitro/develop; overlay = exactly the 4 expected path groups (no scope creep)Build claims (require VM):
linux-pkg buildpkg.sh -b projects/spec-0003-phase-1 sdbproduces exactly one.deb, withpackages/sdb/config.shunchangedInstall / runtime / integrity claims (require VM; covered in Run 1 of the log):
apt install --allow-downgradeson Ubuntu 24.04sdb --version→0.1.dev1 (g3a78e6b50)(setuptools_scm-derived from branch tip)dpkg --verify sdbclean;debsums -c sdbclean before AND after.pycregeneration (the test that justifies theoverride_dh_md5sums -X.pyc -X.pyooverride; this was the dcenter image-sync failure mode that prompted the override)The central design claim (most important — see verification log §"Library isolation" for full evidence):
(design)— aftersudo dpkg -r --force-depends drgn(systemdrgnremoved), the venv'simport drgnstill works and resolves to/opt/venvs/sdb/lib/python3.12/site-packages/drgn/__init__.py.ldd _drgn.soshows every drgn-related native (libkdumpfile, libaddrxlat, libdw, libelf, libdebuginfod, etc.) resolving underdrgn.libs/via auditwheel-injectedRPATH=$ORIGIN/drgn.libs. Conclusive proof thatsdbis not silently linking against system drgn.Not re-verified in Run 1 (with reasoning):
upstreams/developpublished-on-origin) — gated ondelphix-devops-botpush permissions; deferred per the spec's Notes from Phase 1 execution.sdb stacksagainst a kdump vmcore — runtime path unaffected by overlay changes since the original verification run; the library-isolation test above already exercises the same code path.apt upgrade sdb) — requires two distinct builds in a local APT repo; the upgrade mechanic is format-agnostic (only requires monotonically-ordered version strings, which+delphix.<ts>guarantees).If you'd like any of these re-run before merge, say so.
References