feat: add Jupyter and Databricks notebook parsing support#69
Merged
tirth8205 merged 11 commits intotirth8205:mainfrom Mar 31, 2026
Merged
feat: add Jupyter and Databricks notebook parsing support#69tirth8205 merged 11 commits intotirth8205:mainfrom
tirth8205 merged 11 commits intotirth8205:mainfrom
Conversation
Extract code cells from .ipynb files, filter magic/shell commands, concatenate with offset tracking, and parse as Python via tree-sitter. Supports: - Python kernel detection (phase 1) - Magic command filtering (%pip, !ls) - Cell index tracking in node.extra["cell_index"] - Cross-cell function calls and imports - Edge cases: empty notebooks, non-Python kernels, malformed JSON Includes test fixture and 12 tests in TestNotebookParsing.
Split _parse_notebook into two methods: - _parse_notebook: extracts cells from .ipynb JSON, builds list[CellInfo], delegates to _parse_notebook_cells - _parse_notebook_cells: shared method that parses cells grouped by language (Python/R via Tree-sitter, SQL via regex) Also expands supported notebook languages from Python-only to Python and R. Updates test_non_python_kernel to use an actually unsupported language (Scala) since R is now supported.
Detect and parse Databricks-exported .py notebooks (identified by the '# Databricks notebook source' header). Splits on COMMAND delimiters, classifies cells by MAGIC prefix (%sql, %r, %md, %sh), and delegates to the existing _parse_notebook_cells shared method. SQL table refs, Python functions, cross-cell calls, and cell_index tracking all work.
michael-denyer
force-pushed
the
feat/notebook-support
branch
from
ace72bb to
2da0a49
Compare
loc4atnt
pushed a commit
to loc4atnt/code-review-graph
that referenced
this pull request
Apr 9, 2026
Co-Authored-By: Michael Denyer <michael-denyer@users.noreply.github.com>
michael-denyer
added a commit
to michael-denyer/code-review-graph
that referenced
this pull request
Apr 15, 2026
The parser gated CALLS edge emission on `enclosing_func` being set, so calls made from module scope (top-level script glue, CLI entrypoints, `if __name__ == "__main__"` blocks, and Jupyter/Databricks notebook cells) produced zero CALLS edges. Any function invoked only from those contexts was flagged as dead by `find_dead_code`, even when the function was the entire reason the script existed. Notebooks are particularly affected because every cell is module-scope by definition, so the existing notebook parser (PR tirth8205#69) emitted nodes and IMPORTS_FROM edges but no CALLS edges — making the dead-code detector's notebook coverage vacuous. Fix: when `enclosing_func` is None, attribute the CALLS edge to the File node instead of dropping it. Matches the existing convention used by `_extract_value_references` and CONTAINS edges. Applied to all 5 gated emission sites: generic Python/JS/TS path, JSX components, Elixir, Solidity `emit`, and R. Downstream: `detect_entry_points` now filters File-sourced CALLS via `get_all_call_targets(include_file_sources=False)` so script-only callees remain detectable as entry points (otherwise `run_job()` called from `script.py` module scope would look "called" by `script.py` and disappear from flow analysis). Verified end-to-end against a Databricks `.ipynb` that calls `Predict.extract_data_from_sample_ids()` from cell-level code: edge count went from 0 to 14 CALLS edges, and `find_dead_code` no longer flags the method. Tests: - `test_module_scope_calls_attributed_to_file` — bare `.py` script - `test_module_scope_calls_in_notebook` — `.ipynb` file - `test_detect_entry_points_module_scope_caller_is_still_root` — flow analysis treats File-sourced CALLS correctly - `test_module_scope_caller_prevents_dead_code_flag` — end-to-end parse → store → find_dead_code - `test_if_main_block_caller_prevents_dead_code_flag` — same for `__main__` block
This was referenced Apr 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.ipynb(Jupyter/Databricks) notebook parsing — extracts functions, classes, imports, and calls from code cells across Python, SQL, R, and Scala kernels.pynotebook export parsing — detects# Databricks notebook sourceheader and splits on# COMMAND ----------markers_parse_notebook_cellsmethod handles multi-language cell dispatch with per-cell line offset trackingTest plan
.ipynbparsing with Python kernel cells.ipynbwith%python,%sql,%r,%scalamagic commands.pyexport format parsing