fix(claude): steer system prompt away from over-eager notebook creation

[pjdoland]

Summary

Closes #335. In Claude mode the agent was creating a notebook in response to question-style prompts. Canonical repro from the issue: attach package.json, ask "load this json file and summarize for me", and the agent created a notebook to hold the summary instead of replying in chat. The fix refines the system prompt, per the issue thread's consensus that a UI toggle or out-of-box skills would be more surprising than a prompt edit.

Solution

Three pieces of steering work together in _create_system_prompt:

• Soften the anchor concept: the prompt no longer opens with "JupyterLab which is an IDE for Jupyter notebooks" (tautologically primes notebook-thinking). It now names notebooks, plain file editors, and terminals as first-class surfaces.
• Add an explicit chat-default paragraph naming the common question patterns ("summarize this", "explain this", "what does this output mean", "how would I do X", "debug this", "show me what this does") and the create-case patterns ("create a notebook that...", "write me a script to...", "save this as a file", "show me a notebook that..."). Calls out the attach-a-file-and-ask manifestation that Chat UI prompts tend to create a new notebook #335 reported.
• Tighten the JUPYTER_UI_TOOLS_SYSTEM_PROMPT conditional: was "If you create a notebook or run it, save it after creating or running it" (structurally permissive, vestigial run-save tail). Now: "If the user has asked you to create a notebook, save it afterward." Reinforces the chat-default rather than softly contradicting it.

Refactored the prompt assembly out of ClaudeCodeChatParticipant._create_system_prompt into a module-level build_claude_system_prompt(jupyter_ui_tools_enabled, jupyter_root_dir) so the test suite can pin the load-bearing phrases without instantiating the chat participant or touching JupyterLab runtime state. Order is load-bearing (recency bias): the chat-default paragraph must appear before the UI-tools block; pinned by an explicit ordering test.

Testing

Eight new tests in tests/test_claude_system_prompt.py:

• The imperative core: "Default to answering questions directly in your chat reply".
• The attached-file carveout: "When the user attaches a file..." / "do not produce a new notebook to hold the answer".
• The explicit-request examples (four variants, including "show me a notebook that..." which disambiguates the "show me" verb).
• UI-tools inclusion when jupyter_ui_tools_enabled=True, exclusion when False.
• The chat-default guidance survives without UI tools (the bias is independent of the NBI MCP toolset).
• The chat-default paragraph appears BEFORE the UI-tools block (defends against future edits that reorder sections).
• Root-dir interpolation.

Substring assertions pin specific phrases so a future "tighten the prompt" rewrite is forced to update the test, surfacing the intent question.

Full suite: pytest tests/ --ignore=tests/test_claude_client.py (1063 passed), jlpm tsc --noEmit (clean), jlpm lint:check (clean), jlpm jest (259 passed). No Playwright check ran — this is a Python-only prompt edit with no JL runtime touchpoints, and the unit tests pin the contract directly.

Risks / follow-ups

• Same bias may live in notebook_intelligence/prompts.CHAT_SYSTEM_PROMPT (used by the generic + GitHub Copilot participants). Worth an empirical spot-check; port the chat-default paragraph if it repros.
• Claude Agent SDK accepts both system_prompt=str (replaces Claude Code's default) and {"type": "preset", "preset": "claude_code", "append": str} (augments CC's default). NBI passes a string, so CC's own system prompt is replaced. Switching to the preset/append form would let CC's anti-over-eager guidance reinforce ours instead of getting clobbered. Pre-existing behavior, not introduced here; larger refactor.
• Prompt brittleness: the substring assertions will fail on cosmetic rewrites. That's intentional (forces a review of whether the intent is preserved) but worth noting for the next maintainer who touches the prompt.

Issue linkage

Closes #335.

[pjdoland]

Followed up with a Playwright check against the issue's canonical repro after I realized the unit tests only pin the prompt's text, not the model's downstream behavior.

Setup: fresh build of this branch on a clean JL session; workspace contained the same scenario as the issue (a package.json plus a notebook open in the editor).

Repro: opened the NBI sidebar, attached package.json via the workspace file picker, typed "load this json file and summarize for me", sent.

Result: Claude read the file and replied in chat with a prose summary ("Project: sample-project (v1.2.3) ... It's a React 18 project written in TypeScript..."). No new notebook tab was created, and the main-area tab list stayed unchanged through 45 seconds of observation.

So the fix actually behaves the way the docstring promises, not just contains the right words.

[mbektas]

LGTM, thanks!