Add editable XML params for pending jobs #115
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The job view sits inside the project layout's Panel component, not at the viewport root. Use height: 100% to fill the parent Panel rather than 100vh which would overflow. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
The flexbox approach with height: 100% wasn't propagating correctly through the Panel component, causing double scrolling on Task Interface. Reverted to the original calc(100vh - 16rem) approach which worked. TODO: Find a better solution for variable header heights. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Changed from calc(100vh - 16rem) to calc(100vh - 20rem) to account for the project layout's MenuBar and padding that already subtract from the viewport height. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Document how to set up the baselayer compatibility layer in the main branch to enable trivial merging of pipeline/wrapper code between ccp4i2-django and main branches. Includes: - Explanation of why baselayer exists - Step-by-step setup for main branch - Code template for baselayer/__init__.py - Usage patterns for cross-branch code - Merging workflow - Comparison table of what baselayer provides in each branch 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- pipeline/ERROR_HANDLING_PATTERNS.md: CErrorReport usage, try/except patterns, ERROR_CODES dictionary best practices - pipeline/VALIDITY_PATTERNS.md: Content-aware validation using validity() method overrides for conditional requirements - cli/I2RUN_GUIDE.md: Comprehensive i2run documentation including complex data object construction (fullPath=, columnLabels=, fileUse:) - api/API_OVERVIEW.md: REST API endpoints overview and usage examples - Updated README.md and QUICK_REFERENCE.md with references to new docs 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- docs/source/conf.py: Sphinx configuration with autodoc, napoleon, autosummary extensions for generating API docs from docstrings - docs/source/index.rst: Main documentation index - docs/source/api_reference/: Auto-generated API documentation - cpluginscript.rst, cdata.rst, ccontainer.rst, cdatafile.rst - cerror.rst, base_classes.rst, fundamental_types.rst, signal_system.rst - docs/source/cli/: CLI documentation including existing i2run.rst - docs/source/api/: REST API documentation stubs - docs/source/pipeline/: Pipeline development guides - docs/source/core/: Core architecture documentation - docs/Makefile: Build commands (make html, make livehtml) - docs/requirements.txt: Sphinx dependencies Build with: cd docs && pip install -r requirements.txt && make html 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Delete dbapi/ directory (9,296 lines) - superseded by Django ORM - CCP4DbApi.py, CCP4DbUtils.py, CCP4DbMockup.py, CCP4TestDb.py - database_schema.sql, temp_database_schema.sql - stubs/dbapi.py remains as compatibility stub - Move Qt-dependent utils to legacy/utils/: - startup.py - Qt application startup - exportI2Project.py, ImportAllProjects.py, ExportAllProjects.py - importDir.py These utilities are replaced by: - CLI commands: ccp4i2 projects export/import - Django models: server/ccp4x/db/models.py 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- client/README.md: Quick start guide, technology stack overview - client/FRONTEND_DEVELOPMENT.md: Comprehensive developer guide - Architecture (Electron + Next.js dual-mode) - Directory structure - Component hierarchy - Data flow and API layer - State management (SWR, Redux, Context) - Task interface system - Report system - Styling guide - How to add new features - mddocs/README.md: Added Frontend Development section - client/package.json: Updated extraResources (dbapi -> stubs) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Added Frontend Development section with links to client/README.md and client/FRONTEND_DEVELOPMENT.md - Updated directory listing: client/ is now "Electron + Next.js/React frontend" (not "planned") 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Added links to Frontend README and Frontend Development Guide in the "Next Steps" section to improve documentation discoverability. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Picture class (CCP4ReportParser.py): - Use external sceneFile directly without copying when provided - Use deterministic filenames based on label instead of glob-based incrementing - Files now overwrite on regeneration instead of proliferating CPdbDataComposition (CCP4ModelData.py): - Use gemmi's entity_type for proper ligand classification - monomers now contains only significant ligands (NonPolymer, not water, not metal ions, ≥5 atoms) - New format: "chain:resname:seqnum" (e.g., "A:ATP:501") - Added allResidueNames for backward compatibility refmac_report.py: - Removed redundant filtering (now done at source in CPdbDataComposition) - Deterministic filenames: ligand_A_ATP_501.scene.xml - Clearer labels: "Ligand A:ATP:501" 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Add CCell.validity() to skip child validation when cell is optional and unset - Fix checkInputData() to only return errors, not warnings (warnings are for GUI display) - Fix pytest.ini pythonpath (relative to tests/i2run/ directory) - Fix REFERENCE_DATASET schema mismatch: aimless_pipe uses HKL/XYZ but pointless uses HKL_MERGED/HKL_UNMERGED/XYZ - Update aimless_pipe.def.xml with allowUndefined=True for optional cell/dataset/crystalName fields - Update CImportUnmergedStub content_qualifiers for allowUndefined 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Complete integration test that exercises the full job lifecycle via API: - Project creation - Job creation (aimless_pipe task) - Parameter list manipulation - File upload (unmerged MTZ) - Validation checking - Job execution with subprocess - Output file verification Uses file-based SQLite database fixture to enable subprocess access (pytest-django's in-memory test DB is not accessible from subprocesses). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Replace all hardcoded paths (/Applications/ccp4-9, /Users/nmemn/Developer/cdata-codegen) with dynamic path detection using CCP4I2_ROOT and relative paths - Remove DEBUG print statements from core files that were polluting job output: - hierarchy_system.py: connectSignal and children debug traces - cdata_file.py: _find_plugin_parent, _get_db_handler, getFullPath traces - ccontainer.py: copyData trace - aimless_pipe.py: pointless execution traces - Delete unused task_lookup.json (877 lines of stale absolute paths) - Create tests/shell/common.sh for centralized test environment setup - Update all shell and Python test scripts to use dynamic path detection - Clean up stale .pyc files from wrappers/cpatterson 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Backend changes for real-time refinement progress updates: - Add _startProcessWithStreaming() to CPluginScript for line-by-line stdout capture using subprocess.Popen, avoiding file-watching race conditions - Add progressUpdated signal for push-based updates from child jobs - Implement actual file watching in watchFile() using background thread polling - Add unwatchFile() and stopAllFileWatchers() for cleanup Fixes for KPI and validation: - Fix CPerformanceIndicator inheritance - all performance classes now properly inherit from CPerformanceIndicator so find_children_by_type() works - Add CSpaceGroup.fix() method for space group normalization - Fix VALIDATE_* parameter defaults to use getattr with correct def.xml defaults - Stop file watcher before validation to prevent handleXmlChanged from clearing xmlroot (race condition that was wiping Validation/B_factors elements) refmac_i2 streaming mode: - Enable streaming mode with onProcessOutput() feeding logScraper line-by-line - Emit progressUpdated signal on XML flush for parent pipeline notification - Atomic file writes (temp file + rename) to prevent read-during-write issues 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Frontend changes for improved job tracking: - Add RecentlyStartedJobsProvider with 20-second grace period tracking - Jobs marked as starting trigger fast polling even before DB status updates - useIsJobEffectivelyActive hook combines DB status with grace period - Stalled job detection warns when jobs don't start within grace period - Report view polls every 5 seconds during active/grace period API consolidation: - Add job_tree endpoint to ProjectViewSet for single-request job hierarchy - Returns jobs with embedded files, KPIs (float_values, char_values), and children - Replaces 4 separate API calls (jobs/, files/, job_float_values/, job_char_values/) - Uses prefetch_related for optimal query performance Job context menu improvements: - markJobAsStarting called when job is run to trigger grace period - Better UX with progress messages during clone/run operations Debug logging in context_run for Python interpreter discovery 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Better function naming that describes what it does rather than referencing "old" implementation. Updated across: - Function definition in i2_report.py - Imports and calls in reports.py, JobViewSet.py - Test files (test_report.py, test_report_generation.py) - Documentation files (mddocs/) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- refmac_i2: Only emit progressUpdated signal at most every 2 seconds (XML file still written on every flush for consistency) - CPluginScript: Reduce file watcher poll interval from 1s to 2s This should reduce the CPU thrashing observed during prosmart_refmac runs while still providing timely progress updates. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Key fixes: - Enable streaming XML generation in Django mode by detecting DJANGO_SETTINGS_MODULE instead of just checking PySide2 availability - Fix tab selection for queued/running jobs (now defaults to Report tab) - Use job_tree status for consistent status icons across views - Implement conditional polling for exports/active_jobs dialogs Performance improvements: - Only poll exports/ endpoint when export dialog is open - Only poll active_jobs/ endpoint when processes dialog is open - Add grace period tracking for recently started jobs Debug cleanup: - Remove DEBUG prints from i2_report.py and context_run.py - Remove debug output from prosmart_refmac_report.py - Remove fileInfo/params prints from CCP4ReportParser.py - Convert params_xml_handler.py prints to logger calls 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Fix CErrorReport.append() 'class' keyword argument issue by directly appending to _errors list (fixes 5 failing tests) - Add useProjectFiles hook with 5-second polling for task widgets - Update CDataFileElement to use polling so newly created output files appear in file dropdowns - Add Enter key support in accession code field to trigger fetch 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…uctions for post-hoc repair
The aimless_pipe and pointless wrappers use different enumerators for REFERENCE_DATASET: aimless_pipe uses HKL/XYZ while pointless uses HKL_MERGED/HKL_UNMERGED/XYZ. Previously, this parameter was not passed to pointless, causing the XYZIN command to be missing when using coordinate-based reference matching (MODE=MATCH, REFERENCE_DATASET=XYZ). This fix maps the values appropriately: - XYZ -> XYZ (compatible) - HKL -> HKL_MERGED (sensible default for merged reference data) Fixes SubstituteLigand pipeline test failure where pointless was generating an empty HKLREF line instead of the correct XYZIN command. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Error handling improvements: - Add comprehensive ERROR_CODES dictionary (201-208) for each pipeline stage - Wrap all callback handlers in try/except blocks with specific error codes - Include exception details in error messages for better debugging - Add proper return statements after failure reporting to prevent execution Aimless simplification: - Use AUTOCUTOFF=True instead of two sequential aimless runs - Remove aimlessCycle tracking and aimlessCyclesFinished method - Resolution cutoff now determined automatically in single pass Code cleanup: - Remove debug print statements from cootPlugin_finished - Remove unused os import in checkFinishStatus - Replace bare 'except:' blocks with 'except Exception as e:' - Improve error messages with source and destination paths in harvestFile 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- test_asu_contents: Add semantic XML comparison that ignores element ordering and handles source path differences (actual output stores full path in baseName, expected has filename only) - test_refmac: Skip DIFANOMFPHIOUT.mtz check if file doesn't exist, since this output is only produced when refmac outputs DELFAN/PHDELAN columns (depends on input data contentFlag) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The onClick handler was nested inside an IconButton, so only clicking the icon triggered navigation. Now the entire MenuItem is clickable and properly closes the menu after navigation. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Includes Python script to generate PowerPoint presentation documenting the CCP4i2 architecture evolution. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
When pipelines run with doAsync=False, sub-job signal handlers execute synchronously during process(). This commit ensures: 1. prosmart_refmac.startProcess() returns the actual status set by signal handlers (via self._status) instead of hardcoding SUCCEEDED 2. CPluginScript.process() now calls reportStatus() before all early returns, ensuring the 'finished' signal is emitted even when validation, processInputFiles, makeCommandAndScript, or startProcess fail. This is essential for pipelines that rely on the signal to coordinate sub-jobs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Fix isValidEndpoint to reject negative IDs while allowing 0 - Fix useProject to accept undefined/null for conditional fetching - Remove -1 fallback pattern that caused invalid API requests - Fix difference map detection: sub_type === 2 (was inverted) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* Fix Moorhen API endpoint validation and map type detection - Fix isValidEndpoint to reject negative IDs while allowing 0 - Fix useProject to accept undefined/null for conditional fetching - Remove -1 fallback pattern that caused invalid API requests - Fix difference map detection: sub_type === 2 (was inverted) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Remove fallback ID patterns that cause invalid API requests Remove all `|| 0` and `?? 0` fallback patterns when calling useProject and useJob hooks. These caused requests to non-existent /projects/0/ endpoints when the real ID wasn't available yet. Now passes undefined when ID is not available, which correctly skips the API request (SWR key becomes null). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Two issues fixed that caused servalcat_pipe CIFFILE to get .pdb extension
instead of .cif, leading to clipper/MMDB parse failures:
1. DEF file complex defaults not applied (def_xml_handler.py)
- Complex <default> elements like <contentFlag>2</contentFlag> were
only stored in metadata, never applied to object attributes
- Now properly applies dict values to object attributes
2. Old params.xml overwriting DEF defaults (params_xml_handler.py)
- When loading params.xml from legacy jobs, contentFlag=0 would
overwrite the correct DEF default (contentFlag=2)
- Now skips importing contentFlag=0 when DEF has set non-zero value
Result: CIFFILE correctly gets .cif extension based on contentFlag=2
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
- def_xml_handler.py: Apply complex DEF defaults (subType, contentFlag) to object attributes - servalcat.py: Fix DIFFPHIOUT.subType assignment (was incorrectly setting FPHIOUT twice) - servalcat_pipe.def.xml: Add guiLabel qualifiers for validation controls - splitMtz.py: Rename Phs columns to canonical names (PHI/FOM or HLA/HLB/HLC/HLD) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* Add Docker deployment assets for containerized CCP4i2 Introduces Docker configuration for deploying CCP4i2 in containerized environments, supporting both local development and Azure cloud deployment. Contents: - Server Dockerfile: Django REST API with gunicorn/uvicorn - Client Dockerfile: Next.js web frontend (multi-stage build) - docker-compose.yml: Local development orchestration with optional PostgreSQL - Azure Bicep template: Container Apps deployment with PostgreSQL and Azure Files - Azure AD authentication middleware for enterprise deployments - Comprehensive README with usage instructions Key design decisions: - CCP4 mounted at runtime (not bundled) for licensing and size - SQLite default for development, PostgreSQL for production - Flexible startup script supporting both dev and production modes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Add shell container for CCP4 setup and Azure Files documentation Adds a 'shell' service to docker-compose that provides an interactive environment matching the server container, but with read-write access to the CCP4 data volume. This enables: - Installing/building CCP4 inside the Docker environment - Setting up CCP4 on an Azure Files share for cloud deployment - Testing CCP4 commands with containerized paths Also adds documentation for mounting Azure Files locally (macOS/Linux) and using the shell container to prepare CCP4 for Docker/Azure deployment. Usage: docker compose -f Docker/docker-compose.yml --profile shell run --rm shell 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Fix Dockerfile pip install quoting and remove obsolete version - Quote package specs in pip install to prevent shell interpretation of < > characters (e.g., "django>=4.2,<5.0") - Remove obsolete 'version: 3.8' from docker-compose.yml 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Add azcopy-files.sh for fast Azure Files uploads Adds a wrapper script for uploading files to Azure Files using azcopy, which is much faster than working over SMB mounts for large directories like CCP4. Features: - Reads Azure config from Docker/.env (no secrets in script) - Auto-generates time-limited SAS tokens via Azure CLI - Supports both copy and sync modes - Falls back to account key if user delegation fails Usage: # Configure in .env AZURE_STORAGE_ACCOUNT=myaccount AZURE_RESOURCE_GROUP=mygroup # Upload CCP4 ./azcopy-files.sh /tmp/ccp4-9 ccp4-9 Also updates README with recommended azcopy workflow for CCP4 setup. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Fix Moorhen API endpoint validation and map type detection - Fix isValidEndpoint to reject negative IDs while allowing 0 - Fix useProject to accept undefined/null for conditional fetching - Remove -1 fallback pattern that caused invalid API requests - Fix difference map detection: sub_type === 2 (was inverted) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Remove fallback ID patterns that cause invalid API requests Remove all `|| 0` and `?? 0` fallback patterns when calling useProject and useJob hooks. These caused requests to non-existent /projects/0/ endpoints when the real ID wasn't available yet. Now passes undefined when ID is not available, which correctly skips the API request (SWR key becomes null). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Fix DEF defaults, servalcat subType, splitMtz column names, Docker management Server fixes: - def_xml_handler.py: Apply complex DEF defaults (subType, contentFlag) to object attributes - servalcat.py: Fix DIFFPHIOUT.subType assignment (was incorrectly setting FPHIOUT twice) - servalcat_pipe.def.xml: Add guiLabel qualifiers for validation controls - splitMtz.py: Rename Phs columns to canonical names (PHI/FOM or HLA/HLB/HLC/HLD) Client: - servalcat_pipe.tsx: UI updates for validation controls Docker: - manage.sh: Add mount/unmount commands for Azure Files - .env: Add Azure Files configuration variables 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Add Azure Container Apps deployment infrastructure - Copy and adapt bicep templates from ccp4i2-django-azure - Add CCP4_VERSION environment variable support throughout - Update DJANGO_SETTINGS_MODULE to ccp4i2.config.settings - Add deployment scripts: build-and-push, deploy-infrastructure, deploy-applications, deploy-complete - Update startup scripts to use ccp4-python and handle CCP4_VERSION - Add manage.sh Azure Files mount command with auto key retrieval - Update docker-compose.yml with CCP4_VERSION and separate projects mount - Add .gitignore to protect .env.deployment secrets 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Fix Azure Container Apps deployment issues - Add .dockerignore to reduce build context from 4.2GB to ~500MB - Fix PYTHONPATH in startup-server.sh to prioritize container packages (psycopg2-binary) over CCP4's py-packages - Make BioPython import lazy in detect_type.py to avoid numpy load at startup - Make mmcifutils import lazy in digest.py to avoid numpy load at startup - Fix migration 0012 to skip SQLite-specific SQL on PostgreSQL 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Optimize Docker builds and fix Python version compatibility - Optimize client Docker build to use client/ as build context - Remove Electron packages from web-only Docker builds using jq - Add Azure Service Bus dependencies to server Dockerfile - Fix Python 3.10+ syntax compatibility by using ccp4-20251105 - Use version-specific py-packages directories to prevent cross-version Python package conflicts - Add api-client.sh CLI tool for testing the CCP4i2 API 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Secure Django API by making server internal-only - Change server container app ingress from external to internal - Add Container Apps Environment reference to get default domain - Update web app to use internal FQDN for API communication - Server is now only accessible via web proxy, not directly from internet Security flow: Internet → Web App (public, AAD auth) → /api/proxy/* → Server (internal) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Add i2remote CLI for remote CCP4i2 API access A Python CLI that mirrors the local i2 command syntax but communicates with a remote CCP4i2 server via HTTP API. Features: - Same syntax as local i2 command (projects, jobs, files, report, export) - Job status tracking and wait-for-completion - File validation using gemmi (MTZ, PDB, mmCIF) - Configurable via environment variables or ~/.ccp4i2remote.json - Uses ccp4-python for CCP4 data awareness Enables automated testing of remote CCP4i2 deployments with commands like: i2remote jobs 1 i2remote jobs status 1 17 i2remote jobs validate 1 17 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Add fileUse syntax and file upload for CLI job parameters Features: - Add fileUse syntax for set-param to reference outputs from previous jobs Examples: [-1].XYZOUT[0], refmac[-1].HKLOUT, prosmart_refmac[-2].XYZOUT - Add upload-param command for uploading local files as job parameters - Support MTZ column selectors for reflection data uploads - Add resolve_fileuse API endpoint for remote fileUse resolution - Fix MSAL authentication for Azure AD public client flows New files: - server/ccp4i2/lib/utils/files/resolve_fileuse.py - FileUse resolution logic - server/ccp4i2/db/management/commands/upload_file_param.py - Upload command - docs/i2remote.md - CLI documentation Modified: - cli/i2remote.py - Add set-param, get-param, upload-param commands with auth fixes - server/ccp4i2/cli/i2.py - Add set-param, get-param, upload-param commands - server/ccp4i2/api/ProjectViewSet.py - Add resolve_fileuse endpoint - server/ccp4i2/db/management/commands/set_job_parameter.py - FileUse support 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Add Azure AD token injection for authenticated API requests Implement centralized auth token management for the web client: - Add auth-token.ts utility for singleton token getter pattern - Integrate MSAL token acquisition in auth-provider.tsx - Inject Bearer tokens automatically in api-fetch.ts for all API calls This enables authenticated API calls when NEXT_PUBLIC_REQUIRE_AUTH=true, with automatic silent token refresh and interactive fallback. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Add staged upload feature for large files (up to 20GB) Implements SAS URL-based uploads to bypass Azure Container Apps HTTP body size limits. Also fixes project export/import bugs: - Fix 'File' object has no attribute 'objectName' in export_project.py - Fix FileResponse for large file downloads in ProjectExportViewSet - Add StagedUpload model with status tracking workflow - Add blob_sas.py utilities for Azure SAS URL generation - Add StagedUploadViewSet with request/complete/cancel endpoints - Add i2remote CLI commands: upload-request, upload-complete, upload-status - Update Bicep infrastructure with staging-uploads blob container - Pass AZURE_STORAGE_ACCOUNT_NAME to server container 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Add StagedUpload migration 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Refactor: Move Azure staged upload code to separate Django app Moves all Azure-specific staged upload functionality out of the core ccp4i2 codebase into a separate `azure_extensions` Django app located in Docker/azure/. This keeps the core distribution clean for colleagues working on non-Azure deployments. Changes: - Create Docker/azure/azure_extensions/ Django app with: - StagedUpload model and migration - StagedUploadViewSet for /uploads/ API endpoints - blob_sas.py utilities for Azure SAS URL generation - Azure-specific settings module that extends core settings - Remove Azure-specific code from server/ccp4i2/: - StagedUploadViewSet.py - lib/utils/storage/ directory - StagedUpload model from models.py - Azure storage settings - Update server Dockerfile to: - Copy azure_extensions app into container - Use azure_extensions.settings as DJANGO_SETTINGS_MODULE - Update core urls.py to conditionally include azure_extensions URLs The azure_extensions app is only installed in Azure Container App deployments. Non-Azure deployments use the standard ccp4i2.config.settings and don't have access to the staged upload features. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Fix azure_extensions URL loading check Use settings.INSTALLED_APPS check instead of apps.is_installed() since the apps registry may not be fully initialized at URL import time. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Include azure_extensions in Docker build context The .dockerignore was excluding all of Docker/azure/ but we need the azure_extensions Django app for server builds. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Use azure_extensions.settings and fix readiness probe - Change DJANGO_SETTINGS_MODULE from ccp4i2.config.settings to azure_extensions.settings for server and worker apps - Fix readiness probe to use /health/ instead of /projects/ which returns 401 when auth is enabled 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Implement async project import via Service Bus queue - Add queue utility for sending messages to Azure Service Bus - Refactor complete_upload to queue messages instead of sync download - Add force-complete endpoint to bypass SAS URL expiry check - Add reset endpoint to retry stuck uploads - Add import_project and process_unmerged_data handlers to worker - Use system Python subprocess for blob operations (azure-storage-blob) - Add update_staged_upload management command for worker status updates - Add AZURE_STORAGE_ACCOUNT_NAME and AZURE_CLIENT_ID env vars to worker - Add upload-force-complete and upload-reset CLI commands - Fix error_message field to allow NULL values This enables uploading large files (7GB+) that would timeout with synchronous processing. The worker downloads blobs asynchronously and runs import_ccp4_project_zip without HTTP timeout constraints. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Fix export recursive inclusion and add directory polling - Move project exports to CCP4_EXPORT_FILES directory to prevent exports from including previous export files (recursive bloat) - Update ProjectViewSet, ProjectExportViewSet, and export_project.py to use new CCP4_EXPORT_FILES location - Add periodic polling (10s) to directory-viewer component to automatically refresh file listings - Add zip file validation and streaming download to worker - Various Azure deployment and auth improvements 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Fix static icon path for Docker deployment CCP4I2_ROOT was incorrectly calculated as BASE_DIR.parent.parent which resolved to /usr/src instead of /usr/src/app/ccp4i2 where the icon directories (qticons/, svgicons/) are actually located. Changed fallback to use BASE_DIR directly, which correctly points to the ccp4i2 directory containing the icons. Electron app is unaffected as it explicitly sets CCP4I2_ROOT env var. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Serve static assets from Next.js, remove WhiteNoise dependency Static icons and report files are now served directly from Next.js public/ directory instead of being proxied through Django. This eliminates the double-hop (Browser → Next.js → Django) for static assets. Changes: - Copy qticons/ and svgicons/ to renderer/public/ at build time - Update all frontend icon paths from /api/proxy/djangostatic/ to direct paths - Remove WhiteNoise from Django (settings, middleware, pyproject.toml, Dockerfile) - Update htmlBase() to return /report_files/ for Ramachandran backgrounds - Fix copyfiles -u flag to correctly strip path segments for Electron dev 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Adds the ability to directly edit input_params.xml for pending jobs via the "Params as xml" tab in dev mode. This provides fine-grained manual control over job parameters. Server changes: - Extended params_xml endpoint to accept PUT requests - Validates job is PENDING before allowing edits - Validates XML is well-formed before saving - Writes directly to input_params.xml Client changes: - Made Monaco editor editable for pending jobs - Added Save/Reset buttons with unsaved changes indicator - Shows error messages if save fails - Triggers params_xml refresh after successful save 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
input_params.xmlfor pending jobs via the "Params as xml" tab in dev modeChanges
Server (
server/ccp4i2/api/JobViewSet.py):params_xmlendpoint to accept PUT requestsinput_params.xmlClient (
client/renderer/components/job-view.tsx):params_xmlrefresh after successful saveTest plan
🤖 Generated with Claude Code