fix: consolidated memory and storage improvements

[echobt]

Summary

This PR consolidates 6 memory and storage fixes into a single, cohesive change.

Included PRs:

• fix(common): add cleanup for stale file locks to prevent memory leak #44: Add cleanup for stale file locks to prevent memory leak
• fix(engine): add cache size limits to prevent unbounded memory growth #45: Add cache size limits to prevent unbounded memory growth
• fix(storage): add fsync after file writes to prevent data loss #47: Add fsync after file writes to prevent data loss
• fix(engine): bound ToolResponseStore size and cleanup consumed entries #50: Bound ToolResponseStore size and cleanup consumed entries
• fix(server): eliminate TOCTOU races in MCP server and plugin registry #51: Eliminate TOCTOU races in MCP server and plugin registry
• fix(common): improve path validation and tilde expansion #52: Improve path validation and tilde expansion

Key Changes:

• Added periodic cleanup of stale file locks to prevent memory leaks
• Implemented LRU cache limits for config discovery and tokenizer caches
• Added fsync calls after critical file writes for data durability
• Created bounded ToolResponseStore with configurable limits and automatic cleanup
• Fixed time-of-check-time-of-use (TOCTOU) races in MCP server and plugin registry
• Improved path validation with secure tilde expansion

Files Modified:

• src/cortex-common/src/file_locking.rs
• src/cortex-resume/src/session_store.rs
• src/cortex-engine/src/config/config_discovery.rs
• src/cortex-engine/src/tokenizer.rs
• src/cortex-storage/src/sessions/storage.rs
• src/cortex-tui/src/session/storage.rs
• src/cortex-engine/src/tools/response_store.rs (new)
• src/cortex-cli/src/utils/paths.rs
• src/cortex-engine/src/validation.rs
• src/cortex-mcp-server/src/server.rs
• src/cortex-plugins/src/registry.rs

Closes #44, #45, #47, #50, #51, #52

[greptile-apps]

Greptile Overview

Greptile Summary

This PR consolidates 6 critical memory and storage improvements into a cohesive change that significantly enhances system reliability and security.

Key Improvements

Memory Leak Prevention

• File lock cleanup: FileLockManager now automatically removes stale lock entries when reaching 10,000 entries, preventing unbounded growth from accumulated lock references
• Cache limits: Config discovery and tokenizer caches now enforce 1,000-entry limits with simple eviction, preventing memory exhaustion in long-running processes

Data Durability

• fsync integration: All critical file writes now call sync_all() to ensure data reaches disk before continuing, preventing data loss on crash or forceful termination
• Atomic writes: Session stores use write-to-temp-then-rename pattern with fsync for crash-safe updates
• Directory sync: Unix systems also sync parent directories after file operations to ensure directory entries are persisted

Unbounded Growth Prevention

• ToolResponseStore: New bounded storage (500 entries max) with TTL-based cleanup (5 min), automatic eviction, and statistics tracking prevents unbounded accumulation of tool execution results

Security Hardening

• TOCTOU race fixes:
  • MCP server initialization now uses atomic check-and-set within single write lock
  • Plugin registration uses Entry API for atomic check-and-insert
  • Both changes prevent race conditions where concurrent operations could bypass validation
• Path validation: Fixed to correctly use path components instead of string matching, preventing false positives on filenames containing ".."
• Command validation: Added normalization to prevent bypass attacks via extra whitespace, quotes, or path variants (e.g., /bin/rm vs rm)

Code Quality

• Comprehensive test coverage added for all new validation logic
• Clear documentation of security considerations and design decisions
• Consistent error handling patterns across all changes

Confidence Score: 5/5

• This PR is safe to merge with high confidence - all changes are defensive improvements that enhance reliability and security
• All changes are well-tested defensive improvements with clear documentation. The consolidated PR successfully addresses multiple memory and storage issues without introducing breaking changes. TOCTOU fixes use standard Rust patterns, fsync additions follow best practices for durability, and cache limits prevent unbounded growth. No logic errors or security vulnerabilities detected.
• No files require special attention - all changes are clean and well-implemented

Important Files Changed

Filename	Overview
src/cortex-common/src/file_locking.rs	Added automatic cleanup of stale file locks (MAX_LOCK_ENTRIES=10,000) to prevent memory leaks in FileLockManager
src/cortex-engine/src/tools/response_store.rs	New bounded storage for tool responses with configurable limits (MAX_STORE_SIZE=500), TTL, automatic cleanup, and statistics tracking
src/cortex-cli/src/utils/paths.rs	Improved path validation using component analysis to correctly distinguish traversal from filenames containing "..", added comprehensive test coverage
src/cortex-engine/src/validation.rs	Added command normalization to prevent bypass attacks via whitespace, quotes, and path variants
src/cortex-mcp-server/src/server.rs	Fixed TOCTOU race in handle_initialize() by holding write lock during entire check-and-transition operation
src/cortex-plugins/src/registry.rs	Fixed TOCTOU race in plugin registration using Entry API for atomic check-and-insert operation

Sequence Diagram

sequenceDiagram
    participant App as Application
    participant Cache as Config/Token Cache
    participant LockMgr as FileLockManager
    participant Store as ToolResponseStore
    participant FS as FileSystem
    participant MCP as MCP Server
    participant Plugin as Plugin Registry

    Note over Cache,Store: Memory Management Flow
    
    App->>Cache: Request config/token
    Cache->>Cache: Check if size >= 1000
    alt Cache Full
        Cache->>Cache: Evict first entry
    end
    Cache->>Cache: Insert new entry
    Cache-->>App: Return result

    App->>LockMgr: get_lock(path)
    LockMgr->>LockMgr: Check if locks >= 10,000
    alt Too many locks
        LockMgr->>LockMgr: cleanup_stale_entries()
        Note over LockMgr: Remove entries with<br/>Arc::strong_count == 1
    end
    LockMgr-->>App: Return Arc<Mutex>

    App->>Store: store(call_id, result)
    Store->>Store: Check if size >= 500
    alt Store Full
        Store->>Store: Evict oldest entry
    end
    Store->>Store: Insert new response
    Store-->>App: Return evicted status

    Note over App,FS: Data Durability Flow

    App->>FS: Write session data
    FS->>FS: Write to temp file
    FS->>FS: flush()
    FS->>FS: sync_all() [fsync]
    FS->>FS: Atomic rename
    alt Unix
        FS->>FS: Sync parent directory
    end
    FS-->>App: Write complete

    Note over MCP,Plugin: TOCTOU Race Prevention

    App->>MCP: initialize()
    MCP->>MCP: Acquire write lock
    MCP->>MCP: Check state == Uninitialized
    alt Already initialized
        MCP-->>App: Error: Already initialized
    else Not initialized
        MCP->>MCP: Set state = Initializing
        MCP->>MCP: Release lock
        MCP->>MCP: Complete initialization
        MCP-->>App: Success
    end

    App->>Plugin: register_plugin(id)
    Plugin->>Plugin: Acquire write lock
    Plugin->>Plugin: Check entry exists
    alt Already exists
        Plugin-->>App: Error: Already exists
    else New entry
        Plugin->>Plugin: Insert via Entry API
        Plugin->>Plugin: Release lock
        Plugin-->>App: Success
    end

Loading

[greptile-apps]

_{6 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

[echobt]

Closing this PR to consolidate into a single mega-PR combining all bug fixes. The changes will be included in a new consolidated PR.