From db7d7bb6b143f64b2acd9b2fd2bb111610f05d27 Mon Sep 17 00:00:00 2001 From: David Gornshtein Date: Thu, 9 Apr 2026 03:09:29 +0300 Subject: [PATCH 01/14] perf: async analytics/leaderboard + SQLite FTS5 + live partial results MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Stops the GUI from hanging on users with large session histories (4800+ codex rollouts / 1 GB+ JSONL). Tested on a 4873-session corpus where cold Analytics previously took 100+ seconds to respond. ## Core changes - **Non-blocking loadSessions** — sync path reads metadata only, uses the parse/cost disk caches, and queues uncached files for a background warmer. Cold `/api/sessions` now returns in ~300ms instead of blocking for 100+ seconds. - **Async background jobs** for `/api/analytics/cost` and `/api/leaderboard`. HTTP returns a `{status, progress, partialResult}` snapshot immediately; the client polls at 500ms and the job publishes a live partial aggregate on each chunk so users see real numbers climb ($0 → $5562 → $11344 → ... → final). - **Incremental cost aggregator** (`createCostAggregator`) — extracted from `getCostAnalytics` so the job can merge sessions one chunk at a time and finalize a snapshot per yield. - **SQLite + FTS5 index** (`src/sqlite-index.js`) at `~/.codedash/cache/index.sqlite` — persistent sessions/messages/ messages_fts (FTS5 porter+unicode61)/daily_stats/files_seen/ aggregate_cache tables. Full-text search runs via `MATCH` in ~5 ms on 1.2M messages. Uses async `spawn` with `-cmd .timeout 30000` so concurrent writers don't deadlock. - **Aggregate result cache** persisted in `aggregate_cache` with a 5-min quantized fingerprint (`count|max_ts_bucket|filters|helpers`). Repeat visits within the bucket are instant; active codex writes don't invalidate the cache on every request. ## Hot-path fixes found along the way - `getActiveSessions` ran `lsof` per matching process with a 2s timeout and called `loadSessions()` inside the loop → blocked the event loop for minutes when 30+ codex-up wrappers were running. Now: single `ps` call, tight regex (catches `codex`/`codex-up`/ `codex-up-exec` as binary names), batched `lsof -a -d cwd -Fpn -p` (the `-a` flag is critical; without it lsof ORs conditions and returns cwds for unrelated processes), pid→cwd cache, 3s result cache, no inner loadSessions. 80s → 350ms cold, 0ms cached. - `resolveGitRoot` called `git rev-parse` synchronously per unique project path — 56 projects × 2s = 112s of blocked event loop. Now queued to a background resolver with its own disk cache in `~/.codedash/cache/git-root-cache.json`. - `scanCodexSessions` was O(n²) via `.find()` on an array, and re-read every rollout file on each call. Now: `Map` + `cacheOnly` parse mode + background warmer drains uncached files. - `parseClaudeSessionFileAsync` — streaming read via `readline` for files >5 MB (user had a 199 MB session file). Yields to the event loop every 2000 lines via `await new Promise(r => setImmediate(r))` so HTTP requests aren't starved during parse. - Persistent cache paths moved from `os.tmpdir()` to `~/.codedash/cache/` so macOS tmpdir cleanup doesn't wipe a morning's worth of parse work. - Flush handlers on SIGINT/SIGTERM + periodic flush every 50 entries — killing the server mid-warm no longer loses hours of progress. ## Accuracy fixes - `parseClaudeSessionFile` counted every `type=user` entry as a user prompt, but Claude Code stores tool_results as `type=user` with `content: [{type:'tool_result', ...}]`. One measured session had 480 type=user entries but only 17 real user prompts — a 28x overcount. Now checks `content` for a real `text` block. - `isSystemMessage` extended to skip Codex runtime injections that were counted as user prompts: ``, ``, ``, ``, `# CLAUDE.md`, `Warning: The maximum number of unified exec`, `AUTOSTEERING:`, `[Sub-agent results]`. - **Helper session detection** — codex rollouts with `session_meta.payload.originator === 'codex_exec'` (or scripted first-message patterns like `You are in /...`, `Read-only task.`, `Work in /...`, `Pair-local ... lane`, `## X Y Agent`, etc) are flagged `is_helper: true`. `/api/sessions` filters them by default; `?include_helpers=1` opts in. On the test corpus this removes 2166/4873 scripted sub-agent runs. - Leaderboard now counts **unique conversations** (via `group_key`), not retries. Real cost is summed over all rollouts (actual money spent); session count uses deduped groups. On the test corpus: 2869 → 612 unique conversations, 867k → 8458 real prompts, cost stays at the real $53,637. ## Shared grouping helper - `computeSessionGroupKey(s)` in data.js: `tool::project::firstMsg[0..200]` (or `helper::project` for helpers) — computed once per session on load, exposed as `s.group_key`. - `groupSessionsByConversation(sessions)` in frontend/app.js — shared by **Timeline**, **All Sessions**, **Projects view**, **Cloud Sync**, and **Activity/Heatmap**. One helper, one representative per group, `+N more` badge on cards. ## Logging - `CODEDASH_LOG=0` (default) silences stdout spam (previous `[ACTIVE] pid=... codex/waiting cpu=0%` lines were emitted on every /api/active poll). `ERROR`/`WARN`/`JOB` still go to stdout. - All logs (including verbose tags) always go to `~/.codedash/logs/server.log` with timestamps. Set `CODEDASH_LOG=1` to also mirror to stdout. ## Exports / new endpoints - `loadSessionsAsync(progressCb)` — async variant with incremental mtime-based change detection. - `getWarmingStatus()` + `/api/warming` — background parse progress. - `getSqliteBackfillStatus()` + `/api/sqlite-status` — FTS5 ingest progress + index size. - `createCostAggregator()`, `computeSessionCostForAnalytics(session, opencodeCache)`, `buildOpencodeCostCache(sessions)` — so the async jobs can stream-aggregate without re-wiring `getCostAnalytics`. ## Benchmark (user's machine, 4873 sessions / 1.1 GB JSONL) ``` before (v6.15.10) after loadSessions 109 s (cold) 72 ms (cacheOnly path) /api/active 80 s (34 procs) 350 ms cold, 0 ms warm search 3–10 s (rebuild) 5 ms (FTS5 MATCH) analytics 30 s (blocking) first partial in ~500 ms, full in ~15 s, instant on cache hit leaderboard 35 s (blocking) first partial in ~500 ms, full in ~15 s, instant on cache hit ``` Browser cache note: after upgrading, users should hard-reload (Cmd+Shift+R) once so the split frontend modules re-load — the poll loops are new code and old cached JS will show the initial "Loading..." spinner without progressing. --- src/data.js | 1623 +++++++++++++++++++++++++++++------ src/frontend/analytics.js | 177 +++- src/frontend/app.js | 88 +- src/frontend/cloud.js | 13 +- src/frontend/heatmap.js | 10 +- src/frontend/leaderboard.js | 32 +- src/frontend/styles.css | 111 +++ src/server.js | 426 ++++++++- src/sqlite-index.js | 479 +++++++++++ 9 files changed, 2668 insertions(+), 291 deletions(-) create mode 100644 src/sqlite-index.js diff --git a/src/data.js b/src/data.js index 55b1eee..298b45f 100644 --- a/src/data.js +++ b/src/data.js @@ -98,10 +98,18 @@ function parseOpenCodeMcpServer(toolName) { return toolName.slice(0, idx); } +// Persistent cache dir (survives tmpdir cleanup and process kills) +const CODEDASH_CACHE_DIR = path.join(os.homedir(), '.codedash', 'cache'); +try { fs.mkdirSync(CODEDASH_CACHE_DIR, { recursive: true }); } catch {} + // Disk cache for parsed Claude session files (keyed by path + mtime + size) -const PARSED_CACHE_FILE = path.join(os.tmpdir(), 'codedash-parsed-cache.json'); +const PARSED_CACHE_FILE = path.join(CODEDASH_CACHE_DIR, 'parsed-cache.json'); +// Legacy tmpdir path (migration: read once, then ignore) +const LEGACY_PARSED_CACHE_FILE = path.join(os.tmpdir(), 'codedash-parsed-cache.json'); let _parsedDiskCache = null; let _parsedDiskCacheDirty = false; +let _parsedDiskCacheEntriesSinceFlush = 0; +const PARSED_CACHE_FLUSH_EVERY = 50; // flush after every N new entries // Reverse index: file path -> cache key (avoids repeated fs.statSync) const _fileCacheKeyIndex = {}; @@ -110,20 +118,490 @@ function _loadParsedDiskCache() { try { if (fs.existsSync(PARSED_CACHE_FILE)) { _parsedDiskCache = JSON.parse(fs.readFileSync(PARSED_CACHE_FILE, 'utf8')); + } else if (fs.existsSync(LEGACY_PARSED_CACHE_FILE)) { + // Migrate from tmpdir once + _parsedDiskCache = JSON.parse(fs.readFileSync(LEGACY_PARSED_CACHE_FILE, 'utf8')); + _parsedDiskCacheDirty = true; } } catch {} if (!_parsedDiskCache) _parsedDiskCache = {}; } -function _saveParsedDiskCache() { +function _saveParsedDiskCache(force) { if (!_parsedDiskCacheDirty || !_parsedDiskCache) return; + if (!force && _parsedDiskCacheEntriesSinceFlush < PARSED_CACHE_FLUSH_EVERY) return; try { - fs.writeFileSync(PARSED_CACHE_FILE, JSON.stringify(_parsedDiskCache)); + // Atomic write: write to .tmp then rename + const tmp = PARSED_CACHE_FILE + '.tmp'; + fs.writeFileSync(tmp, JSON.stringify(_parsedDiskCache)); + fs.renameSync(tmp, PARSED_CACHE_FILE); _parsedDiskCacheDirty = false; + _parsedDiskCacheEntriesSinceFlush = 0; + } catch {} +} + +// ── Disk cache for computed session cost (path+mtime+size → cost) ───────── +const COST_CACHE_FILE = path.join(CODEDASH_CACHE_DIR, 'cost-cache.json'); +let _costDiskCache = null; +let _costDiskCacheDirty = false; +let _costDiskCacheEntriesSinceFlush = 0; +const COST_CACHE_FLUSH_EVERY = 50; + +function _loadCostDiskCache() { + if (_costDiskCache) return; + try { + if (fs.existsSync(COST_CACHE_FILE)) { + _costDiskCache = JSON.parse(fs.readFileSync(COST_CACHE_FILE, 'utf8')); + } + } catch {} + if (!_costDiskCache) _costDiskCache = {}; +} + +function _saveCostDiskCache(force) { + if (!_costDiskCacheDirty || !_costDiskCache) return; + if (!force && _costDiskCacheEntriesSinceFlush < COST_CACHE_FLUSH_EVERY) return; + try { + const tmp = COST_CACHE_FILE + '.tmp'; + fs.writeFileSync(tmp, JSON.stringify(_costDiskCache)); + fs.renameSync(tmp, COST_CACHE_FILE); + _costDiskCacheDirty = false; + _costDiskCacheEntriesSinceFlush = 0; } catch {} } -function parseClaudeSessionFile(sessionFile) { +// ── Background parse warming state ──────────────────────── +// Populated by sync loadSessions() with file paths whose parse cache is stale. +// A singleton background task drains this set with setImmediate yielding so +// the HTTP event loop stays responsive. UI can read _warmingStatus for progress. +const _pendingParseFiles = new Set(); +let _warmingRunning = false; +const _warmingStatus = { + running: false, + done: 0, + total: 0, + phase: 'idle', + startedAt: 0, + finishedAt: 0, +}; + +// ── SQLite ingest helper ──────────────────────────────────── +// Parse a single Claude session file fully (messages + metadata) and push +// a batch row into the SQLite index. Used by the background warmer after +// the parsed-cache entry has been created. Cheap when already indexed. +let _sqliteIngestBatch = []; +const _SQLITE_INGEST_FLUSH_EVERY = 5; + +function _ingestClaudeFileToSqlite(filePath) { + let sqliteIndex; + try { sqliteIndex = require('./sqlite-index'); } catch { return; } + try { + if (sqliteIndex.isFileCurrent(filePath)) return; // already indexed + } catch { return; } + + let stat; + try { stat = fs.statSync(filePath); } catch { return; } + + // Full read — at ingest time we need messages for FTS. Streaming for big + // files would be nicer but the warmer yields between files already. + let lines; + try { lines = readLines(filePath); } catch { return; } + + const sid = path.basename(filePath, '.jsonl'); + let projectPath = ''; + let tool = 'claude'; + let firstTs = stat.mtimeMs; + let lastTs = stat.mtimeMs; + let firstMsg = ''; + let userMsgCount = 0; + let totalMsgCount = 0; + const msgs = []; + const dayMsgs = {}; // day → {messages, first_ts, last_ts} + let seq = 0; + + for (const line of lines) { + try { + const entry = JSON.parse(line); + if (entry.type !== 'user' && entry.type !== 'assistant') continue; + const role = entry.type; + const content = extractContent((entry.message || {}).content); + if (!content) continue; + const ts = entry.timestamp ? (typeof entry.timestamp === 'number' ? entry.timestamp : new Date(entry.timestamp).getTime()) : 0; + if (ts) { + if (ts < firstTs) firstTs = ts; + if (ts > lastTs) lastTs = ts; + } + if (!projectPath && entry.cwd) projectPath = entry.cwd; + totalMsgCount++; + if (role === 'user') { + userMsgCount++; + if (!firstMsg && content) firstMsg = content.slice(0, 200); + } + msgs.push({ seq: seq++, role, ts, content: content.slice(0, 8000) }); + // Day breakdown for user messages only + if (role === 'user' && ts > 1000000000000) { + const d = new Date(ts); + const day = d.getFullYear() + '-' + String(d.getMonth()+1).padStart(2,'0') + '-' + String(d.getDate()).padStart(2,'0'); + if (!dayMsgs[day]) dayMsgs[day] = { messages: 0, first_ts: ts, last_ts: ts }; + dayMsgs[day].messages++; + if (ts < dayMsgs[day].first_ts) dayMsgs[day].first_ts = ts; + if (ts > dayMsgs[day].last_ts) dayMsgs[day].last_ts = ts; + } + } catch {} + } + + // Convert dayMsgs to daily_stats rows + const daily = {}; + for (const day in dayMsgs) { + const dm = dayMsgs[day]; + daily[day] = { + messages: dm.messages, + hours: Math.min((dm.last_ts - dm.first_ts) / 3600000, 16), + }; + } + + _sqliteIngestBatch.push({ + filePath, + session: { + id: sid, + tool, + project: projectPath, + project_short: projectPath.replace(os.homedir(), '~'), + first_ts: firstTs, + last_ts: lastTs, + messages: totalMsgCount, + user_messages: userMsgCount, + file_size: stat.size, + first_message: firstMsg, + source_mtime: stat.mtimeMs, + source_size: stat.size, + }, + messages: msgs, + daily, + }); + + // Don't flush here — backfill loop awaits the flush between files. +} + +// Ingest a single Codex rollout JSONL into SQLite. +function _ingestCodexFileToSqlite(filePath) { + let sqliteIndex; + try { sqliteIndex = require('./sqlite-index'); } catch { return; } + try { if (sqliteIndex.isFileCurrent(filePath)) return; } catch { return; } + + let stat; + try { stat = fs.statSync(filePath); } catch { return; } + + // Extract session id from filename + const basename = path.basename(filePath, '.jsonl'); + const uuidMatch = basename.match(/([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})/); + if (!uuidMatch) return; + const sid = uuidMatch[1]; + + let lines; + try { lines = readLines(filePath); } catch { return; } + + let projectPath = ''; + let firstTs = stat.mtimeMs; + let lastTs = stat.mtimeMs; + let firstMsg = ''; + let userMsgCount = 0; + let totalMsgCount = 0; + const msgs = []; + const dayMsgs = {}; + let seq = 0; + + for (const line of lines) { + try { + const entry = JSON.parse(line); + if (entry.type === 'session_meta' && entry.payload && entry.payload.cwd && !projectPath) { + projectPath = entry.payload.cwd; + continue; + } + if (entry.type !== 'response_item' || !entry.payload) continue; + const role = entry.payload.role; + if (role !== 'user' && role !== 'assistant') continue; + const content = extractContent(entry.payload.content); + if (!content || isSystemMessage(content)) continue; + let ts = 0; + const tsVal = entry.timestamp || entry.ts; + if (typeof tsVal === 'number') ts = tsVal; + else if (typeof tsVal === 'string') ts = Date.parse(tsVal) || 0; + if (ts && ts < firstTs) firstTs = ts; + if (ts && ts > lastTs) lastTs = ts; + + totalMsgCount++; + if (role === 'user') { + userMsgCount++; + if (!firstMsg) firstMsg = content.slice(0, 200); + } + msgs.push({ seq: seq++, role, ts, content: content.slice(0, 8000) }); + if (role === 'user' && ts > 1000000000000) { + const d = new Date(ts); + const day = d.getFullYear() + '-' + String(d.getMonth()+1).padStart(2,'0') + '-' + String(d.getDate()).padStart(2,'0'); + if (!dayMsgs[day]) dayMsgs[day] = { messages: 0, first_ts: ts, last_ts: ts }; + dayMsgs[day].messages++; + if (ts < dayMsgs[day].first_ts) dayMsgs[day].first_ts = ts; + if (ts > dayMsgs[day].last_ts) dayMsgs[day].last_ts = ts; + } + } catch {} + } + + const daily = {}; + for (const day in dayMsgs) { + const dm = dayMsgs[day]; + daily[day] = { + messages: dm.messages, + hours: Math.min((dm.last_ts - dm.first_ts) / 3600000, 16), + }; + } + + _sqliteIngestBatch.push({ + filePath, + session: { + id: sid, + tool: 'codex', + project: projectPath, + project_short: projectPath.replace(os.homedir(), '~'), + first_ts: firstTs, + last_ts: lastTs, + messages: totalMsgCount, + user_messages: userMsgCount, + file_size: stat.size, + first_message: firstMsg, + source_mtime: stat.mtimeMs, + source_size: stat.size, + }, + messages: msgs, + daily, + }); + + // Don't flush here — backfill loop awaits the flush between files. +} + +async function _flushSqliteIngestBatch() { + if (_sqliteIngestBatch.length === 0) return; + const batch = _sqliteIngestBatch; + _sqliteIngestBatch = []; // take ownership before the async call + try { + const sqliteIndex = require('./sqlite-index'); + await sqliteIndex.indexBatchAsync(batch); + } catch (e) { + try { console.error('sqlite ingest flush failed:', e.message); } catch {} + } +} + +// ── SQLite backfill ──────────────────────────────────────── +// One-shot background task that iterates all existing Claude session files +// and ingests them into SQLite (if not already there, matched by mtime+size +// via files_seen). Yields between files. +let _sqliteBackfillRunning = false; +const _sqliteBackfillStatus = { + running: false, + done: 0, + total: 0, + phase: 'idle', + startedAt: 0, + finishedAt: 0, +}; + +function getSqliteBackfillStatus() { + return Object.assign({}, _sqliteBackfillStatus); +} + +function _ensureSqliteBackfillRunning() { + if (_sqliteBackfillRunning) return; + let sqliteIndex; + try { sqliteIndex = require('./sqlite-index'); } catch { return; } + + _sqliteBackfillRunning = true; + _sqliteBackfillStatus.running = true; + _sqliteBackfillStatus.startedAt = Date.now(); + _sqliteBackfillStatus.phase = 'scanning'; + _sqliteBackfillStatus.done = 0; + + setImmediate(async () => { + try { + sqliteIndex.ensureSchema(); + + // Enumerate all Claude JSONL files + Codex session files + const allFiles = []; // array of {file, kind} + const walkClaude = (dir) => { + try { + for (const proj of fs.readdirSync(dir)) { + const pDir = path.join(dir, proj); + try { + if (!fs.statSync(pDir).isDirectory()) continue; + for (const f of fs.readdirSync(pDir)) { + if (f.endsWith('.jsonl')) allFiles.push({ file: path.join(pDir, f), kind: 'claude' }); + } + } catch {} + } + } catch {} + }; + if (fs.existsSync(PROJECTS_DIR)) walkClaude(PROJECTS_DIR); + for (const extra of EXTRA_CLAUDE_DIRS) { + const ep = path.join(extra, 'projects'); + if (fs.existsSync(ep)) walkClaude(ep); + } + // Codex session files (~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl) + const codexSessDir = path.join(CODEX_DIR, 'sessions'); + if (fs.existsSync(codexSessDir)) { + const walkCodex = (dir) => { + try { + for (const entry of fs.readdirSync(dir, { withFileTypes: true })) { + const full = path.join(dir, entry.name); + if (entry.isDirectory()) walkCodex(full); + else if (entry.name.endsWith('.jsonl')) allFiles.push({ file: full, kind: 'codex' }); + } + } catch {} + }; + walkCodex(codexSessDir); + } + + _sqliteBackfillStatus.total = allFiles.length; + _sqliteBackfillStatus.phase = 'ingesting'; + + // Sort newest first so fresh sessions hit the index sooner + allFiles.sort((a, b) => { + try { return fs.statSync(b.file).mtimeMs - fs.statSync(a.file).mtimeMs; } catch { return 0; } + }); + + // Single bulk read of files_seen — avoids N sync SQL calls + const filesSeen = sqliteIndex.loadAllFilesSeen(); + + for (const item of allFiles) { + try { + let stat; + try { stat = fs.statSync(item.file); } catch { _sqliteBackfillStatus.done++; continue; } + const seen = filesSeen.get(item.file); + const isCurrent = seen && seen.mtime === stat.mtimeMs && seen.size === stat.size; + if (!isCurrent) { + if (item.kind === 'codex') { + _ingestCodexFileToSqlite(item.file); + } else { + _ingestClaudeFileToSqlite(item.file); + } + if (_sqliteIngestBatch.length >= _SQLITE_INGEST_FLUSH_EVERY) { + await _flushSqliteIngestBatch(); + } + } + } catch {} + _sqliteBackfillStatus.done++; + // Yield every 10 files so HTTP requests can slot in + if (_sqliteBackfillStatus.done % 10 === 0) { + await new Promise(r => setImmediate(r)); + } + } + await _flushSqliteIngestBatch(); + + _sqliteBackfillStatus.phase = 'done'; + _sqliteBackfillStatus.finishedAt = Date.now(); + } catch (e) { + _sqliteBackfillStatus.phase = 'error: ' + (e && e.message || 'unknown'); + _sqliteBackfillStatus.finishedAt = Date.now(); + } finally { + _sqliteBackfillStatus.running = false; + _sqliteBackfillRunning = false; + } + }); +} + +function _ensureWarmingRunning() { + if (_warmingRunning) return; + if (_pendingParseFiles.size === 0 && _pendingCodexFiles.size === 0) return; + _warmingRunning = true; + _warmingStatus.running = true; + _warmingStatus.startedAt = Date.now(); + _warmingStatus.done = 0; + _warmingStatus.total = _pendingParseFiles.size + _pendingCodexFiles.size; + _warmingStatus.phase = 'parsing session files'; + setImmediate(async () => { + try { + while (_pendingParseFiles.size > 0 || _pendingCodexFiles.size > 0) { + // Claude batch (newest-first) + if (_pendingParseFiles.size > 0) { + const batch = Array.from(_pendingParseFiles); + batch.sort((a, b) => { + try { return fs.statSync(b).mtimeMs - fs.statSync(a).mtimeMs; } catch { return 0; } + }); + _warmingStatus.total = _warmingStatus.done + batch.length + _pendingCodexFiles.size; + for (const file of batch) { + _pendingParseFiles.delete(file); + try { + let stat; + try { stat = fs.statSync(file); } catch { _warmingStatus.done++; continue; } + if (stat.size >= 5 * 1024 * 1024) { + await parseClaudeSessionFileAsync(file); + } else { + parseClaudeSessionFile(file); + await new Promise(r => setImmediate(r)); + } + // Also push into persistent SQLite index (batched) + try { _ingestClaudeFileToSqlite(file); } catch {} + } catch {} + _warmingStatus.done++; + } + _flushSqliteIngestBatch(); + } + // Codex batch + if (_pendingCodexFiles.size > 0) { + const batch = Array.from(_pendingCodexFiles); + batch.sort((a, b) => { + try { return fs.statSync(b).mtimeMs - fs.statSync(a).mtimeMs; } catch { return 0; } + }); + _warmingStatus.total = _warmingStatus.done + batch.length + _pendingParseFiles.size; + for (const file of batch) { + _pendingCodexFiles.delete(file); + try { + parseCodexSessionFile(file); + await new Promise(r => setImmediate(r)); + } catch {} + _warmingStatus.done++; + } + } + // Invalidate session cache so next /api/sessions picks up new detail + _sessionsCache = null; + _sessionsCacheTs = 0; + } + _saveParsedDiskCache(true); + _flushSqliteIngestBatch(); + _warmingStatus.phase = 'done'; + _warmingStatus.finishedAt = Date.now(); + } catch (e) { + _warmingStatus.phase = 'error: ' + (e && e.message || 'unknown'); + _warmingStatus.finishedAt = Date.now(); + } finally { + _warmingStatus.running = false; + _warmingRunning = false; + } + }); +} + +function getWarmingStatus() { + return Object.assign({}, _warmingStatus, { + pending: _pendingParseFiles.size, + }); +} + +// Flush all caches on process exit / signals so partial progress isn't lost +let _flushHandlersInstalled = false; +function _installFlushHandlers() { + if (_flushHandlersInstalled) return; + _flushHandlersInstalled = true; + const flushAll = () => { + try { _saveParsedDiskCache(true); } catch {} + try { _saveCostDiskCache(true); } catch {} + try { if (typeof _saveDailyStatsDiskCache === 'function') _saveDailyStatsDiskCache(); } catch {} + try { if (typeof _saveGitRootDiskCache === 'function') _saveGitRootDiskCache(); } catch {} + try { if (typeof _flushSqliteIngestBatch === 'function') _flushSqliteIngestBatch(); } catch {} + }; + process.on('exit', flushAll); + process.on('SIGINT', () => { flushAll(); process.exit(0); }); + process.on('SIGTERM', () => { flushAll(); process.exit(0); }); +} +_installFlushHandlers(); + +function parseClaudeSessionFile(sessionFile, opts) { if (!fs.existsSync(sessionFile)) return null; let stat; @@ -139,6 +617,14 @@ function parseClaudeSessionFile(sessionFile) { _fileCacheKeyIndex[sessionFile] = cacheKey; if (_parsedDiskCache[cacheKey]) return _parsedDiskCache[cacheKey]; + // Cache-only mode: skip actual file read when not cached. Caller gets + // null-ish placeholder and the real parse happens in a background job. + // This is what keeps the sync loadSessions() fast on cold caches. + if (opts && opts.cacheOnly) { + _pendingParseFiles.add(sessionFile); + return null; + } + let lines; try { lines = readLines(sessionFile); @@ -158,11 +644,27 @@ function parseClaudeSessionFile(sessionFile) { const mcpSet = new Set(); const skillSet = new Set(); + // Helper: is this `type=user` entry a REAL user prompt, not a tool_result? + // In Claude Code JSONL, tool_result messages are stored as type='user' with + // content=[{type:'tool_result', ...}]. We want to count only messages whose + // content contains real text from the human. + const isRealUserPrompt = (entry) => { + const c = (entry.message || {}).content; + if (typeof c === 'string') return c.trim().length > 0; + if (Array.isArray(c)) { + for (const p of c) { + if (p && p.type === 'text' && p.text && p.text.trim()) return true; + } + return false; + } + return false; + }; + for (const line of lines) { try { const entry = JSON.parse(line); if (entry.type === 'user' || entry.type === 'assistant') msgCount++; - if (entry.type === 'user') userMsgCount++; + if (entry.type === 'user' && isRealUserPrompt(entry)) userMsgCount++; if (entry.timestamp) { if (entry.timestamp < firstTs) firstTs = entry.timestamp; if (entry.timestamp > lastTs) lastTs = entry.timestamp; @@ -225,6 +727,123 @@ function parseClaudeSessionFile(sessionFile) { // Cache to disk _parsedDiskCache[cacheKey] = result; _parsedDiskCacheDirty = true; + _parsedDiskCacheEntriesSinceFlush++; + // Periodic flush so long operations don't lose progress on kill + if (_parsedDiskCacheEntriesSinceFlush >= PARSED_CACHE_FLUSH_EVERY) { + _saveParsedDiskCache(true); + } + return result; +} + +// Async variant: for large files (>5 MB) reads in chunks with setImmediate +// between chunks so the Node event loop can handle other requests. Same +// result shape and cache as sync version. +async function parseClaudeSessionFileAsync(sessionFile) { + if (!fs.existsSync(sessionFile)) return null; + let stat; + try { stat = fs.statSync(sessionFile); } catch { return null; } + + _loadParsedDiskCache(); + const cacheKey = sessionFile + '|' + stat.mtimeMs + '|' + stat.size; + _fileCacheKeyIndex[sessionFile] = cacheKey; + if (_parsedDiskCache[cacheKey]) return _parsedDiskCache[cacheKey]; + + // Small files: use sync path (faster, avoids Promise overhead) + const BIG_THRESHOLD = 5 * 1024 * 1024; // 5 MB + if (stat.size < BIG_THRESHOLD) return parseClaudeSessionFile(sessionFile); + + // Big file: stream read + parse line by line with periodic yielding. + let projectPath = ''; + let tool = 'claude'; + let msgCount = 0; + let firstMsg = ''; + let customTitle = ''; + let firstTs = stat.mtimeMs; + let lastTs = stat.mtimeMs; + let userMsgCount = 0; + let entrypointFound = false; + let worktreeOriginalCwd = ''; + const mcpSet = new Set(); + const skillSet = new Set(); + + const readline = require('readline'); + const stream = fs.createReadStream(sessionFile, { encoding: 'utf8', highWaterMark: 1 << 20 }); + const rl = readline.createInterface({ input: stream, crlfDelay: Infinity }); + + const isRealUserPromptAsync = (entry) => { + const c = (entry.message || {}).content; + if (typeof c === 'string') return c.trim().length > 0; + if (Array.isArray(c)) { + for (const p of c) { + if (p && p.type === 'text' && p.text && p.text.trim()) return true; + } + return false; + } + return false; + }; + + let linesSinceYield = 0; + for await (const line of rl) { + if (!line) continue; + try { + const entry = JSON.parse(line); + if (entry.type === 'user' || entry.type === 'assistant') msgCount++; + if (entry.type === 'user' && isRealUserPromptAsync(entry)) userMsgCount++; + if (entry.timestamp) { + if (entry.timestamp < firstTs) firstTs = entry.timestamp; + if (entry.timestamp > lastTs) lastTs = entry.timestamp; + } + if (!projectPath && entry.type === 'user' && entry.cwd) projectPath = entry.cwd; + if (!worktreeOriginalCwd && entry.type === 'worktree-state' && entry.worktreeSession && entry.worktreeSession.originalCwd) { + worktreeOriginalCwd = entry.worktreeSession.originalCwd; + } + if (!entrypointFound && entry.type === 'user' && entry.entrypoint) { + entrypointFound = true; + if (entry.entrypoint !== 'cli') tool = 'claude-ext'; + } + if (entry.type === 'custom-title' && typeof entry.customTitle === 'string') { + const title = entry.customTitle.trim(); + if (title) customTitle = title.slice(0, 200); + } + if (!firstMsg && entry.type === 'user' && entry.message && entry.message.content) { + const content = extractContent(entry.message.content).trim(); + if (content) firstMsg = content.slice(0, 200); + } + if (entry.type === 'assistant') { + const aContent = (entry.message || {}).content; + if (Array.isArray(aContent)) { + for (const block of aContent) { + if (!block || block.type !== 'tool_use') continue; + const name = block.name || ''; + if (name.startsWith('mcp__')) { + const parts = name.split('__'); + if (parts.length >= 3) mcpSet.add(parts[1]); + } else if (name === 'Skill') { + const sk = (block.input || {}).skill; + if (sk) skillSet.add(sk.includes(':') ? sk.split(':')[0] : sk); + } + } + } + } + } catch {} + if (++linesSinceYield >= 2000) { + linesSinceYield = 0; + await new Promise(r => setImmediate(r)); + } + } + + const result = { + projectPath, tool, msgCount, userMsgCount, + firstMsg, customTitle, firstTs, lastTs, + fileSize: stat.size, worktreeOriginalCwd, + mcpServers: Array.from(mcpSet), skills: Array.from(skillSet), + }; + _parsedDiskCache[cacheKey] = result; + _parsedDiskCacheDirty = true; + _parsedDiskCacheEntriesSinceFlush++; + if (_parsedDiskCacheEntriesSinceFlush >= PARSED_CACHE_FLUSH_EVERY) { + _saveParsedDiskCache(true); + } return result; } @@ -926,13 +1545,25 @@ function loadCursorVscdbDetail(sessionId) { return { messages: messages.slice(0, 200) }; } -function parseCodexSessionFile(sessionFile) { +function parseCodexSessionFile(sessionFile, opts) { if (!fs.existsSync(sessionFile)) return null; let stat; + try { stat = fs.statSync(sessionFile); } catch { return null; } + + // Reuse the same parsed-cache keyed by path+mtime+size + _loadParsedDiskCache(); + const cacheKey = 'codex:' + sessionFile + '|' + stat.mtimeMs + '|' + stat.size; + if (_parsedDiskCache[cacheKey]) return _parsedDiskCache[cacheKey]; + + // Cache-only mode: queue for background parsing, don't block + if (opts && opts.cacheOnly) { + _pendingCodexFiles.add(sessionFile); + return null; + } + let lines; try { - stat = fs.statSync(sessionFile); lines = readLines(sessionFile); } catch { return null; @@ -955,6 +1586,25 @@ function parseCodexSessionFile(sessionFile) { let firstMsg = ''; let firstTs = stat.mtimeMs; let lastTs = stat.mtimeMs; + // Detect sub-agent/scripted codex sessions. Multiple signals: + // 1. originator='codex_exec' or source='exec' (standard `codex exec`) + // 2. first user prompt matches known auto-script patterns (team scripts + // that spawn codex without the exec flag). Covers: "You are in /path. + // Task: X", "Read-only task. Inspect ...", "Work in /path", "Pair-local + // lane", "## Memory Writing Agent", etc. + let isHelper = false; + // Regexes checked against the first user prompt (see after the loop) + const AUTO_SCRIPT_PATTERNS = [ + /^You are in \//, + /^Read-only task\./, + /^Work (only )?in \//, + /^Pair-local [^\n]{1,60} lane/, + /^## [A-Z][a-z]+ [A-Z][a-z]+ Agent/, // "## Memory Writing Agent:..." + /^Read \/[^\s]+\/AGENTS\.md/, + /^Read \$[A-Z_]+\//, + /^\[Sub-agent results\]/, + /^You are OMX /, + ]; const mcpSet = new Set(); for (const line of lines) { @@ -966,8 +1616,11 @@ function parseCodexSessionFile(sessionFile) { if (ts > lastTs) lastTs = ts; } - if (entry.type === 'session_meta' && entry.payload && entry.payload.cwd && !projectPath) { - projectPath = entry.payload.cwd; + if (entry.type === 'session_meta' && entry.payload) { + if (entry.payload.cwd && !projectPath) projectPath = entry.payload.cwd; + const originator = entry.payload.originator || ''; + const source = entry.payload.source || ''; + if (originator === 'codex_exec' || source === 'exec') isHelper = true; continue; } @@ -995,7 +1648,14 @@ function parseCodexSessionFile(sessionFile) { } catch {} } - return { + // Secondary helper detection by first user prompt pattern + if (!isHelper && firstMsg) { + for (const re of AUTO_SCRIPT_PATTERNS) { + if (re.test(firstMsg)) { isHelper = true; break; } + } + } + + const result = { projectPath, msgCount, userMsgCount, @@ -1004,11 +1664,23 @@ function parseCodexSessionFile(sessionFile) { lastTs, fileSize: stat.size, mcpServers: Array.from(mcpSet), + isHelper, }; + _parsedDiskCache[cacheKey] = result; + _parsedDiskCacheDirty = true; + _parsedDiskCacheEntriesSinceFlush++; + if (_parsedDiskCacheEntriesSinceFlush >= PARSED_CACHE_FLUSH_EVERY) { + _saveParsedDiskCache(true); + } + return result; } +// Queue for background codex file parsing (drained by ensureWarmingRunning) +const _pendingCodexFiles = new Set(); + function scanCodexSessions() { - const sessions = []; + // Map for O(1) session lookups (was O(n²) with .find) + const sessionsById = new Map(); const codexTitles = parseCodexSessionIndex(CODEX_DIR); const codexHistory = path.join(CODEX_DIR, 'history.jsonl'); if (fs.existsSync(codexHistory)) { @@ -1016,12 +1688,11 @@ function scanCodexSessions() { for (const line of lines) { try { const d = JSON.parse(line); - // Codex uses session_id, ts (seconds), text const sid = d.session_id || d.sessionId || d.id; if (!sid) continue; const ts = d.ts ? d.ts * 1000 : (d.timestamp || Date.now()); - if (!sessions.find(s => s.id === sid)) { - sessions.push({ + if (!sessionsById.has(sid)) { + sessionsById.set(sid, { id: sid, tool: 'codex', project: d.project || d.cwd || '', @@ -1040,10 +1711,10 @@ function scanCodexSessions() { } // Enrich with session files from ~/.codex/sessions/ + // Cache-only: uses the parse cache; uncached files go to background queue. const codexSessionsDir = path.join(CODEX_DIR, 'sessions'); if (fs.existsSync(codexSessionsDir)) { try { - // Walk year/month/day directories const files = []; const walkDir = (dir) => { for (const entry of fs.readdirSync(dir, { withFileTypes: true })) { @@ -1055,37 +1726,61 @@ function scanCodexSessions() { walkDir(codexSessionsDir); for (const f of files) { - // Extract session ID from filename (rollout-DATE-UUID.jsonl) const basename = path.basename(f, '.jsonl'); const uuidMatch = basename.match(/([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})/); if (!uuidMatch) continue; const sid = uuidMatch[1]; - const summary = parseCodexSessionFile(f); - if (!summary) continue; - - const existing = sessions.find(s => s.id === sid); + const summary = parseCodexSessionFile(f, { cacheOnly: true }); + if (!summary) { + // No cached summary yet — create a minimal placeholder from stat + // so the session still shows in the list. Background job will enrich. + let size = 0, mtimeMs = 0; + try { const st = fs.statSync(f); size = st.size; mtimeMs = st.mtimeMs; } catch {} + const existing = sessionsById.get(sid); + if (existing) { + existing.has_detail = true; + existing.file_size = size; + existing._detail_pending = true; + } else { + sessionsById.set(sid, { + id: sid, + tool: 'codex', + project: '', + project_short: '', + first_ts: mtimeMs, + last_ts: mtimeMs, + messages: 0, + first_message: codexTitles[sid] || '', + has_detail: true, + file_size: size, + detail_messages: 0, + user_messages: 0, + mcp_servers: [], + skills: [], + _detail_pending: true, + }); + } + continue; + } + const existing = sessionsById.get(sid); if (existing) { existing.has_detail = true; existing.file_size = summary.fileSize; existing.messages = summary.msgCount; existing.detail_messages = summary.msgCount; existing.user_messages = summary.userMsgCount || 0; - if (codexTitles[sid]) { - existing.first_message = codexTitles[sid]; - } else if (summary.firstMsg && !existing.first_message) { - existing.first_message = summary.firstMsg; - } + if (summary.isHelper) existing.is_helper = true; + if (codexTitles[sid]) existing.first_message = codexTitles[sid]; + else if (summary.firstMsg && !existing.first_message) existing.first_message = summary.firstMsg; if (summary.projectPath && !existing.project) { existing.project = summary.projectPath; existing.project_short = summary.projectPath.replace(os.homedir(), '~'); } existing.first_ts = Math.min(existing.first_ts, summary.firstTs); existing.last_ts = Math.max(existing.last_ts, summary.lastTs); - if (summary.mcpServers && summary.mcpServers.length > 0) { - existing.mcp_servers = summary.mcpServers; - } + if (summary.mcpServers && summary.mcpServers.length > 0) existing.mcp_servers = summary.mcpServers; } else { - sessions.push({ + sessionsById.set(sid, { id: sid, tool: 'codex', project: summary.projectPath, @@ -1100,13 +1795,14 @@ function scanCodexSessions() { user_messages: summary.userMsgCount || 0, mcp_servers: summary.mcpServers || [], skills: [], + is_helper: summary.isHelper || false, }); } } } catch {} } - return sessions; + return Array.from(sessionsById.values()); } // ── Git root resolver ─────────────────────────────────────── @@ -1121,46 +1817,105 @@ function scanCodexSessions() { // from the session cwd string. Works without git for standard worktree layouts. const _gitRootCache = {}; -const GIT_ROOT_CACHE_FILE = path.join(os.tmpdir(), 'codedash-gitroot-cache.json'); +// Persistent dir (survives /tmp cleanup). Legacy file in tmpdir migrated once. +const GIT_ROOT_CACHE_FILE = path.join(CODEDASH_CACHE_DIR, 'git-root-cache.json'); +const LEGACY_GIT_ROOT_CACHE_FILE = path.join(os.tmpdir(), 'codedash-gitroot-cache.json'); let _gitRootDiskCache = null; +let _gitRootDirty = false; +// Queue of project paths that still need git resolution. Filled by sync +// loadSessions() when a path is uncached, drained in background by +// _ensureGitRootResolverRunning() so the sync path returns immediately. +const _pendingGitRoots = new Set(); +let _gitRootResolverRunning = false; function _loadGitRootDiskCache() { if (_gitRootDiskCache) return; try { if (fs.existsSync(GIT_ROOT_CACHE_FILE)) { _gitRootDiskCache = JSON.parse(fs.readFileSync(GIT_ROOT_CACHE_FILE, 'utf8')); - // Pre-fill memory cache from disk - Object.assign(_gitRootCache, _gitRootDiskCache); + } else if (fs.existsSync(LEGACY_GIT_ROOT_CACHE_FILE)) { + _gitRootDiskCache = JSON.parse(fs.readFileSync(LEGACY_GIT_ROOT_CACHE_FILE, 'utf8')); + _gitRootDirty = true; } + if (_gitRootDiskCache) Object.assign(_gitRootCache, _gitRootDiskCache); } catch {} if (!_gitRootDiskCache) _gitRootDiskCache = {}; } function _saveGitRootDiskCache() { + if (!_gitRootDirty) return; try { - fs.writeFileSync(GIT_ROOT_CACHE_FILE, JSON.stringify(_gitRootCache)); + const tmp = GIT_ROOT_CACHE_FILE + '.tmp'; + fs.writeFileSync(tmp, JSON.stringify(_gitRootCache)); + fs.renameSync(tmp, GIT_ROOT_CACHE_FILE); + _gitRootDirty = false; } catch {} } +// Sync resolver: never shells out from the hot path. Uses cache; queues +// unknown paths for background resolution. +// ── Conversation group key ───────────────────────────────── +// Two sessions belong to the same conversation group if they share the same +// project AND the same initial user prompt (first ~200 chars). This dedupes +// `codex exec` retries, `codex resume` chains, and sub-agent re-runs that +// would otherwise show as N separate cards in the UI. Helper sessions +// (is_helper=true) all collapse into their own single bucket per project. +function computeSessionGroupKey(s) { + const proj = (s.project || 'unknown').trim(); + // Normalize first message: lowercase, collapse whitespace, truncate + let msg = (s.first_message || '').replace(/\s+/g, ' ').trim().slice(0, 200); + if (s.is_helper) { + // All helpers in the same project collapse into one group. + return 'helper::' + proj; + } + if (!msg) return s.tool + '::' + proj + '::' + s.id; // no prompt — unique + return s.tool + '::' + proj + '::' + msg; +} + function resolveGitRoot(projectPath) { if (!projectPath) return ''; _loadGitRootDiskCache(); if (_gitRootCache[projectPath] !== undefined) return _gitRootCache[projectPath]; - // Skip remote/non-existent paths + // Fast skip: non-existent paths can never be git roots, record & done. if (!fs.existsSync(projectPath)) { _gitRootCache[projectPath] = ''; + _gitRootDirty = true; return ''; } - try { - const root = execFileSync('git', ['-C', projectPath, 'rev-parse', '--show-toplevel'], { - encoding: 'utf8', timeout: 2000, windowsHide: true, stdio: ['pipe', 'pipe', 'pipe'] - }).trim(); - _gitRootCache[projectPath] = root; - return root; - } catch { - _gitRootCache[projectPath] = ''; - return ''; - } + // Queue for background resolver — sync path returns '' for now + _pendingGitRoots.add(projectPath); + _ensureGitRootResolverRunning(); + return ''; +} + +function _ensureGitRootResolverRunning() { + if (_gitRootResolverRunning) return; + if (_pendingGitRoots.size === 0) return; + _gitRootResolverRunning = true; + setImmediate(async () => { + try { + while (_pendingGitRoots.size > 0) { + const snapshot = Array.from(_pendingGitRoots); + for (const p of snapshot) { + _pendingGitRoots.delete(p); + if (_gitRootCache[p] !== undefined) continue; + let root = ''; + try { + root = execFileSync('git', ['-C', p, 'rev-parse', '--show-toplevel'], { + encoding: 'utf8', timeout: 2000, windowsHide: true, stdio: ['pipe', 'pipe', 'pipe'], + }).trim(); + } catch {} + _gitRootCache[p] = root; + _gitRootDirty = true; + // Yield between shell-outs so HTTP stays responsive + await new Promise(r => setImmediate(r)); + } + _saveGitRootDiskCache(); + } + } finally { + _gitRootResolverRunning = false; + } + }); } const _gitInfoCache = {}; @@ -1205,15 +1960,15 @@ function getProjectGitInfo(projectPath) { let _sessionsCache = null; let _sessionsCacheTs = 0; -const SESSIONS_CACHE_TTL = 60000; // 60 seconds — hot cache, invalidated by file changes +const SESSIONS_CACHE_TTL = 60000; // 60 seconds — hot cache, extended if no file changes -// Track file mtimes for smart invalidation +// Track history/projects mtime so we only rescan when files actually changed. +// Lets loadSessions() extend the cache window past TTL when nothing happened. let _historyMtime = 0; let _historySize = 0; let _projectsDirMtime = 0; function _sessionsNeedRescan() { - // Check if history.jsonl or projects dir changed since last scan try { if (fs.existsSync(HISTORY_FILE)) { const st = fs.statSync(HISTORY_FILE); @@ -1374,11 +2129,11 @@ function _loadCursorVscdbInBackground() { function loadSessions() { const now = Date.now(); if (_sessionsCache) { - // Hot cache: return immediately if within TTL and no file changes + // Hot cache: return immediately within TTL window if ((now - _sessionsCacheTs) < SESSIONS_CACHE_TTL) return _sessionsCache; - // Extended cache: even after TTL, only rescan if files actually changed + // Extended cache: after TTL, only rescan if files actually changed if (!_sessionsNeedRescan()) { - _sessionsCacheTs = now; // extend TTL + _sessionsCacheTs = now; return _sessionsCache; } } @@ -1539,12 +2294,19 @@ function loadSessions() { } if (sessionFile) { - const summary = parseClaudeSessionFile(sessionFile); + // Cache-only: never read disk from the sync path. Uncached files are + // queued for background warming — subsequent calls will see full data. + const summary = parseClaudeSessionFile(sessionFile, { cacheOnly: true }); if (summary) mergeClaudeSessionDetail(s, summary, sessionFile); else { + // Placeholder: session exists on disk, actual details pending s.has_detail = true; - try { s.file_size = fs.statSync(sessionFile).size; } catch { s.file_size = 0; } s._session_file = sessionFile; + s._detail_pending = true; + try { s.file_size = fs.statSync(sessionFile).size; } catch { s.file_size = 0; } + if (s.detail_messages === undefined) s.detail_messages = 0; + if (!s.mcp_servers) s.mcp_servers = []; + if (!s.skills) s.skills = []; } } else if (!s.has_detail) { s.has_detail = false; @@ -1556,6 +2318,7 @@ function loadSessions() { } // Scan project dirs for orphan sessions (e.g. Claude Extension sessions not in history.jsonl) + // Cache-only in sync path; uncached files get queued and background job parses them. if (fs.existsSync(PROJECTS_DIR)) { try { for (const proj of fs.readdirSync(PROJECTS_DIR)) { @@ -1566,12 +2329,37 @@ function loadSessions() { const sid = file.replace('.jsonl', ''); const filePath = path.join(projDir, file); if (sessions[sid]) { - const summary = parseClaudeSessionFile(filePath); + const summary = parseClaudeSessionFile(filePath, { cacheOnly: true }); if (summary) mergeClaudeSessionDetail(sessions[sid], summary, filePath); + else sessions[sid]._detail_pending = true; + continue; + } + const summary = parseClaudeSessionFile(filePath, { cacheOnly: true }); + if (!summary) { + // Create a placeholder from stat + history of same sid (none here) + let size = 0; + try { size = fs.statSync(filePath).size; } catch {} + sessions[sid] = { + id: sid, + tool: 'claude', + project: '', + project_short: '', + first_ts: 0, + last_ts: 0, + messages: 0, + first_message: '', + has_detail: true, + file_size: size, + detail_messages: 0, + mcp_servers: [], + skills: [], + _claude_dir: CLAUDE_DIR, + _session_file: filePath, + _detail_pending: true, + worktree_original_cwd: '', + }; continue; } - const summary = parseClaudeSessionFile(filePath); - if (!summary) continue; sessions[sid] = { id: sid, tool: summary.tool, @@ -1595,6 +2383,12 @@ function loadSessions() { } catch {} } + // If any Claude files were uncached, _pendingParseFiles has been populated + // by parseClaudeSessionFile(cacheOnly: true). Kick off the warmer. + _ensureWarmingRunning(); + // Kick off one-shot SQLite backfill (noop if already running/done) + _ensureSqliteBackfillRunning(); + // Ensure all sessions have mcp_servers/skills (defaults for non-Claude) for (const s of Object.values(sessions)) { if (!s.mcp_servers) s.mcp_servers = []; @@ -1625,12 +2419,27 @@ function loadSessions() { s.date = dt.getFullYear() + '-' + String(dt.getMonth()+1).padStart(2,'0') + '-' + String(dt.getDate()).padStart(2,'0'); // Priority: worktree-state.originalCwd (container-safe) > git rev-parse > path heuristic (frontend) s.git_root = s.worktree_original_cwd || (s.project ? (_gitRootCache[s.project] || '') : ''); + // Conversation group key — sessions with the same project + initial + // prompt are treated as retries/resumes of the same conversation. Used + // by the shared groupSessions() helper in all views (Timeline, All + // Sessions, Cloud Sync) so we don't render 50 duplicate cards. + s.group_key = computeSessionGroupKey(s); } // Flag for frontend: true = cursor vscdb still loading, will have more data soon result._loading = !_cursorVscdbSessions && _cursorVscdbLoading; + // Warming state: true when background parse of session files is in progress + result._warming = _warmingStatus.running; + if (_warmingStatus.running || _warmingStatus.pending > 0) { + result._warmingProgress = { + done: _warmingStatus.done, + total: _warmingStatus.total, + phase: _warmingStatus.phase, + pending: _pendingParseFiles.size, + }; + } - // Flush disk caches + // Flush disk caches (non-force: only writes if dirty beyond threshold) _saveParsedDiskCache(); _saveGitRootDiskCache(); _updateScanMarkers(); @@ -1640,6 +2449,91 @@ function loadSessions() { return result; } +// ── Async pre-warm of parse cache + incremental change detection ───────── +// Lists all Claude session JSONL files and parses those whose cache entry +// is stale (mtime or size changed) or missing. Yields between files so the +// HTTP event loop stays responsive. Reports progress via callback. +async function loadSessionsAsync(progressCb) { + _loadParsedDiskCache(); + _loadCostDiskCache(); + + const report = (phase, done, total, extra) => { + if (typeof progressCb === 'function') { + try { progressCb({ phase, done, total, ...(extra || {}) }); } catch {} + } + }; + + // Phase 1: enumerate all Claude JSONL files (fast, just readdir) + report('scanning files', 0, 0); + const allClaudeFiles = []; + const walkClaude = (dir) => { + try { + for (const proj of fs.readdirSync(dir)) { + const projDir = path.join(dir, proj); + try { + if (!fs.statSync(projDir).isDirectory()) continue; + for (const file of fs.readdirSync(projDir)) { + if (!file.endsWith('.jsonl')) continue; + allClaudeFiles.push(path.join(projDir, file)); + } + } catch {} + } + } catch {} + }; + if (fs.existsSync(PROJECTS_DIR)) walkClaude(PROJECTS_DIR); + for (const extra of EXTRA_CLAUDE_DIRS) { + const ep = path.join(extra, 'projects'); + if (fs.existsSync(ep)) walkClaude(ep); + } + + // Phase 2: figure out which files need (re)parsing — incremental path + const toParse = []; + let cachedCount = 0; + for (const f of allClaudeFiles) { + let stat; + try { stat = fs.statSync(f); } catch { continue; } + const key = f + '|' + stat.mtimeMs + '|' + stat.size; + if (_parsedDiskCache[key]) { + _fileCacheKeyIndex[f] = key; + cachedCount++; + } else { + toParse.push({ file: f, size: stat.size }); + } + } + // Parse newest (largest mtime) first so dashboards reflect fresh data fastest + toParse.sort((a, b) => { + try { return fs.statSync(b.file).mtimeMs - fs.statSync(a.file).mtimeMs; } catch { return 0; } + }); + + const total = allClaudeFiles.length; + report('parsing session files', cachedCount, total, { cached: cachedCount, toParse: toParse.length }); + + // Phase 3: parse uncached files with yielding + let done = cachedCount; + for (const { file, size } of toParse) { + try { + if (size >= 5 * 1024 * 1024) { + await parseClaudeSessionFileAsync(file); + } else { + parseClaudeSessionFile(file); + // Tiny yield so poll requests can slot in between files + await new Promise(r => setImmediate(r)); + } + } catch {} + done++; + report('parsing session files', done, total, { cached: cachedCount, toParse: toParse.length }); + } + _saveParsedDiskCache(true); + + // Phase 4: invalidate any stale in-memory session cache, then build full + // sessions list from now-warm parse cache (sync call, fast) + _sessionsCache = null; + _sessionsCacheTs = 0; + report('aggregating sessions', total, total); + const sessions = loadSessions(); + return sessions; +} + function loadSessionDetail(sessionId, project) { const found = findSessionFile(sessionId, project); if (!found) return { error: 'Session file not found', messages: [] }; @@ -1852,7 +2746,7 @@ function exportSessionMarkdown(sessionId, project) { // Session file index: sessionId -> file path (built once, avoids O(sessions*projects) scans) let _sessionFileIndex = null; let _sessionFileIndexTs = 0; -const SESSION_FILE_INDEX_TTL = 120000; // 2 minutes — dirs rarely change +const SESSION_FILE_INDEX_TTL = 30000; // 30 seconds function _buildSessionFileIndex() { const now = Date.now(); @@ -1996,14 +2890,27 @@ function isSystemMessage(text) { if (!text) return true; var t = text.trim(); if (t === 'exit' || t === 'quit' || t === '/exit') return true; + // XML-wrapped injected context (Claude Code + Codex) if (t.startsWith('')) return true; - // Codex developer role system prompts + if (t.startsWith('')) return true; + if (t.startsWith('')) return true; + if (t.startsWith('')) return true; + if (t.startsWith('')) return true; + // Agent instruction docs and skill metadata + if (t.startsWith('# AGENTS.md')) return true; + if (t.startsWith('# CLAUDE.md')) return true; + // Codex developer/system prompts if (t.startsWith('You are Codex')) return true; if (t.startsWith('Filesystem sandboxing')) return true; + // Codex runtime nudges / auto-steering (not real user prompts) + if (t.startsWith('Warning: The maximum number of unified exec')) return true; + if (t.indexOf('AUTOSTEERING:') >= 0 && t.length < 400) return true; + // Sub-agent delegate result injection + if (t.startsWith('[Sub-agent results]')) return true; + if (t.startsWith('[sub-agent result]')) return true; return false; } @@ -2188,14 +3095,34 @@ function getSearchIndex(sessions) { function searchFullText(query, sessions) { if (!query || query.length < 2) return []; + + // Prefer the persistent SQLite FTS5 index — O(log n), no RAM bloat. + try { + const sqliteIndex = require('./sqlite-index'); + const rows = sqliteIndex.search(query, 200); + if (rows && rows.length > 0) { + // Group by session_id, keep up to 3 snippets per session + const bySession = {}; + for (const r of rows) { + if (!bySession[r.session_id]) bySession[r.session_id] = []; + if (bySession[r.session_id].length >= 3) continue; + bySession[r.session_id].push({ + role: r.role, + snippet: (r.snippet || '').replace(/<>/g, ''), + }); + } + return Object.keys(bySession).map(sid => ({ sessionId: sid, matches: bySession[sid] })); + } + } catch (e) { + // Fall through to in-memory fallback + } + + // Fallback: in-memory scan (pre-SQLite ingest completion) const q = query.toLowerCase(); const index = getSearchIndex(sessions); const results = []; - for (const entry of index) { if (entry.fullText.indexOf(q) === -1) continue; - - // Find matching messages with snippets const matches = []; for (const t of entry.texts) { if (matches.length >= 3) break; @@ -2209,12 +3136,8 @@ function searchFullText(query, sessions) { }); } } - - if (matches.length > 0) { - results.push({ sessionId: entry.sessionId, matches }); - } + if (matches.length > 0) results.push({ sessionId: entry.sessionId, matches }); } - return results; } @@ -2298,27 +3221,7 @@ function getModelPricing(model) { } // ── Compute real cost from session file token usage ──────── - -// Disk cache for computed session costs -const COST_CACHE_FILE = path.join(os.tmpdir(), 'codedash-cost-cache.json'); -let _costDiskCache = null; - -function _loadCostDiskCache() { - if (_costDiskCache) return; - try { - if (fs.existsSync(COST_CACHE_FILE)) { - _costDiskCache = JSON.parse(fs.readFileSync(COST_CACHE_FILE, 'utf8')); - } - } catch {} - if (!_costDiskCache) _costDiskCache = {}; -} - -function _saveCostDiskCache() { - if (!_costDiskCache) return; - try { - fs.writeFileSync(COST_CACHE_FILE, JSON.stringify(_costDiskCache)); - } catch {} -} +// (COST_CACHE_FILE/_costDiskCache/_loadCostDiskCache/_saveCostDiskCache defined at top) const EMPTY_COST = { cost: 0, inputTokens: 0, outputTokens: 0, cacheReadTokens: 0, cacheCreateTokens: 0, contextPctSum: 0, contextTurnCount: 0, model: '' }; @@ -2458,57 +3361,214 @@ function computeSessionCost(sessionId, project) { } const result = { cost: totalCost, inputTokens: totalInput, outputTokens: totalOutput, cacheReadTokens: totalCacheRead, cacheCreateTokens: totalCacheCreate, contextPctSum, contextTurnCount, model }; - if (cacheKey) _costDiskCache[cacheKey] = result; + if (cacheKey) { + _costDiskCache[cacheKey] = result; + _costDiskCacheDirty = true; + _costDiskCacheEntriesSinceFlush++; + if (_costDiskCacheEntriesSinceFlush >= COST_CACHE_FLUSH_EVERY) { + _saveCostDiskCache(true); + } + } _costMemCache[sessionId] = result; return result; } // ── Cost analytics ──────────────────────────────────────── -// Analytics result cache — avoids recomputing 31k sessions every request -const ANALYTICS_CACHE_FILE = path.join(os.tmpdir(), 'codedash-analytics-cache.json'); -let _analyticsCacheResult = null; -let _analyticsCacheKey = null; +// ── Incremental cost aggregator ─────────────────────────────── +// Streaming version of getCostAnalytics: create an aggregator, feed sessions +// one at a time via merge(), read a snapshot at any point via finalize(). +// Lets the background job expose live partial results to the UI. +function createCostAggregator() { + const state = { + byDay: {}, byProject: {}, byWeek: {}, byAgent: {}, + totalCost: 0, totalTokens: 0, + totalInputTokens: 0, totalOutputTokens: 0, + totalCacheReadTokens: 0, totalCacheCreateTokens: 0, + globalContextPctSum: 0, globalContextTurnCount: 0, + firstDate: null, lastDate: null, + sessionsWithData: 0, + sessionCosts: [], + agentNoCostData: {}, + seenAgents: new Set(), + processedCount: 0, + }; + return { + state, + merge(session, costData) { + state.processedCount++; + if (!state.seenAgents.has(session.tool)) { + state.seenAgents.add(session.tool); + if (!state.byAgent[session.tool]) state.byAgent[session.tool] = { cost: 0, sessions: 0, tokens: 0, estimated: false }; + } + const cost = costData.cost; + const tokens = costData.inputTokens + costData.outputTokens + costData.cacheReadTokens + costData.cacheCreateTokens; + if (cost === 0 && tokens === 0) { + if (!state.agentNoCostData[session.tool]) state.agentNoCostData[session.tool] = 0; + state.agentNoCostData[session.tool]++; + return; + } + state.sessionsWithData++; + state.totalCost += cost; + state.totalTokens += tokens; + state.totalInputTokens += costData.inputTokens; + state.totalOutputTokens += costData.outputTokens; + state.totalCacheReadTokens += costData.cacheReadTokens; + state.totalCacheCreateTokens += costData.cacheCreateTokens; + + const agent = session.tool || 'unknown'; + if (!state.byAgent[agent]) state.byAgent[agent] = { cost: 0, sessions: 0, tokens: 0, estimated: false }; + state.byAgent[agent].cost += cost; + state.byAgent[agent].sessions++; + state.byAgent[agent].tokens += tokens; + if (agent === 'codex') state.byAgent[agent].estimated = true; + if (agent === 'cursor' && costData.model && costData.model.includes('-estimated')) state.byAgent[agent].estimated = true; + if (agent === 'opencode' && !costData.model) state.byAgent[agent].estimated = true; + + state.globalContextPctSum += costData.contextPctSum; + state.globalContextTurnCount += costData.contextTurnCount; + + const day = session.date || 'unknown'; + if (session.date) { + if (!state.firstDate || session.date < state.firstDate) state.firstDate = session.date; + if (!state.lastDate || session.date > state.lastDate) state.lastDate = session.date; + } + if (!state.byDay[day]) state.byDay[day] = { cost: 0, sessions: 0, tokens: 0 }; + state.byDay[day].cost += cost; + state.byDay[day].sessions++; + state.byDay[day].tokens += tokens; + + if (session.date) { + const d = new Date(session.date); + const weekStart = new Date(d); + weekStart.setDate(d.getDate() - d.getDay()); + const weekKey = weekStart.toISOString().slice(0, 10); + if (!state.byWeek[weekKey]) state.byWeek[weekKey] = { cost: 0, sessions: 0 }; + state.byWeek[weekKey].cost += cost; + state.byWeek[weekKey].sessions++; + } + + const proj = session.project_short || session.project || 'unknown'; + if (!state.byProject[proj]) state.byProject[proj] = { cost: 0, sessions: 0, tokens: 0 }; + state.byProject[proj].cost += cost; + state.byProject[proj].sessions++; + state.byProject[proj].tokens += tokens; + + state.sessionCosts.push({ id: session.id, cost, project: proj, date: session.date, last_ts: session.last_ts || 0 }); + }, + finalize() { + // Sort top sessions by cost (snapshot a shallow copy so partials stay consistent) + const topCopy = state.sessionCosts.slice().sort((a, b) => b.cost - a.cost); + const days = state.firstDate && state.lastDate + ? Math.max(1, Math.round((new Date(state.lastDate) - new Date(state.firstDate)) / 86400000) + 1) + : 1; + const now = Date.now(); + const todayStr = new Date().toISOString().slice(0, 10); + const hoursElapsedToday = (now - new Date(todayStr).getTime()) / 3600000; + let last1hCost = 0; + let todayCost = 0; + for (const sc of topCopy) { + if (sc.last_ts >= now - 3600000) last1hCost += sc.cost; + if (sc.date === todayStr) todayCost += sc.cost; + } + return { + totalCost: state.totalCost, + totalTokens: state.totalTokens, + totalInputTokens: state.totalInputTokens, + totalOutputTokens: state.totalOutputTokens, + totalCacheReadTokens: state.totalCacheReadTokens, + totalCacheCreateTokens: state.totalCacheCreateTokens, + avgContextPct: state.globalContextTurnCount > 0 ? Math.round(state.globalContextPctSum / state.globalContextTurnCount) : 0, + dailyRate: state.totalCost / days, + firstDate: state.firstDate, + lastDate: state.lastDate, + days, + totalSessions: state.sessionsWithData, + byDay: state.byDay, + byWeek: state.byWeek, + byProject: state.byProject, + topSessions: topCopy.slice(0, 10), + byAgent: state.byAgent, + agentNoCostData: state.agentNoCostData, + last1hCost, + todayCost, + hoursElapsedToday: Math.max(1, hoursElapsedToday), + processedCount: state.processedCount, + }; + }, + }; +} -function _analyticsKey(sessions) { - // Key: session count + newest session mtime - let newest = 0; - for (const s of sessions) { - if (s.last_ts > newest) newest = s.last_ts; +// Compute per-session cost respecting special per-tool paths used by +// getCostAnalytics (opencode batch, cursor vscdb tokens, ...). +function computeSessionCostForAnalytics(session, opencodeCostCache) { + if (session.tool === 'opencode' && opencodeCostCache && opencodeCostCache[session.id]) { + return opencodeCostCache[session.id]; } - return sessions.length + ':' + newest; + if (session.tool === 'cursor') { + const inp = session._cursor_input_tokens || 0; + const out = session._cursor_output_tokens || 0; + if (inp > 0 || out > 0) { + const model = session._cursor_model || ''; + const pricing = getModelPricing(model); + return { cost: inp * pricing.input + out * pricing.output, inputTokens: inp, outputTokens: out, cacheReadTokens: 0, cacheCreateTokens: 0, contextPctSum: 0, contextTurnCount: 0, model: model }; + } + if (session.user_messages > 0 || session.messages > 0) { + const userMsgs = session.user_messages || Math.ceil((session.messages || 0) * 0.07); + const model = session._cursor_model || 'claude-sonnet'; + const pricing = getModelPricing(model); + const estInput = userMsgs * 2000; + const estOutput = userMsgs * 1000; + return { cost: estInput * pricing.input + estOutput * pricing.output, inputTokens: estInput, outputTokens: estOutput, cacheReadTokens: 0, cacheCreateTokens: 0, contextPctSum: 0, contextTurnCount: 0, model: model + '-estimated' }; + } + return EMPTY_COST; + } + return computeSessionCost(session.id, session.project); } -function getCostAnalytics(sessions) { - // Fast cache check — if sessions haven't changed, return cached result - const key = _analyticsKey(sessions); - if (_analyticsCacheResult && _analyticsCacheKey === key) return _analyticsCacheResult; - - // Try disk cache - if (!_analyticsCacheResult) { - try { - if (fs.existsSync(ANALYTICS_CACHE_FILE)) { - const cached = JSON.parse(fs.readFileSync(ANALYTICS_CACHE_FILE, 'utf8')); - if (cached._key === key) { - _analyticsCacheResult = cached.data; - _analyticsCacheKey = key; - return cached.data; - } +// Pre-compute OpenCode costs in one batch SQL — used by the streaming path +function buildOpencodeCostCache(sessions) { + const cache = {}; + const opencodeSessions = sessions.filter(s => s.tool === 'opencode'); + if (opencodeSessions.length === 0 || !fs.existsSync(OPENCODE_DB)) return cache; + try { + const batchRows = execFileSync('sqlite3', [ + OPENCODE_DB, + `SELECT session_id, data FROM message WHERE json_extract(data, '$.role') = 'assistant' ORDER BY time_created` + ], { encoding: 'utf8', timeout: 30000, windowsHide: true }).trim(); + if (batchRows) { + for (const row of batchRows.split('\n')) { + const sepIdx = row.indexOf('|'); + if (sepIdx < 0) continue; + const sessId = row.slice(0, sepIdx); + const jsonStr = row.slice(sepIdx + 1); + try { + const msgData = JSON.parse(jsonStr); + const t = msgData.tokens || {}; + const inp = t.input || 0; + const out = (t.output || 0) + (t.reasoning || 0); + const cacheRead = (t.cache && t.cache.read) || 0; + const cacheCreate = (t.cache && t.cache.write) || 0; + if (inp === 0 && out === 0) continue; + if (!cache[sessId]) cache[sessId] = { cost: 0, inputTokens: 0, outputTokens: 0, cacheReadTokens: 0, cacheCreateTokens: 0, contextPctSum: 0, contextTurnCount: 0, model: '' }; + const c = cache[sessId]; + if (!c.model && msgData.modelID) c.model = msgData.modelID; + const pricing = getModelPricing(msgData.modelID || c.model); + c.inputTokens += inp; + c.outputTokens += out; + c.cacheReadTokens += cacheRead; + c.cacheCreateTokens += cacheCreate; + c.cost += inp * pricing.input + cacheCreate * pricing.cache_create + cacheRead * pricing.cache_read + out * pricing.output; + const ctx = inp + cacheCreate + cacheRead; + if (ctx > 0) { c.contextPctSum += (ctx / CONTEXT_WINDOW) * 100; c.contextTurnCount++; } + } catch {} } - } catch {} - } - - const result = _computeCostAnalytics(sessions); - - // Save to cache - _analyticsCacheResult = result; - _analyticsCacheKey = key; - try { fs.writeFileSync(ANALYTICS_CACHE_FILE, JSON.stringify({ _key: key, data: result })); } catch {} - - return result; + } + } catch {} + return cache; } -function _computeCostAnalytics(sessions) { +function getCostAnalytics(sessions) { const byDay = {}; const byProject = {}; const byWeek = {}; @@ -2675,7 +3735,8 @@ function _computeCostAnalytics(sessions) { if (sc.date === todayStr) todayCost += sc.cost; } - _saveCostDiskCache(); + _saveCostDiskCache(true); + _saveParsedDiskCache(true); return { totalCost, @@ -2704,13 +3765,63 @@ function _computeCostAnalytics(sessions) { // ── Active sessions detection ───────────────────────────── +// ── Active sessions (cached, non-blocking) ────────────────── +// getActiveSessions() is called from /api/active which the browser polls on +// a timer. The previous version shelled out to `ps` + `lsof` synchronously +// once per matching process, doing an O(N) sync scan that blocked the event +// loop for tens of seconds when many processes had "codex"/"claude" in their +// cmdline (e.g. codex-up-exec wrapper spawned by Claude Code tooling). +// +// New design: +// 1. Cached result served for up to 3 seconds. +// 2. Tighter cmd matching — must be a real agent CLI, not a substring hit. +// 3. pid→cwd is remembered across calls (pids are stable for the process +// lifetime). We only look up cwd for pids we've never seen. +// 4. lsof is run ONCE as a single batch call for all unknown pids, with +// a hard 2-second total timeout. +// 5. No inner loadSessions() calls; session matching uses the in-memory +// sessions cache if it already exists, otherwise cwd-match is skipped +// and a background refresh fires. +let _activeCache = null; +let _activeCacheTs = 0; +const ACTIVE_CACHE_TTL = 3000; // 3 seconds +const _pidCwdCache = new Map(); // pid → cwd (stable for process lifetime) + +// Real agent CLI invocations — tighter than a substring match. +// Matches the binary name at the START of cmd (or after a /path/). +// Includes codex-up variants (user wrapper that spawns codex exec). +const AGENT_CLI_MATCHERS = [ + { tool: 'claude', re: /(^|\/)claude(\s|$)/ }, + { tool: 'codex', re: /(^|\/)codex(-up(-exec)?)?(\s|$)/ }, + { tool: 'opencode',re: /(^|\/)opencode(\s|$)/ }, + { tool: 'kiro', re: /(^|\/)kiro-cli(\s|$)/ }, + { tool: 'cursor', re: /(^|\/)cursor-agent(\s|$)/ }, +]; + +function _matchAgentCli(cmd) { + for (const m of AGENT_CLI_MATCHERS) { + if (m.re.test(cmd)) return m.tool; + } + return ''; +} + function getActiveSessions() { + const now = Date.now(); + if (_activeCache && (now - _activeCacheTs) < ACTIVE_CACHE_TTL) { + return _activeCache; + } const active = []; - const seenPids = new Set(); - // 1. Claude Code — read PID files for session ID mapping + // Skip on Windows + if (process.platform === 'win32') { + _activeCache = active; + _activeCacheTs = now; + return active; + } + + // 1. Read Claude Code PID files (provides cwd + sessionId directly) const sessionsDir = path.join(CLAUDE_DIR, 'sessions'); - const claudePidMap = {}; // pid → {sessionId, cwd, startedAt} + const claudePidMap = {}; if (fs.existsSync(sessionsDir)) { for (const file of fs.readdirSync(sessionsDir)) { if (!file.endsWith('.json')) continue; @@ -2721,104 +3832,129 @@ function getActiveSessions() { } } - // 2. Scan ALL agent processes via ps - const agentPatterns = [ - { pattern: 'claude', tool: 'claude', match: /\bclaude\b/ }, - { pattern: 'codex', tool: 'codex', match: /\bcodex\b/ }, - { pattern: 'opencode', tool: 'opencode', match: /\bopencode\b/ }, - { pattern: 'kiro', tool: 'kiro', match: /kiro-cli/ }, - { pattern: 'cursor-agent', tool: 'cursor', match: /cursor-agent/ }, - ]; - - // Skip process scanning on Windows (no ps/grep) - if (process.platform === 'win32') return active; - + // 2. Single `ps` call + let psOut = ''; try { - const psOut = execSync( - 'ps aux 2>/dev/null | grep -E "claude|codex|opencode|kiro-cli|cursor-agent" | grep -v grep || true', - { encoding: 'utf8', timeout: 3000, stdio: ['pipe', 'pipe', 'pipe'] } + psOut = execSync( + 'ps -eo pid=,pcpu=,rss=,stat=,command= 2>/dev/null', + { encoding: 'utf8', timeout: 2000, stdio: ['pipe', 'pipe', 'pipe'], maxBuffer: 4 * 1024 * 1024 } ); + } catch { + _activeCache = active; + _activeCacheTs = now; + return active; + } - for (const line of psOut.split('\n').filter(Boolean)) { - const parts = line.trim().split(/\s+/); - if (parts.length < 11) continue; - - const pid = parseInt(parts[1]); - if (seenPids.has(pid)) continue; + // 3. Parse + filter with strict agent CLI matching + const matches = []; + const livePids = new Set(); + for (const line of psOut.split('\n')) { + if (!line) continue; + const m = line.match(/^\s*(\d+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(.*)$/); + if (!m) continue; + const pid = parseInt(m[1]); + if (!pid) continue; + livePids.add(pid); + const cpu = parseFloat(m[2]) || 0; + const rss = parseInt(m[3]) || 0; + const stat = m[4] || ''; + const cmd = m[5] || ''; + + // Skip wrappers + if (cmd.includes('node bin/cli') || cmd.includes('/codedash') || cmd.includes('npm ')) continue; + if (cmd.startsWith('grep ') || cmd.includes(' grep ')) continue; + + const tool = _matchAgentCli(cmd); + if (!tool) continue; + + matches.push({ pid, cpu, rss, stat, cmd, tool }); + } - const cpu = parseFloat(parts[2]) || 0; - const rss = parseInt(parts[5]) || 0; - const stat = parts[7] || ''; - const cmd = parts.slice(10).join(' '); + // Drop stale pids from cwd cache + for (const pid of Array.from(_pidCwdCache.keys())) { + if (!livePids.has(pid)) _pidCwdCache.delete(pid); + } - // Determine tool - let tool = ''; - for (const ap of agentPatterns) { - if (ap.match.test(cmd)) { tool = ap.tool; break; } + // 4. Collect unknown pids (not in PID files, not in cache). Look up cwd + // via a SINGLE batch lsof call with a hard 1-second timeout. + const unknownPids = []; + for (const m of matches) { + if (claudePidMap[m.pid] && claudePidMap[m.pid].cwd) continue; + if (_pidCwdCache.has(m.pid)) continue; + unknownPids.push(m.pid); + } + if (unknownPids.length > 0) { + try { + // -a ANDs the -d/-p conditions (without it, lsof ORs and returns cwd for + // all processes, not just the requested pids). + const out = execSync( + `lsof -a -d cwd -Fpn -p ${unknownPids.join(',')} 2>/dev/null`, + { encoding: 'utf8', timeout: 1500, stdio: ['pipe', 'pipe', 'pipe'], maxBuffer: 1 * 1024 * 1024 } + ); + // lsof -Fpn output: p\nn\np\nn... + let curPid = 0; + for (const line of out.split('\n')) { + if (line.startsWith('p')) { + curPid = parseInt(line.slice(1)) || 0; + } else if (line.startsWith('n') && curPid) { + _pidCwdCache.set(curPid, line.slice(1)); + } } - if (!tool) continue; - - // Skip node/npm/shell wrappers — only main processes - if (cmd.includes('node bin/cli') || cmd.includes('npm') || cmd.includes('grep')) continue; - - seenPids.add(pid); - - // Get session ID from Claude PID files - let sessionId = ''; - let cwd = ''; - let startedAt = 0; - let sessionSource = ''; - if (claudePidMap[pid]) { - sessionId = claudePidMap[pid].sessionId || ''; - cwd = claudePidMap[pid].cwd || ''; - startedAt = claudePidMap[pid].startedAt || 0; - if (sessionId) sessionSource = 'pid-file'; + // Mark pids we asked about but didn't get cwd for — empty string so we + // don't keep asking on every poll. + for (const pid of unknownPids) { + if (!_pidCwdCache.has(pid)) _pidCwdCache.set(pid, ''); } + } catch { + // Timeout / error: mark all as unknown so we don't retry each poll + for (const pid of unknownPids) _pidCwdCache.set(pid, ''); + } + } - // Try to get cwd from lsof if not from PID file - if (!cwd) { - try { - const lsofOut = execSync(`lsof -d cwd -p ${pid} -Fn 2>/dev/null`, { encoding: 'utf8', timeout: 2000, stdio: ['pipe', 'pipe', 'pipe'] }); - const match = lsofOut.match(/\nn(\/[^\n]+)/); - if (match) cwd = match[1]; - } catch {} - } + // 5. Assemble results using only already-loaded sessions cache for cwd-match. + // Never call loadSessions() here — it may invalidate and re-parse. + const cachedSessions = _sessionsCache || []; + + for (const m of matches) { + let sessionId = ''; + let cwd = ''; + let startedAt = 0; + let sessionSource = ''; + + if (claudePidMap[m.pid]) { + sessionId = claudePidMap[m.pid].sessionId || ''; + cwd = claudePidMap[m.pid].cwd || ''; + startedAt = claudePidMap[m.pid].startedAt || 0; + if (sessionId) sessionSource = 'pid-file'; + } + if (!cwd) cwd = _pidCwdCache.get(m.pid) || ''; - // Try to find session ID by matching cwd + tool to loaded sessions - if (!sessionId) { - const allS = loadSessions(); - const match = allS.find(s => s.tool === tool && s.project === cwd); - if (match) { - sessionId = match.id; - sessionSource = 'cwd-match'; - } - // If still no match, find latest session of this tool - if (!sessionId) { - const latest = allS.filter(s => s.tool === tool).sort((a,b) => b.last_ts - a.last_ts)[0]; - if (latest) { - sessionId = latest.id; - sessionSource = 'fallback-latest'; - } - } + if (!sessionId && cachedSessions.length > 0) { + const match = cachedSessions.find(s => s.tool === m.tool && s.project === cwd); + if (match) { + sessionId = match.id; + sessionSource = 'cwd-match'; } - - const status = cpu < 1 && (stat.includes('S') || stat.includes('T')) ? 'waiting' : 'active'; - - active.push({ - pid: pid, - sessionId: sessionId, - cwd: cwd, - startedAt: startedAt, - kind: tool, - entrypoint: tool, - status: status, - cpu: cpu, - memoryMB: Math.round(rss / 1024), - _sessionSource: sessionSource, - }); } - } catch {} + const status = m.cpu < 1 && (m.stat.includes('S') || m.stat.includes('T')) ? 'waiting' : 'active'; + + active.push({ + pid: m.pid, + sessionId, + cwd, + startedAt, + kind: m.tool, + entrypoint: m.tool, + status, + cpu: m.cpu, + memoryMB: Math.round(m.rss / 1024), + _sessionSource: sessionSource, + }); + } + + _activeCache = active; + _activeCacheTs = Date.now(); return active; } @@ -2852,7 +3988,8 @@ const fmtLocalDay = (ts) => { }; // Disk cache for per-session daily message breakdown -const DAILY_STATS_CACHE_FILE = path.join(os.tmpdir(), 'codedash-daily-stats-cache.json'); +const DAILY_STATS_CACHE_FILE = path.join(CODEDASH_CACHE_DIR, 'daily-stats-cache.json'); +const LEGACY_DAILY_STATS_CACHE_FILE = path.join(os.tmpdir(), 'codedash-daily-stats-cache.json'); let _dailyStatsDiskCache = null; function _loadDailyStatsDiskCache() { @@ -2860,6 +3997,8 @@ function _loadDailyStatsDiskCache() { try { if (fs.existsSync(DAILY_STATS_CACHE_FILE)) { _dailyStatsDiskCache = JSON.parse(fs.readFileSync(DAILY_STATS_CACHE_FILE, 'utf8')); + } else if (fs.existsSync(LEGACY_DAILY_STATS_CACHE_FILE)) { + _dailyStatsDiskCache = JSON.parse(fs.readFileSync(LEGACY_DAILY_STATS_CACHE_FILE, 'utf8')); } } catch {} if (!_dailyStatsDiskCache) _dailyStatsDiskCache = {}; @@ -2868,7 +4007,9 @@ function _loadDailyStatsDiskCache() { function _saveDailyStatsDiskCache() { if (!_dailyStatsDiskCache) return; try { - fs.writeFileSync(DAILY_STATS_CACHE_FILE, JSON.stringify(_dailyStatsDiskCache)); + const tmp = DAILY_STATS_CACHE_FILE + '.tmp'; + fs.writeFileSync(tmp, JSON.stringify(_dailyStatsDiskCache)); + fs.renameSync(tmp, DAILY_STATS_CACHE_FILE); } catch {} } @@ -2920,37 +4061,7 @@ function _computeSessionDailyBreakdown(s, found) { return { msgsByDay, tsByDay }; } -// Daily stats result cache -const DAILY_RESULT_CACHE_FILE = path.join(os.tmpdir(), 'codedash-daily-result-cache.json'); -let _dailyResultCache = null; -let _dailyResultCacheKey = null; - function getDailyStats(sessions) { - const key = _analyticsKey(sessions); - if (_dailyResultCache && _dailyResultCacheKey === key) return _dailyResultCache; - - // Try disk cache - if (!_dailyResultCache) { - try { - if (fs.existsSync(DAILY_RESULT_CACHE_FILE)) { - const cached = JSON.parse(fs.readFileSync(DAILY_RESULT_CACHE_FILE, 'utf8')); - if (cached._key === key) { - _dailyResultCache = cached.data; - _dailyResultCacheKey = key; - return cached.data; - } - } - } catch {} - } - - const result = _computeDailyStats(sessions); - _dailyResultCache = result; - _dailyResultCacheKey = key; - try { fs.writeFileSync(DAILY_RESULT_CACHE_FILE, JSON.stringify({ _key: key, data: result })); } catch {} - return result; -} - -function _computeDailyStats(sessions) { const byDay = {}; const ensureDay = (date) => { if (!byDay[date]) byDay[date] = { date, sessions: 0, messages: 0, hours: 0, cost: 0, agents: {} }; @@ -3005,12 +4116,10 @@ function _computeDailyStats(sessions) { const day = s.date || fmtLocalDay(s.last_ts); const d = ensureDay(day); d.sessions++; - // Use exact user_messages count if available, otherwise estimate + // Only count EXACT user_messages. The 0.5 estimate was wildly wrong + // because Claude type=user entries include tool_results (up to 28x inflation). if (s.user_messages > 0) { d.messages += s.user_messages; - } else { - const totalMsgEst = s.detail_messages || s.messages || 0; - d.messages += Math.ceil(totalMsgEst * 0.5); } d.hours += Math.min((s.last_ts - s.first_ts) / 3600000, 16); d.cost += sessionCost; @@ -3023,6 +4132,8 @@ function _computeDailyStats(sessions) { d.cost = Math.round(d.cost * 100) / 100; } _saveDailyStatsDiskCache(); + _saveCostDiskCache(true); + _saveParsedDiskCache(true); return Object.values(byDay).sort((a, b) => b.date.localeCompare(a.date)); } @@ -3083,6 +4194,12 @@ function getLeaderboardStats() { module.exports = { loadSessions, + loadSessionsAsync, + getWarmingStatus, + getSqliteBackfillStatus, + createCostAggregator, + computeSessionCostForAnalytics, + buildOpencodeCostCache, loadSessionDetail, getProjectGitInfo, getLeaderboardStats, diff --git a/src/frontend/analytics.js b/src/frontend/analytics.js index d414be9..78f94f8 100644 --- a/src/frontend/analytics.js +++ b/src/frontend/analytics.js @@ -9,9 +9,43 @@ async function renderAnalytics(container) { if (dateFrom) params.push('from=' + dateFrom); if (dateTo) params.push('to=' + dateTo); if (params.length) url += '?' + params.join('&'); - var resp = await fetch(url); - var data = await resp.json(); + // Poll loop — server runs a background job, we render live partial snapshots + var pollStart = Date.now(); + var data = null; + while (true) { + var resp = await fetch(url); + var payload = await resp.json(); + if (payload.status === 'done') { data = payload; break; } + if (payload.status === 'error') { + container.innerHTML = '
Analytics failed: ' + escHtml(payload.error || 'unknown') + '
'; + return; + } + var p = (payload.progress || {}); + var done = p.done || 0, total = p.total || 0; + var pct = total > 0 ? Math.round(done / total * 100) : 0; + var phase = p.phase || 'working'; + var elapsed = Math.round((payload.elapsedMs || (Date.now() - pollStart)) / 1000); + if (payload.partialResult && payload.partialResult.totalSessions > 0) { + renderAnalyticsUI(container, payload.partialResult, { + live: true, done: done, total: total, pct: pct, phase: phase, elapsed: elapsed, + }); + } else { + container.innerHTML = + '
' + + '
Computing cost analytics…
' + + '
' + escHtml(phase) + ' — ' + done + ' / ' + total + ' sessions (' + pct + '%)
' + + '
' + + '
elapsed ' + elapsed + 's · cached for next time
' + + '
'; + } + await new Promise(function(r){ setTimeout(r, 500); }); + } + + renderAnalyticsUI(container, data, { live: false }); + return; + /* === below is the original inline render block; kept so the upstream + split structure is preserved and renderAnalyticsUI uses the same code path === */ var html = '
'; html += '

Cost Analytics

'; @@ -227,6 +261,145 @@ async function renderAnalytics(container) { } } +// Renders the full Cost Analytics UI from either partial (live) or final data. +// When opts.live is true, prepends a sticky progress banner. +function renderAnalyticsUI(container, data, opts) { + opts = opts || {}; + var html = '
'; + + if (opts.live) { + var pct = opts.pct || 0; + html += '
'; + html += '
Computing cost analytics — live' + (opts.elapsed || 0) + 's
'; + html += '
' + escHtml(opts.phase || '') + ' — ' + (opts.done || 0) + ' / ' + (opts.total || 0) + ' sessions (' + pct + '%) · numbers update as sessions are processed
'; + html += '
'; + html += '
'; + } + + html += '

Cost Analytics' + (opts.live ? ' (live)' : '') + '

'; + + // Summary cards + html += '
'; + html += '
$' + (data.totalCost || 0).toFixed(2) + 'Total cost (API-equivalent)
'; + html += '
' + formatTokens(data.totalTokens || 0) + 'Total tokens
'; + html += '
$' + (data.dailyRate || 0).toFixed(2) + 'Avg per day (' + (data.days || 1) + ' days)
'; + html += '
' + (data.totalSessions || 0) + 'Sessions
'; + html += '
'; + + // Burn rate + var todayCost = data.todayCost || 0; + var last1hCost = data.last1hCost || 0; + var dailyRate = data.dailyRate || 0; + var hoursElapsed = data.hoursElapsedToday || 1; + var projectedDaily = todayCost / (hoursElapsed / 24); + var paceRatio = dailyRate > 0 ? projectedDaily / dailyRate : 0; + var burnClass = paceRatio >= 2 ? 'burn-high' : paceRatio >= 1.3 ? 'burn-medium' : 'burn-low'; + var paceLabel = paceRatio >= 2 ? '🔥 ' + Math.round(paceRatio) + 'x avg' : paceRatio >= 1.3 ? '↑ ' + paceRatio.toFixed(1) + 'x avg' : dailyRate > 0 ? '✓ normal' : ''; + html += '
'; + html += '
Burn Rate
'; + html += '
'; + html += '
$' + todayCost.toFixed(3) + 'today' + (paceLabel ? '' + paceLabel + '' : '') + '
'; + html += '
$' + last1hCost.toFixed(3) + 'last hour
'; + if (dailyRate > 0) { + html += '
$' + projectedDaily.toFixed(2) + 'projected today
'; + } + html += '
'; + + // Token breakdown + if (data.totalInputTokens !== undefined) { + var totalTok = (data.totalInputTokens || 0) + (data.totalOutputTokens || 0) + (data.totalCacheReadTokens || 0) + (data.totalCacheCreateTokens || 0); + var pctOf = function(n) { return totalTok > 0 ? Math.round(n / totalTok * 100) : 0; }; + html += '
'; + html += '

Token Breakdown

'; + html += '
'; + html += '
' + formatTokens(data.totalInputTokens || 0) + 'Input' + pctOf(data.totalInputTokens || 0) + '%
'; + html += '
' + formatTokens(data.totalOutputTokens || 0) + 'Output' + pctOf(data.totalOutputTokens || 0) + '%
'; + html += '
' + formatTokens(data.totalCacheReadTokens || 0) + 'Cache read' + pctOf(data.totalCacheReadTokens || 0) + '%
'; + html += '
' + formatTokens(data.totalCacheCreateTokens || 0) + 'Cache write' + pctOf(data.totalCacheCreateTokens || 0) + '%
'; + if (data.avgContextPct > 0) { + html += '
' + data.avgContextPct + '%Avg context usedof 200K
'; + } + html += '
'; + } + + // Cost by agent + var agentEntries = Object.entries(data.byAgent || {}).filter(function(e) { return e[1].sessions > 0; }); + if (agentEntries.length >= 1) { + agentEntries.sort(function(a, b) { return b[1].cost - a[1].cost; }); + html += '

Cost by Agent

'; + html += '
'; + var maxAgentCost = agentEntries[0][1].cost || 1; + agentEntries.forEach(function(entry) { + var name = entry[0]; var info = entry[1]; + var p = maxAgentCost > 0 ? (info.cost / maxAgentCost * 100) : 0; + var label = { 'claude': 'Claude Code', 'claude-ext': 'Claude Ext', 'codex': 'Codex', 'opencode': 'OpenCode', 'cursor': 'Cursor', 'kiro': 'Kiro' }[name] || name; + var estMark = info.estimated ? ' ~est.' : ''; + html += '
'; + html += '' + label + estMark + ''; + html += '
'; + html += '$' + info.cost.toFixed(2) + ' (' + info.sessions + ' sess.)'; + html += '
'; + }); + html += '
'; + } + + // Daily cost chart (last 30 days) + var dayKeys = Object.keys(data.byDay || {}).sort(); + var last30 = dayKeys.slice(-30); + if (last30.length > 0) { + var maxCost = Math.max.apply(null, last30.map(function(d) { return data.byDay[d].cost; })); + html += '

Daily Cost (last 30 days)

'; + html += '
'; + last30.forEach(function(d) { + var c = data.byDay[d]; + var p = maxCost > 0 ? (c.cost / maxCost * 100) : 0; + var label = d.slice(5); + html += '
'; + html += '
'; + html += '
' + label + '
'; + html += '
'; + }); + html += '
'; + } + + // Cost by project + var projects = Object.entries(data.byProject || {}).sort(function(a, b) { return b[1].cost - a[1].cost; }); + var topProjects = projects.slice(0, 10); + if (topProjects.length > 0) { + var maxProjCost = topProjects[0][1].cost || 1; + html += '

Cost by Project

'; + html += '
'; + topProjects.forEach(function(entry) { + var name = entry[0]; var info = entry[1]; + var p = maxProjCost > 0 ? (info.cost / maxProjCost * 100) : 0; + html += '
'; + html += '' + escHtml(name) + ''; + html += '
'; + html += '$' + info.cost.toFixed(2) + ''; + html += '
'; + }); + html += '
'; + } + + // Top expensive sessions + if (data.topSessions && data.topSessions.length > 0) { + html += '

Most Expensive Sessions

'; + html += '
'; + data.topSessions.forEach(function(s) { + html += '
'; + html += '$' + s.cost.toFixed(2) + ''; + html += '' + escHtml(s.project || '') + ''; + html += '' + (s.date || '') + ''; + html += '' + (s.id || '').slice(0, 8) + ''; + html += '
'; + }); + html += '
'; + } + + html += '
'; + container.innerHTML = html; +} + function formatTokens(n) { if (n >= 1000000) return (n / 1000000).toFixed(1) + 'M'; if (n >= 1000) return (n / 1000).toFixed(0) + 'K'; diff --git a/src/frontend/app.js b/src/frontend/app.js index 6fd757a..87eaf97 100644 --- a/src/frontend/app.js +++ b/src/frontend/app.js @@ -691,6 +691,9 @@ function renderCard(s, idx) { } html += '
'; html += '' + s.messages + ' msgs'; + if (s._group_extra > 0) { + html += '+' + s._group_extra + ' more (' + s._group_total_msgs + ' total msgs)'; + } if (s.file_size) { html += '' + formatBytes(s.file_size) + ''; } @@ -968,8 +971,14 @@ function render() { } var renderFn = layout === 'list' ? renderListCard : renderCard; - var visible = sessions.slice(0, renderLimit); - var hasMore = sessions.length > renderLimit; + + // Dedupe conversation groups first (same project+initial prompt collapses + // into a single representative card with "+N more" badge). Shared helper + // used by Timeline / Cloud Sync too. + var conversationGroups = groupSessionsByConversation(sessions); + var dedupedSessions = conversationGroups.map(function(g) { return withGroupBadge(g.representative, g); }); + var visible = dedupedSessions.slice(0, renderLimit); + var hasMore = dedupedSessions.length > renderLimit; if (grouped) { renderGrouped(content, visible, renderFn); @@ -980,7 +989,7 @@ function render() { } if (hasMore) { - content.innerHTML += '
'; + content.innerHTML += '
'; } } @@ -1024,14 +1033,64 @@ function renderGrouped(container, sessions, renderFn) { container.innerHTML = html; } +// ── Shared session grouping ─────────────────────────────────── +// Collapses near-duplicate sessions (same project + same initial prompt) into +// a single "conversation group". Representative is the newest session; the +// rest are attached as `group_members` so the UI can show "+N more" and let +// the user drill down. Used by Timeline / All Sessions / Cloud Sync views — +// all call this helper, no per-view dedup logic. +function groupSessionsByConversation(sessions) { + var groups = {}; + var order = []; + sessions.forEach(function(s) { + var k = s.group_key || s.id; + if (!groups[k]) { + groups[k] = { + key: k, + representative: s, + members: [s], + total_msgs: s.messages || 0, + total_file_size: s.file_size || 0, + }; + order.push(k); + return; + } + var g = groups[k]; + g.members.push(s); + g.total_msgs += s.messages || 0; + g.total_file_size += s.file_size || 0; + // Newest last_ts wins as representative + if (s.last_ts > g.representative.last_ts) g.representative = s; + }); + return order.map(function(k) { return groups[k]; }); +} + +// Decorate a session with group metadata so renderCard can show "+N more". +function withGroupBadge(s, group) { + if (!group || group.members.length <= 1) return s; + var extra = group.members.length - 1; + // Clone shallow so we don't mutate the original + return Object.assign({}, s, { + _group_count: group.members.length, + _group_extra: extra, + _group_total_msgs: group.total_msgs, + _group_members: group.members, + }); +} + function renderTimeline(container, sessions) { - // Group by date + // Group by date, then within each date dedupe near-duplicate conversations var byDate = {}; sessions.forEach(function(s) { var d = s.date || 'unknown'; if (!byDate[d]) byDate[d] = []; byDate[d].push(s); }); + // Replace each day's flat list with grouped representatives + Object.keys(byDate).forEach(function(d) { + var groups = groupSessionsByConversation(byDate[d]); + byDate[d] = groups.map(function(g) { return withGroupBadge(g.representative, g); }); + }); var dates = Object.keys(byDate).sort().reverse(); if (dates.length === 0) { @@ -1069,6 +1128,9 @@ function renderQACard(s, idx) { html += '' + escHtml((s.first_message || '').slice(0, 160)) + ''; html += ''; html += '' + s.messages + ' msgs'; + if (s._group_extra > 0) { + html += '+' + s._group_extra + ''; + } if (costStr) html += '' + costStr + ''; html += '' + timeAgo(s.last_ts) + ''; html += ''; @@ -1098,17 +1160,27 @@ function renderProjects(container, sessions) { var html = '
'; sorted.forEach(function(entry) { var name = entry[0]; - var list = entry[1].slice().sort(function(a, b) { return b.last_ts - a.last_ts; }); + var rawList = entry[1]; + // Dedupe conversation groups within the project (same first prompt + // collapses into one representative card with "+N more" badge). + var convGroups = groupSessionsByConversation(rawList); + var list = convGroups + .map(function(g) { return withGroupBadge(g.representative, g); }) + .sort(function(a, b) { return b.last_ts - a.last_ts; }); + var color = getProjectColor(name); - var totalMsgs = list.reduce(function(s, e) { return s + (e.messages || 0); }, 0); - var totalCost = list.reduce(function(s, e) { return s + estimateCost(e.file_size); }, 0); + var totalMsgs = rawList.reduce(function(s, e) { return s + (e.messages || 0); }, 0); + var totalCost = rawList.reduce(function(s, e) { return s + estimateCost(e.file_size); }, 0); var costLabel = totalCost > 0 ? ' · ~$' + totalCost.toFixed(2) : ''; + var dedupLabel = rawList.length !== list.length + ? ' (' + rawList.length + ' raw → ' + list.length + ' unique)' + : ''; html += '
'; html += '
'; html += ''; html += '' + escHtml(name) + ''; - html += '' + list.length + ' sessions · ' + totalMsgs + ' msgs' + escHtml(costLabel) + ''; + html += '' + list.length + ' sessions' + dedupLabel + ' · ' + totalMsgs + ' msgs' + escHtml(costLabel) + ''; html += '▼'; html += '
'; html += '
'; diff --git a/src/frontend/cloud.js b/src/frontend/cloud.js index 80ba19a..54cfaa2 100644 --- a/src/frontend/cloud.js +++ b/src/frontend/cloud.js @@ -82,10 +82,17 @@ async function renderCloud(container) { html += '
'; if (cloudLocalSessions) { - for (var i = 0; i < cloudLocalSessions.length; i++) { - var ls = cloudLocalSessions[i]; + // Dedupe near-duplicate conversations via the shared helper — same + // logic as Timeline, All Sessions etc. + var _groups = groupSessionsByConversation(cloudLocalSessions); + for (var i = 0; i < _groups.length; i++) { + var g = _groups[i]; + var ls = g.representative; + var extra = g.members.length - 1; var inCloud = cloudSessionIds.has(ls.id); - var sub = (ls.project_short || '') + ' \u00b7 ' + (ls.messages || 0) + ' msgs'; + var subParts = [(ls.project_short || ''), (ls.messages || 0) + ' msgs']; + if (extra > 0) subParts.push('+' + extra + ' more (' + g.total_msgs + ' total)'); + var sub = subParts.join(' \u00b7 '); var btnHtml = inCloud ? 'in cloud' : ''; diff --git a/src/frontend/heatmap.js b/src/frontend/heatmap.js index 17eaef8..7b50d4e 100644 --- a/src/frontend/heatmap.js +++ b/src/frontend/heatmap.js @@ -11,10 +11,16 @@ function renderHeatmap(container) { var now = new Date(); var oneYearAgo = new Date(now.getFullYear() - 1, now.getMonth(), now.getDate()); - // Count sessions per day + by tool + // Dedupe sessions into conversation groups so retry/resume chains count + // as ONE session. Otherwise a `codex exec` loop looks like 1000 sessions + // in a single day. Uses the shared helper from app.js. + var dedupedGroups = groupSessionsByConversation(allSessions); + var dedupedSessions = dedupedGroups.map(function(g) { return g.representative; }); + + // Count sessions per day + by tool (deduped) var counts = {}; var toolCounts = {}; - allSessions.forEach(function(s) { + dedupedSessions.forEach(function(s) { var d = s.date; if (!d) return; counts[d] = (counts[d] || 0) + 1; diff --git a/src/frontend/leaderboard.js b/src/frontend/leaderboard.js index 6139e13..73f6212 100644 --- a/src/frontend/leaderboard.js +++ b/src/frontend/leaderboard.js @@ -170,8 +170,36 @@ async function githubLogout() { async function renderLeaderboard(container) { container.innerHTML = '
Loading stats...
'; try { - var resp = await fetch('/api/leaderboard'); - var data = await resp.json(); + // Poll loop with live partial — same pattern as analytics + var pollStart = Date.now(); + var data = null; + while (true) { + var resp = await fetch('/api/leaderboard'); + var payload = await resp.json(); + if (payload.status === 'done') { data = payload; break; } + if (payload.status === 'error') { + container.innerHTML = '
Leaderboard failed: ' + escHtml(payload.error || 'unknown') + '
'; + return; + } + var p = (payload.progress || {}); + var done = p.done || 0, total = p.total || 0; + var pct = total > 0 ? Math.round(done / total * 100) : 0; + var phase = p.phase || 'working'; + var elapsed = Math.round((payload.elapsedMs || (Date.now() - pollStart)) / 1000); + var partialLine = ''; + if (payload.partialResult && payload.partialResult.totals) { + var t = payload.partialResult.totals; + partialLine = ' · $' + (t.cost||0).toFixed(2) + ' / ' + (t.messages||0) + ' msgs so far'; + } + container.innerHTML = + '
' + + '
Building leaderboard stats…
' + + '
' + escHtml(phase) + ' — ' + done + ' / ' + total + ' sessions (' + pct + '%)
' + + '
' + + '
elapsed ' + elapsed + 's' + partialLine + '
' + + '
'; + await new Promise(function(r){ setTimeout(r, 500); }); + } var ghResp = await fetch('/api/github/profile'); var gh = await ghResp.json(); diff --git a/src/frontend/styles.css b/src/frontend/styles.css index 07585bb..92c1bc0 100644 --- a/src/frontend/styles.css +++ b/src/frontend/styles.css @@ -575,6 +575,19 @@ body { color: var(--accent-green); white-space: nowrap; } +.group-badge { + display: inline-flex; + align-items: center; + gap: 3px; + padding: 2px 7px; + border-radius: 10px; + font-size: 10px; + font-weight: 600; + background: rgba(168,85,247,0.18); + color: #c084fc; + white-space: nowrap; + cursor: help; +} [data-theme="light"] .cost-badge { background: rgba(52,199,89,0.12); } @@ -2472,6 +2485,104 @@ body { word-break: break-word; } +/* ── Analytics live banner + progress ───────────────────────── */ + +.analytics-live-banner { + background: linear-gradient(90deg, rgba(96,165,250,0.12), rgba(96,165,250,0.04)); + border: 1px solid rgba(96,165,250,0.35); + border-radius: 6px; + padding: 10px 14px; + margin: 0 20px 16px; +} +.analytics-live-banner .alb-row { + display: flex; + align-items: center; + gap: 10px; + margin-bottom: 4px; +} +.analytics-live-banner .alb-title { + font-size: 13px; + font-weight: 600; + color: var(--accent-blue, #60a5fa); + flex: 1; +} +.analytics-live-banner .alb-elapsed { + font-size: 12px; + color: var(--text-secondary, #aaa); + font-variant-numeric: tabular-nums; +} +.analytics-live-banner .alb-phase { + font-size: 11px; + color: var(--text-secondary, #aaa); + margin-bottom: 8px; + font-variant-numeric: tabular-nums; +} +.analytics-live-banner .alb-pulse { + display: inline-block; + width: 8px; + height: 8px; + border-radius: 50%; + background: #60a5fa; + box-shadow: 0 0 0 0 rgba(96,165,250,0.7); + animation: alb-pulse 1.5s infinite; +} +@keyframes alb-pulse { + 0% { box-shadow: 0 0 0 0 rgba(96,165,250,0.7); } + 70% { box-shadow: 0 0 0 8px rgba(96,165,250,0); } + 100% { box-shadow: 0 0 0 0 rgba(96,165,250,0); } +} +.analytics-live-banner .progress-bar-outer { + width: 100%; + height: 6px; + background: rgba(255,255,255,0.06); + border-radius: 3px; + overflow: hidden; +} +.analytics-live-banner .progress-bar-inner { + height: 100%; + background: linear-gradient(90deg, var(--accent-blue, #60a5fa), #3b82f6); + transition: width 0.5s ease-out; +} + +.analytics-progress { + max-width: 520px; + margin: 60px auto; + padding: 28px 32px; + border: 1px solid var(--border-color, #333); + border-radius: 8px; + background: var(--panel-bg, #1a1a1a); + text-align: center; +} +.analytics-progress .progress-title { + font-size: 16px; + font-weight: 600; + color: var(--text-primary, #eee); + margin-bottom: 8px; +} +.analytics-progress .progress-phase { + font-size: 13px; + color: var(--text-secondary, #aaa); + margin-bottom: 16px; + font-variant-numeric: tabular-nums; +} +.analytics-progress .progress-bar-outer { + width: 100%; + height: 8px; + background: rgba(255,255,255,0.06); + border-radius: 4px; + overflow: hidden; + margin-bottom: 12px; +} +.analytics-progress .progress-bar-inner { + height: 100%; + background: linear-gradient(90deg, var(--accent-blue, #60a5fa), var(--accent-blue, #3b82f6)); + transition: width 0.4s ease-out; +} +.analytics-progress .progress-subtle { + font-size: 11px; + color: var(--text-muted, #777); +} + /* ── Cost Analytics ─────────────────────────────────────────── */ .analytics-container { padding: 20px; } diff --git a/src/server.js b/src/server.js index 4a31b92..f09d34f 100644 --- a/src/server.js +++ b/src/server.js @@ -3,20 +3,299 @@ const http = require('http'); const https = require('https'); const { URL } = require('url'); const { exec } = require('child_process'); -const { loadSessions, loadSessionDetail, deleteSession, getGitCommits, exportSessionMarkdown, getSessionPreview, searchFullText, getActiveSessions, getSessionReplay, getCostAnalytics, computeSessionCost, getProjectGitInfo, getLeaderboardStats } = require('./data'); +const { loadSessions, loadSessionsAsync, getWarmingStatus, getSqliteBackfillStatus, createCostAggregator, computeSessionCostForAnalytics, buildOpencodeCostCache, loadSessionDetail, deleteSession, getGitCommits, exportSessionMarkdown, getSessionPreview, searchFullText, getActiveSessions, getSessionReplay, getCostAnalytics, computeSessionCost, getProjectGitInfo, getLeaderboardStats } = require('./data'); +const sqliteIndex = require('./sqlite-index'); const { detectTerminals, openInTerminal, focusTerminalByPid } = require('./terminals'); const { convertSession } = require('./convert'); const { generateHandoff } = require('./handoff'); const { CHANGELOG } = require('./changelog'); const { getHTML } = require('./html'); +// ── Background jobs for long-running stats ──────────────── +// Analytics and leaderboard run as async tasks that publish live partial +// results. The HTTP endpoint returns progress + partial snapshot so the UI +// can show numbers climbing in real time instead of a blank loading screen. +const _jobs = { + analytics: null, + leaderboard: null, +}; + +// Fingerprint for persistent cache lookup. Quantized to 5-min buckets so +// that codex sessions appending their files every second don't invalidate +// the cache on every request. +function _analyticsFingerprint(sessions, filterFrom, filterTo, extra) { + let maxTs = 0; + for (const s of sessions) if (s.last_ts > maxTs) maxTs = s.last_ts; + const bucket = Math.floor(maxTs / (5 * 60000)); + return [sessions.length, bucket, filterFrom || '', filterTo || '', extra || ''].join('|'); +} + +async function _runAnalyticsJob(filterFrom, filterTo, includeHelpers) { + const key = (filterFrom || '') + '|' + (filterTo || '') + '|h' + (includeHelpers ? '1' : '0'); + const job = { + state: 'running', + progress: { done: 0, total: 0, phase: 'scanning files' }, + result: null, partialResult: null, error: null, key, + startedAt: Date.now(), finishedAt: 0, + }; + _jobs.analytics = job; + try { + let sessions = await loadSessionsAsync((p) => { + job.progress = { done: p.done || 0, total: p.total || 0, phase: p.phase || 'parsing' }; + }); + if (!includeHelpers) sessions = sessions.filter(s => !s.is_helper); + if (filterFrom) sessions = sessions.filter(s => s.date >= filterFrom); + if (filterTo) sessions = sessions.filter(s => s.date <= filterTo); + + const fingerprint = _analyticsFingerprint(sessions, filterFrom, filterTo, includeHelpers ? 'h1' : 'h0'); + try { + const cached = await sqliteIndex.getAggregateCacheAsync('analytics', fingerprint); + if (cached && cached.result) { + job.result = cached.result; + job.state = 'done'; + job.finishedAt = Date.now(); + log('JOB', `analytics served from cache`); + return; + } + } catch (e) { log('WARN', 'analytics cache read failed: ' + e.message); } + + job.progress.phase = 'opencode batch query'; + const opencodeCostCache = buildOpencodeCostCache(sessions); + + job.progress = { done: 0, total: sessions.length, phase: 'aggregating' }; + const agg = createCostAggregator(); + const CHUNK = 50; + for (let i = 0; i < sessions.length; i += CHUNK) { + const slice = sessions.slice(i, i + CHUNK); + for (const s of slice) { + try { + const cost = computeSessionCostForAnalytics(s, opencodeCostCache); + agg.merge(s, cost); + } catch {} + } + job.progress.done = Math.min(i + CHUNK, sessions.length); + try { job.partialResult = agg.finalize(); } catch {} + await new Promise(r => setImmediate(r)); + } + + job.result = agg.finalize(); + job.partialResult = null; + job.state = 'done'; + job.finishedAt = Date.now(); + log('JOB', `analytics done in ${job.finishedAt - job.startedAt}ms (${sessions.length} sessions)`); + try { await sqliteIndex.setAggregateCache('analytics', fingerprint, job.result); } catch {} + } catch (e) { + job.state = 'error'; + job.error = e.message || String(e); + job.finishedAt = Date.now(); + log('ERROR', `analytics job failed: ${job.error}`); + } +} + +async function _runLeaderboardJob(includeHelpers) { + const job = { + state: 'running', + progress: { done: 0, total: 0, phase: 'scanning files' }, + result: null, partialResult: null, error: null, key: 'lb|h' + (includeHelpers ? '1' : '0'), + startedAt: Date.now(), finishedAt: 0, + }; + _jobs.leaderboard = job; + try { + let sessions = await loadSessionsAsync((p) => { + job.progress = { done: p.done || 0, total: p.total || 0, phase: p.phase || 'parsing' }; + }); + if (!includeHelpers) sessions = sessions.filter(s => !s.is_helper); + // Dedupe to conversation groups. Strategy: + // - count of sessions = unique groups (612) + // - msgs/hours = sum over REPRESENTATIVES only (avoids counting + // agent→agent turns from 900 retries of the same prompt) + // - cost = sum over ALL sessions (real money actually spent) + // The representative is the session with the MOST user_messages + // within the group (so we keep the "fullest" run). + const groupMap = new Map(); + const allRollouts = sessions; // keep for cost sum + for (const s of sessions) { + const k = s.group_key || s.id; + const existing = groupMap.get(k); + if (!existing || (s.user_messages || 0) > (existing.user_messages || 0)) { + groupMap.set(k, s); + } + } + const representatives = Array.from(groupMap.values()); + const uniqueSessionCount = representatives.length; + sessions = representatives; // the per-session loop iterates representatives + const fingerprint = _analyticsFingerprint(sessions, '', '', includeHelpers ? 'h1' : 'h0'); + try { + const cached = await sqliteIndex.getAggregateCacheAsync('leaderboard', fingerprint); + if (cached && cached.result) { + job.result = cached.result; + job.state = 'done'; + job.finishedAt = Date.now(); + log('JOB', 'leaderboard served from cache'); + return; + } + } catch {} + + const opencodeCostCache = buildOpencodeCostCache(allRollouts); + + // Real cost = sum over ALL rollouts (not deduped). Real money spent. + let realTotalCost = 0; + for (const s of allRollouts) { + try { realTotalCost += computeSessionCostForAnalytics(s, opencodeCostCache).cost || 0; } catch {} + } + + const todayStr = new Date().toISOString().slice(0, 10); + const fmtDay = (ts) => { + const d = new Date(ts); + return d.getFullYear() + '-' + String(d.getMonth()+1).padStart(2,'0') + '-' + String(d.getDate()).padStart(2,'0'); + }; + const byDay = {}; + const ensureDay = (date) => { + if (!byDay[date]) byDay[date] = { date, sessions: 0, messages: 0, hours: 0, cost: 0, agents: {} }; + return byDay[date]; + }; + let totalMessages = 0, totalHours = 0, totalCost = 0; + const agentTotals = {}; + + job.progress = { done: 0, total: sessions.length, phase: 'aggregating' }; + const CHUNK = 50; + for (let i = 0; i < sessions.length; i += CHUNK) { + const slice = sessions.slice(i, i + CHUNK); + for (const s of slice) { + try { + if (!s.first_ts || !s.last_ts) continue; + const tool = s.tool || 'unknown'; + const cost = computeSessionCostForAnalytics(s, opencodeCostCache).cost || 0; + const day = s.date || fmtDay(s.last_ts); + const d = ensureDay(day); + d.sessions++; + // Only count EXACT user_messages (parsed from JSONL). Sessions + // without parsed metadata yet contribute 0 — numbers grow as the + // background warmer finishes. This avoids the msgs*0.5 estimate + // that massively inflates totals on Claude (tool_results) and on + // partially-parsed runs. + const msgs = s.user_messages > 0 ? s.user_messages : 0; + d.messages += msgs; + d.hours += Math.min((s.last_ts - s.first_ts) / 3600000, 16); + d.cost += cost; + d.agents[tool] = (d.agents[tool] || 0) + 1; + totalMessages += msgs; + totalHours += Math.min((s.last_ts - s.first_ts) / 3600000, 16); + totalCost += cost; + agentTotals[tool] = (agentTotals[tool] || 0) + 1; + } catch {} + } + job.progress.done = Math.min(i + CHUNK, sessions.length); + try { + const daily = Object.values(byDay).sort((a, b) => b.date.localeCompare(a.date)); + let streak = 0; + const dt = new Date(); + for (let k = 0; k < 365; k++) { + const d2 = dt.toISOString().slice(0, 10); + if (byDay[d2]) { streak++; dt.setDate(dt.getDate() - 1); } else break; + } + job.partialResult = { + anon: { id: '', name: 'You' }, + today: byDay[todayStr] || { sessions: 0, messages: 0, hours: 0, cost: 0, agents: {} }, + totals: { + sessions: uniqueSessionCount, + messages: totalMessages, + hours: Math.round(totalHours * 10) / 10, + cost: Math.round(realTotalCost * 100) / 100, + }, + agents: agentTotals, + streak, + daily: daily.slice(0, 30), + activeDays: daily.length, + }; + } catch {} + await new Promise(r => setImmediate(r)); + } + + let anon = { id: '', name: 'anon' }; + try { + const { getOrCreateAnonId } = require('./data'); + anon = getOrCreateAnonId(); + } catch {} + const daily = Object.values(byDay).sort((a, b) => b.date.localeCompare(a.date)); + let streak = 0; + const dt = new Date(); + for (let k = 0; k < 365; k++) { + const d2 = dt.toISOString().slice(0, 10); + if (byDay[d2]) { streak++; dt.setDate(dt.getDate() - 1); } else break; + } + job.result = { + anon, + today: byDay[todayStr] || { sessions: 0, messages: 0, hours: 0, cost: 0, agents: {} }, + totals: { + sessions: uniqueSessionCount, + messages: totalMessages, + hours: Math.round(totalHours * 10) / 10, + cost: Math.round(realTotalCost * 100) / 100, + }, + agents: agentTotals, + streak, + daily: daily.slice(0, 30), + activeDays: daily.length, + }; + job.partialResult = null; + job.state = 'done'; + job.finishedAt = Date.now(); + log('JOB', `leaderboard done in ${job.finishedAt - job.startedAt}ms (${sessions.length} sessions)`); + try { await sqliteIndex.setAggregateCache('leaderboard', fingerprint, job.result); } catch {} + } catch (e) { + job.state = 'error'; + job.error = e.message || String(e); + job.finishedAt = Date.now(); + log('ERROR', `leaderboard job failed: ${job.error}`); + } +} + +function _jobResponse(job) { + if (!job) return { status: 'idle' }; + if (job.state === 'done') { + return { status: 'done', result: job.result, ms: job.finishedAt - job.startedAt }; + } + if (job.state === 'error') { + return { status: 'error', error: job.error }; + } + return { + status: 'running', + progress: job.progress, + partialResult: job.partialResult || null, + elapsedMs: Date.now() - job.startedAt, + }; +} + // ── Logging ────────────────────────────────── -const LOG_VERBOSE = process.env.CODEDASH_LOG !== '0'; +// CODEDASH_LOG=0 (default) → quiet stdout (only ERROR/WARN/JOB), full log to +// ~/.codedash/logs/server.log. CODEDASH_LOG=1 → verbose stdout mirror. +const fs_log = require('fs'); +const path_log = require('path'); +const os_log = require('os'); +const LOG_DIR = path_log.join(os_log.homedir(), '.codedash', 'logs'); +try { fs_log.mkdirSync(LOG_DIR, { recursive: true }); } catch {} +const LOG_FILE = path_log.join(LOG_DIR, 'server.log'); +let _logStream = null; +try { _logStream = fs_log.createWriteStream(LOG_FILE, { flags: 'a' }); } catch {} + +const LOG_MODE = (process.env.CODEDASH_LOG === '1' || process.env.CODEDASH_LOG === 'verbose') ? 'verbose' : 'quiet'; +const STDOUT_TAGS = new Set(['ERROR', 'WARN', 'JOB']); function log(tag, msg, data) { - if (!LOG_VERBOSE && tag !== 'ERROR') return; const ts = new Date().toLocaleTimeString('en-GB'); - const color = tag === 'ERROR' ? '\x1b[31m' : tag === 'WARN' ? '\x1b[33m' : tag === 'API' ? '\x1b[36m' : '\x1b[2m'; + if (_logStream) { + try { + let plain = `${ts} [${tag}] ${msg}`; + if (data !== undefined) { + const str = typeof data === 'object' ? JSON.stringify(data) : String(data); + plain += ' ' + (str.length > 500 ? str.slice(0, 500) + '...' : str); + } + _logStream.write(plain + '\n'); + } catch {} + } + if (LOG_MODE !== 'verbose' && !STDOUT_TAGS.has(tag)) return; + const color = tag === 'ERROR' ? '\x1b[31m' : tag === 'WARN' ? '\x1b[33m' : tag === 'API' ? '\x1b[36m' : tag === 'JOB' ? '\x1b[32m' : '\x1b[2m'; let line = ` ${color}${ts} [${tag}]\x1b[0m ${msg}`; if (data !== undefined) { const str = typeof data === 'object' ? JSON.stringify(data) : String(data); @@ -33,7 +312,7 @@ function startServer(host, port, openBrowser = true) { // Log all API requests (skip static & frequent polls) const isApi = pathname.startsWith('/api/'); - const isFrequent = pathname === '/api/active' || pathname === '/api/version'; + const isFrequent = pathname === '/api/active' || pathname === '/api/version' || pathname === '/api/warming' || pathname === '/api/sessions' || pathname === '/api/sqlite-status'; if (isApi && !isFrequent) { const params = Object.fromEntries(parsed.searchParams); log('API', `${req.method} ${pathname}`, Object.keys(params).length ? params : undefined); @@ -66,12 +345,23 @@ function startServer(host, port, openBrowser = true) { // ── Sessions API ──────────────────────── else if (req.method === 'GET' && pathname === '/api/sessions') { const sessions = loadSessions(); + // By default hide codex_exec helper sessions (sub-agent runs) — the + // user can request them via ?include_helpers=1. Counts for helpers are + // sent as a header so the UI can show a "N helper sessions hidden" toggle. + const includeHelpers = parsed.searchParams.get('include_helpers') === '1'; + const helperCount = sessions.filter(s => s.is_helper).length; + const filtered = includeHelpers ? sessions : sessions.filter(s => !s.is_helper); + // Preserve transient flags on the array + filtered._loading = sessions._loading; + filtered._warming = sessions._warming; + filtered._warmingProgress = sessions._warmingProgress; + const byTool = {}; - sessions.forEach(s => { byTool[s.tool] = (byTool[s.tool] || 0) + 1; }); - log('DATA', `loaded ${sessions.length} sessions${sessions._loading ? ' (cursor loading...)' : ''}`, byTool); - // Send _loading flag as header to avoid polluting array response + filtered.forEach(s => { byTool[s.tool] = (byTool[s.tool] || 0) + 1; }); + log('DATA', `loaded ${filtered.length} sessions (hidden ${helperCount} helpers)${sessions._loading ? ' (cursor loading...)' : ''}`, byTool); if (sessions._loading) res.setHeader('X-Loading', '1'); - json(res, sessions); + if (helperCount > 0) res.setHeader('X-Helper-Count', String(helperCount)); + json(res, filtered); } else if (req.method === 'GET' && pathname.startsWith('/api/session/') && !pathname.includes('/export')) { @@ -164,6 +454,17 @@ function startServer(host, port, openBrowser = true) { json(res, info || { error: 'No git repo found' }); } + // ── Warming + SQLite index status ───────── + else if (req.method === 'GET' && pathname === '/api/warming') { + json(res, getWarmingStatus()); + } + + else if (req.method === 'GET' && pathname === '/api/sqlite-status') { + let indexStatus = {}; + try { indexStatus = sqliteIndex.getIndexStatus(); } catch {} + json(res, { backfill: getSqliteBackfillStatus(), index: indexStatus }); + } + // ── Active sessions ───────────────────── else if (req.method === 'GET' && pathname === '/api/active') { const active = getActiveSessions(); @@ -263,8 +564,8 @@ function startServer(host, port, openBrowser = true) { // ── Full-text search ────────────────────── else if (req.method === 'GET' && pathname === '/api/search') { const q = parsed.searchParams.get('q') || ''; - const sessions = loadSessions(); - const results = searchFullText(q, sessions); + // Uses SQLite FTS5 index directly — no loadSessions needed + const results = searchFullText(q, []); json(res, results); } @@ -284,15 +585,56 @@ function startServer(host, port, openBrowser = true) { json(res, data); } - // ── Cost analytics ────────────────────── + // ── Cost analytics (async job + live partial) ────────────────── else if (req.method === 'GET' && pathname === '/api/analytics/cost') { - let sessions = loadSessions(); - const from = parsed.searchParams.get('from'); - const to = parsed.searchParams.get('to'); - if (from) sessions = sessions.filter(s => s.date >= from); - if (to) sessions = sessions.filter(s => s.date <= to); - const data = getCostAnalytics(sessions); - json(res, data); + const from = parsed.searchParams.get('from') || ''; + const to = parsed.searchParams.get('to') || ''; + const includeHelpers = parsed.searchParams.get('include_helpers') === '1'; + const key = from + '|' + to + '|h' + (includeHelpers ? '1' : '0'); + const job = _jobs.analytics; + if (job && job.state === 'done' && job.key === key) { + json(res, { status: 'done', ...job.result }); + } else if (job && job.state === 'running' && job.key === key) { + json(res, _jobResponse(job)); + } else { + // Try persistent cache FIRST so repeat visits don't trigger a rerun. + // Uses the sync in-memory sessions cache (fast when hot). + let served = false; + try { + const sessions = loadSessions(); + let filtered = includeHelpers ? sessions : sessions.filter(s => !s.is_helper); + if (from) filtered = filtered.filter(s => s.date >= from); + if (to) filtered = filtered.filter(s => s.date <= to); + const fingerprint = _analyticsFingerprint(filtered, from, to, includeHelpers ? 'h1' : 'h0'); + sqliteIndex.getAggregateCacheAsync('analytics', fingerprint).then(cached => { + if (served) return; + if (cached && cached.result) { + served = true; + _jobs.analytics = { + state: 'done', key, result: cached.result, partialResult: null, + progress: { done: filtered.length, total: filtered.length, phase: 'cached' }, + error: null, startedAt: cached.computedAt, finishedAt: cached.computedAt, + }; + json(res, { status: 'done', ...cached.result, _cached: true }); + } else { + served = true; + _runAnalyticsJob(from, to, includeHelpers); + json(res, _jobResponse(_jobs.analytics)); + } + }).catch(() => { + if (served) return; + served = true; + _runAnalyticsJob(from, to, includeHelpers); + json(res, _jobResponse(_jobs.analytics)); + }); + } catch (e) { + if (!served) { + served = true; + _runAnalyticsJob(from, to, includeHelpers); + json(res, _jobResponse(_jobs.analytics)); + } + } + } } // ── LLM Config ──────────────────────────── @@ -356,9 +698,51 @@ function startServer(host, port, openBrowser = true) { } // ── Leaderboard stats ──────────────────── + // ── Leaderboard (async job + live partial) ───────────────── else if (req.method === 'GET' && pathname === '/api/leaderboard') { - const stats = getLeaderboardStats(); - json(res, stats); + const includeHelpers = parsed.searchParams.get('include_helpers') === '1'; + const expectedKey = 'lb|h' + (includeHelpers ? '1' : '0'); + const job = _jobs.leaderboard; + if (job && job.state === 'done' && job.key === expectedKey) { + json(res, { status: 'done', ...job.result }); + } else if (job && job.state === 'running' && job.key === expectedKey) { + json(res, _jobResponse(job)); + } else { + // Try persistent cache first (instant on repeat visits) + let served = false; + try { + const sessions = loadSessions(); + const filtered = includeHelpers ? sessions : sessions.filter(s => !s.is_helper); + const fingerprint = _analyticsFingerprint(filtered, '', '', includeHelpers ? 'h1' : 'h0'); + sqliteIndex.getAggregateCacheAsync('leaderboard', fingerprint).then(cached => { + if (served) return; + if (cached && cached.result) { + served = true; + _jobs.leaderboard = { + state: 'done', key: expectedKey, result: cached.result, partialResult: null, + progress: { done: filtered.length, total: filtered.length, phase: 'cached' }, + error: null, startedAt: cached.computedAt, finishedAt: cached.computedAt, + }; + json(res, { status: 'done', ...cached.result, _cached: true }); + } else { + served = true; + _runLeaderboardJob(includeHelpers); + json(res, _jobResponse(_jobs.leaderboard)); + } + }).catch(() => { + if (served) return; + served = true; + _runLeaderboardJob(includeHelpers); + json(res, _jobResponse(_jobs.leaderboard)); + }); + } catch (e) { + if (!served) { + served = true; + _runLeaderboardJob(includeHelpers); + json(res, _jobResponse(_jobs.leaderboard)); + } + } + } } else if (req.method === 'POST' && pathname === '/api/leaderboard/sync') { diff --git a/src/sqlite-index.js b/src/sqlite-index.js new file mode 100644 index 0000000..3f725eb --- /dev/null +++ b/src/sqlite-index.js @@ -0,0 +1,479 @@ +// SQLite + FTS5 index for sessions, messages, and full-text search. +// +// Why SQLite: (1) codedash already shells to `sqlite3` CLI for opencode/kiro, +// zero new deps. (2) FTS5 is built into modern sqlite3 and is the best +// local full-text search engine available. (3) Persistent, mmap'd B-tree — +// billions of rows possible, queries in milliseconds. +// +// Schema: +// sessions one row per session (metadata + aggregated cost) +// messages one row per user/assistant message, chronological +// messages_fts FTS5 virtual table shadowing messages.content +// files_seen (path, mtime, size) for incremental ingest +// +// All ingest goes through SQL transactions. Queries avoid Node ↔ child_process +// overhead by using a single sqlite3 invocation per call. + +const fs = require('fs'); +const path = require('path'); +const os = require('os'); +const { spawn, spawnSync } = require('child_process'); + +const DB_DIR = path.join(os.homedir(), '.codedash', 'cache'); +try { fs.mkdirSync(DB_DIR, { recursive: true }); } catch {} +const DB_FILE = path.join(DB_DIR, 'index.sqlite'); + +// ── sqlite3 CLI helpers ───────────────────────────────────── +// Query path stays sync via spawnSync (fast, small SQL). Write path (large +// batches) goes through async spawn so it doesn't block the event loop. + +// Use sqlite3 -cmd to set busy_timeout before running the query. This sets +// it on the connection without producing extra JSON output rows. +const _CMD_BUSY = '.timeout 30000'; + +function _exec(sql, opts) { + opts = opts || {}; + const args = ['-cmd', _CMD_BUSY, DB_FILE]; + if (opts.json) args.push('-json'); + args.push(sql); + const r = spawnSync('sqlite3', args, { + encoding: 'utf8', + maxBuffer: 64 * 1024 * 1024, + timeout: opts.timeout || 60000, + windowsHide: true, + }); + if (r.error) throw r.error; + if (r.status !== 0) throw new Error('sqlite3 exit ' + r.status + ': ' + r.stderr); + return r.stdout || ''; +} + +function _execJson(sql, opts) { + const out = _exec(sql, Object.assign({}, opts || {}, { json: true })).trim(); + if (!out) return []; + try { return JSON.parse(out); } catch { return []; } +} + +// Async variant: streams SQL to sqlite3 stdin without blocking the event loop. +// Returns a Promise with stdout. +function _execAsync(sql, opts) { + opts = opts || {}; + return new Promise((resolve, reject) => { + const args = ['-cmd', _CMD_BUSY, DB_FILE]; + if (opts.json) args.push('-json'); + const child = spawn('sqlite3', args, { + stdio: ['pipe', 'pipe', 'pipe'], + windowsHide: true, + }); + let stdout = ''; + let stderr = ''; + let finished = false; + const to = setTimeout(() => { + if (!finished) { + finished = true; + try { child.kill('SIGKILL'); } catch {} + reject(new Error('sqlite3 timeout')); + } + }, opts.timeout || 120000); + + child.stdout.on('data', d => { stdout += d.toString('utf8'); }); + child.stderr.on('data', d => { stderr += d.toString('utf8'); }); + child.on('error', e => { if (!finished) { finished = true; clearTimeout(to); reject(e); } }); + child.on('close', code => { + if (finished) return; + finished = true; + clearTimeout(to); + if (code !== 0) return reject(new Error('sqlite3 exit ' + code + ': ' + stderr)); + resolve(stdout); + }); + // Write SQL to stdin and close + try { + child.stdin.write(sql); + child.stdin.end(); + } catch (e) { + if (!finished) { finished = true; clearTimeout(to); reject(e); } + } + }); +} + +// ── Schema ────────────────────────────────────────────────── + +const SCHEMA = ` +PRAGMA journal_mode = WAL; +PRAGMA synchronous = NORMAL; +PRAGMA temp_store = MEMORY; +PRAGMA mmap_size = 268435456; +PRAGMA busy_timeout = 30000; +PRAGMA foreign_keys = ON; + +CREATE TABLE IF NOT EXISTS sessions ( + id TEXT PRIMARY KEY, + tool TEXT NOT NULL, + project TEXT, + project_short TEXT, + first_ts REAL, + last_ts REAL, + messages INTEGER DEFAULT 0, + user_messages INTEGER DEFAULT 0, + file_size INTEGER DEFAULT 0, + cost REAL DEFAULT 0, + input_tokens INTEGER DEFAULT 0, + output_tokens INTEGER DEFAULT 0, + cache_read_tokens INTEGER DEFAULT 0, + cache_create_tokens INTEGER DEFAULT 0, + model TEXT, + first_message TEXT, + mcp_servers TEXT, -- JSON array + skills TEXT, -- JSON array + source_file TEXT, + source_mtime REAL, + source_size INTEGER, + indexed_at INTEGER +); +CREATE INDEX IF NOT EXISTS idx_sessions_tool ON sessions(tool); +CREATE INDEX IF NOT EXISTS idx_sessions_last_ts ON sessions(last_ts DESC); +CREATE INDEX IF NOT EXISTS idx_sessions_project ON sessions(project); + +CREATE TABLE IF NOT EXISTS messages ( + rowid INTEGER PRIMARY KEY AUTOINCREMENT, + session_id TEXT NOT NULL, + seq INTEGER NOT NULL, + role TEXT NOT NULL, + ts REAL, + content TEXT, + FOREIGN KEY (session_id) REFERENCES sessions(id) ON DELETE CASCADE +); +CREATE INDEX IF NOT EXISTS idx_messages_session ON messages(session_id, seq); + +CREATE VIRTUAL TABLE IF NOT EXISTS messages_fts USING fts5( + content, + session_id UNINDEXED, + role UNINDEXED, + seq UNINDEXED, + tokenize = 'porter unicode61' +); + +CREATE TABLE IF NOT EXISTS daily_stats ( + session_id TEXT NOT NULL, + day TEXT NOT NULL, + messages INTEGER DEFAULT 0, + hours REAL DEFAULT 0, + tool TEXT, + PRIMARY KEY (session_id, day) +); +CREATE INDEX IF NOT EXISTS idx_daily_day ON daily_stats(day); + +CREATE TABLE IF NOT EXISTS files_seen ( + path TEXT PRIMARY KEY, + mtime REAL, + size INTEGER, + session_id TEXT, + indexed_at INTEGER +); + +-- Persistent aggregate result cache (analytics, leaderboard, ...) +-- Keyed by query fingerprint; holds a single JSON blob row per kind. +CREATE TABLE IF NOT EXISTS aggregate_cache ( + kind TEXT NOT NULL, -- 'analytics' | 'leaderboard' | ... + fingerprint TEXT NOT NULL, -- input hash (filters + data version) + result_json TEXT NOT NULL, + computed_at INTEGER NOT NULL, + PRIMARY KEY (kind, fingerprint) +); +`; + +let _schemaReady = false; +function ensureSchema() { + if (_schemaReady) return; + try { + _exec(SCHEMA); + _schemaReady = true; + } catch (e) { + throw new Error('Failed to init SQLite index: ' + e.message); + } +} + +// ── Incremental ingest ─────────────────────────────────────── + +function sqlEscape(s) { + if (s === null || s === undefined) return 'NULL'; + return "'" + String(s).replace(/'/g, "''") + "'"; +} + +// Check if a file path is already indexed with matching mtime+size. +function isFileCurrent(filePath) { + ensureSchema(); + let stat; + try { stat = fs.statSync(filePath); } catch { return false; } + const rows = _execJson( + `SELECT mtime, size FROM files_seen WHERE path = ${sqlEscape(filePath)} LIMIT 1` + ); + if (rows.length === 0) return false; + return rows[0].mtime === stat.mtimeMs && rows[0].size === stat.size; +} + +// Bulk version: load the entire files_seen map into memory in a single query. +// Used by the backfill loop to avoid an N×sync-SQL fanout. +function loadAllFilesSeen() { + ensureSchema(); + const rows = _execJson(`SELECT path, mtime, size FROM files_seen`); + const map = new Map(); + for (const r of rows) map.set(r.path, { mtime: r.mtime, size: r.size }); + return map; +} + +// Bulk-index a batch of sessions + messages asynchronously. +// Builds a single transaction SQL blob, pipes to sqlite3 via async spawn. +// Returns a Promise that resolves when sqlite3 finishes writing. +function indexBatchAsync(batch) { + if (!batch || batch.length === 0) return Promise.resolve(); + ensureSchema(); + + const now = Date.now(); + const parts = ['BEGIN;']; + + for (const item of batch) { + const s = item.session; + if (!s || !s.id) continue; + + // Delete old rows for this session (re-ingest case) + parts.push(`DELETE FROM sessions WHERE id = ${sqlEscape(s.id)};`); + parts.push(`DELETE FROM messages WHERE session_id = ${sqlEscape(s.id)};`); + parts.push(`DELETE FROM messages_fts WHERE session_id = ${sqlEscape(s.id)};`); + parts.push(`DELETE FROM daily_stats WHERE session_id = ${sqlEscape(s.id)};`); + + // Insert session + parts.push( + `INSERT INTO sessions (id, tool, project, project_short, first_ts, last_ts, messages, user_messages, file_size, cost, input_tokens, output_tokens, cache_read_tokens, cache_create_tokens, model, first_message, mcp_servers, skills, source_file, source_mtime, source_size, indexed_at) VALUES (` + + [ + sqlEscape(s.id), + sqlEscape(s.tool || 'unknown'), + sqlEscape(s.project || ''), + sqlEscape(s.project_short || ''), + s.first_ts || 0, + s.last_ts || 0, + s.messages || 0, + s.user_messages || 0, + s.file_size || 0, + s.cost || 0, + s.input_tokens || 0, + s.output_tokens || 0, + s.cache_read_tokens || 0, + s.cache_create_tokens || 0, + sqlEscape(s.model || ''), + sqlEscape((s.first_message || '').slice(0, 500)), + sqlEscape(JSON.stringify(s.mcp_servers || [])), + sqlEscape(JSON.stringify(s.skills || [])), + sqlEscape(item.filePath || ''), + s.source_mtime || 0, + s.source_size || 0, + now, + ].join(', ') + ');' + ); + + // Messages + FTS + const msgs = item.messages || []; + for (const m of msgs) { + if (!m.content) continue; + const esc = sqlEscape(m.content.slice(0, 8000)); + parts.push( + `INSERT INTO messages (session_id, seq, role, ts, content) VALUES (` + + [ + sqlEscape(s.id), + m.seq || 0, + sqlEscape(m.role || ''), + m.ts || 0, + esc, + ].join(', ') + ');' + ); + parts.push( + `INSERT INTO messages_fts (content, session_id, role, seq) VALUES (` + + [esc, sqlEscape(s.id), sqlEscape(m.role || ''), m.seq || 0].join(', ') + ');' + ); + } + + // Daily breakdown + const daily = item.daily || {}; + for (const day in daily) { + const d = daily[day]; + parts.push( + `INSERT INTO daily_stats (session_id, day, messages, hours, tool) VALUES (` + + [ + sqlEscape(s.id), + sqlEscape(day), + d.messages || 0, + d.hours || 0, + sqlEscape(s.tool || 'unknown'), + ].join(', ') + ');' + ); + } + + // Record file stamp + if (item.filePath && s.source_mtime && s.source_size) { + parts.push( + `INSERT OR REPLACE INTO files_seen (path, mtime, size, session_id, indexed_at) VALUES (` + + [ + sqlEscape(item.filePath), + s.source_mtime, + s.source_size, + sqlEscape(s.id), + now, + ].join(', ') + ');' + ); + } + } + + parts.push('COMMIT;'); + return _execAsync(parts.join('\n'), { timeout: 120000 }).catch((e) => { + // Try to rollback via sync exec (fire and forget) + try { _exec('ROLLBACK;'); } catch {} + throw e; + }); +} + +// Alias for backward compat — returns the promise. +const indexBatch = indexBatchAsync; + +// ── Query API ─────────────────────────────────────────────── + +function search(query, limit) { + ensureSchema(); + if (!query || query.trim().length < 2) return []; + limit = Math.max(1, Math.min(500, limit || 100)); + + // Escape FTS5 special chars — we treat the input as a phrase + const phrase = '"' + query.replace(/"/g, '""') + '"'; + const sql = ` + SELECT session_id, role, seq, + snippet(messages_fts, 0, '<<', '>>', '...', 20) AS snippet + FROM messages_fts + WHERE content MATCH ${sqlEscape(phrase)} + ORDER BY rank + LIMIT ${limit}; + `; + try { + return _execJson(sql, { timeout: 15000 }); + } catch { + return []; + } +} + +function getSessionStats() { + ensureSchema(); + const rows = _execJson(` + SELECT + COUNT(*) AS session_count, + SUM(messages) AS total_messages, + SUM(user_messages) AS total_user_messages, + SUM(cost) AS total_cost, + SUM(input_tokens + output_tokens + cache_read_tokens + cache_create_tokens) AS total_tokens + FROM sessions; + `); + return rows[0] || {}; +} + +function getDailyStats(limit) { + ensureSchema(); + limit = limit || 30; + return _execJson(` + SELECT day, + SUM(messages) AS messages, + SUM(hours) AS hours, + COUNT(DISTINCT session_id) AS sessions + FROM daily_stats + GROUP BY day + ORDER BY day DESC + LIMIT ${limit}; + `); +} + +function getCountByTool() { + ensureSchema(); + return _execJson(`SELECT tool, COUNT(*) AS n FROM sessions GROUP BY tool ORDER BY n DESC;`); +} + +// ── Aggregate result cache ────────────────────────────────── +// Persistent cache for pre-computed analytics/leaderboard results. +// The caller decides the fingerprint (e.g. max(session.last_ts) + session +// count + filter). On cache hit: O(1) SQL read, milliseconds. + +function getAggregateCache(kind, fingerprint) { + ensureSchema(); + const rows = _execJson( + `SELECT result_json, computed_at FROM aggregate_cache WHERE kind = ${sqlEscape(kind)} AND fingerprint = ${sqlEscape(fingerprint)} LIMIT 1` + ); + if (rows.length === 0) return null; + try { + return { + result: JSON.parse(rows[0].result_json), + computedAt: rows[0].computed_at, + }; + } catch { + return null; + } +} + +async function getAggregateCacheAsync(kind, fingerprint) { + ensureSchema(); + const out = await _execAsync( + `SELECT result_json, computed_at FROM aggregate_cache WHERE kind = ${sqlEscape(kind)} AND fingerprint = ${sqlEscape(fingerprint)} LIMIT 1`, + { json: true, timeout: 10000 } + ); + let rows; + try { rows = JSON.parse(out.trim() || '[]'); } catch { return null; } + if (!rows || rows.length === 0) return null; + try { + return { + result: JSON.parse(rows[0].result_json), + computedAt: rows[0].computed_at, + }; + } catch { + return null; + } +} + +async function setAggregateCache(kind, fingerprint, result) { + ensureSchema(); + const json = JSON.stringify(result); + const sql = + `DELETE FROM aggregate_cache WHERE kind = ${sqlEscape(kind)};\n` + // keep only latest per kind + `INSERT INTO aggregate_cache (kind, fingerprint, result_json, computed_at) VALUES (` + + [sqlEscape(kind), sqlEscape(fingerprint), sqlEscape(json), Date.now()].join(', ') + + `);`; + try { await _execAsync(sql, { timeout: 30000 }); } catch {} +} + +function getIndexStatus() { + ensureSchema(); + const sess = _execJson(`SELECT COUNT(*) AS n FROM sessions`); + const msgs = _execJson(`SELECT COUNT(*) AS n FROM messages`); + const files = _execJson(`SELECT COUNT(*) AS n FROM files_seen`); + let size = 0; + try { size = fs.statSync(DB_FILE).size; } catch {} + return { + sessions: (sess[0] || {}).n || 0, + messages: (msgs[0] || {}).n || 0, + files: (files[0] || {}).n || 0, + db_bytes: size, + db_path: DB_FILE, + }; +} + +module.exports = { + DB_FILE, + ensureSchema, + isFileCurrent, + indexBatch, + indexBatchAsync, + loadAllFilesSeen, + search, + getSessionStats, + getDailyStats, + getCountByTool, + getIndexStatus, + getAggregateCache, + getAggregateCacheAsync, + setAggregateCache, + _exec, + _execAsync, + _execJson, +}; From 6bda4fd36363dfc393cd404d8f5a7371daf1825b Mon Sep 17 00:00:00 2001 From: Dave from DS Date: Thu, 9 Apr 2026 12:42:22 +0300 Subject: [PATCH 02/14] Update src/data.js Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- src/data.js | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/src/data.js b/src/data.js index 298b45f..06f32d9 100644 --- a/src/data.js +++ b/src/data.js @@ -789,9 +789,14 @@ async function parseClaudeSessionFileAsync(sessionFile) { const entry = JSON.parse(line); if (entry.type === 'user' || entry.type === 'assistant') msgCount++; if (entry.type === 'user' && isRealUserPromptAsync(entry)) userMsgCount++; - if (entry.timestamp) { - if (entry.timestamp < firstTs) firstTs = entry.timestamp; - if (entry.timestamp > lastTs) lastTs = entry.timestamp; + const rawTimestamp = entry.timestamp; + const normalizedTimestamp = + typeof rawTimestamp === 'number' + ? rawTimestamp + : (typeof rawTimestamp === 'string' ? Date.parse(rawTimestamp) : NaN); + if (Number.isFinite(normalizedTimestamp)) { + if (normalizedTimestamp < firstTs) firstTs = normalizedTimestamp; + if (normalizedTimestamp > lastTs) lastTs = normalizedTimestamp; } if (!projectPath && entry.type === 'user' && entry.cwd) projectPath = entry.cwd; if (!worktreeOriginalCwd && entry.type === 'worktree-state' && entry.worktreeSession && entry.worktreeSession.originalCwd) { From 70a5bcca89cb115df8a39a21060409c45c6aee33 Mon Sep 17 00:00:00 2001 From: Dave from DS Date: Thu, 9 Apr 2026 12:42:42 +0300 Subject: [PATCH 03/14] Update src/server.js Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- src/server.js | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/server.js b/src/server.js index f09d34f..756db13 100644 --- a/src/server.js +++ b/src/server.js @@ -564,8 +564,8 @@ function startServer(host, port, openBrowser = true) { // ── Full-text search ────────────────────── else if (req.method === 'GET' && pathname === '/api/search') { const q = parsed.searchParams.get('q') || ''; - // Uses SQLite FTS5 index directly — no loadSessions needed - const results = searchFullText(q, []); + // Prefer the SQLite FTS5 index, but provide sessions so in-memory fallback can work + const results = searchFullText(q, loadSessions()); json(res, results); } From 35894820150a12a21c7f2934562a20e411879bab Mon Sep 17 00:00:00 2001 From: Dave from DS Date: Thu, 9 Apr 2026 12:43:21 +0300 Subject: [PATCH 04/14] Update src/server.js Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- src/server.js | 39 ++++++++++++++++++++++++++++++--------- 1 file changed, 30 insertions(+), 9 deletions(-) diff --git a/src/server.js b/src/server.js index 756db13..744b5a6 100644 --- a/src/server.js +++ b/src/server.js @@ -606,26 +606,47 @@ function startServer(host, port, openBrowser = true) { if (from) filtered = filtered.filter(s => s.date >= from); if (to) filtered = filtered.filter(s => s.date <= to); const fingerprint = _analyticsFingerprint(filtered, from, to, includeHelpers ? 'h1' : 'h0'); + const placeholderJob = { + state: 'running', + key, + result: null, + partialResult: null, + progress: { done: 0, total: filtered.length, phase: 'cache' }, + error: null, + startedAt: Date.now(), + finishedAt: null, + }; + _jobs.analytics = placeholderJob; sqliteIndex.getAggregateCacheAsync('analytics', fingerprint).then(cached => { if (served) return; if (cached && cached.result) { served = true; - _jobs.analytics = { - state: 'done', key, result: cached.result, partialResult: null, - progress: { done: filtered.length, total: filtered.length, phase: 'cached' }, - error: null, startedAt: cached.computedAt, finishedAt: cached.computedAt, - }; + if (_jobs.analytics === placeholderJob) { + _jobs.analytics = { + state: 'done', key, result: cached.result, partialResult: null, + progress: { done: filtered.length, total: filtered.length, phase: 'cached' }, + error: null, startedAt: cached.computedAt, finishedAt: cached.computedAt, + }; + } json(res, { status: 'done', ...cached.result, _cached: true }); } else { served = true; - _runAnalyticsJob(from, to, includeHelpers); - json(res, _jobResponse(_jobs.analytics)); + if (_jobs.analytics === placeholderJob) { + _runAnalyticsJob(from, to, includeHelpers); + json(res, _jobResponse(_jobs.analytics)); + } else { + json(res, _jobResponse(_jobs.analytics)); + } } }).catch(() => { if (served) return; served = true; - _runAnalyticsJob(from, to, includeHelpers); - json(res, _jobResponse(_jobs.analytics)); + if (_jobs.analytics === placeholderJob) { + _runAnalyticsJob(from, to, includeHelpers); + json(res, _jobResponse(_jobs.analytics)); + } else { + json(res, _jobResponse(_jobs.analytics)); + } }); } catch (e) { if (!served) { From 62bed4c6a3bb4260d2eb5e82bb4b022b927e9136 Mon Sep 17 00:00:00 2001 From: Dave from DS Date: Thu, 9 Apr 2026 12:43:33 +0300 Subject: [PATCH 05/14] Update src/data.js Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- src/data.js | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/src/data.js b/src/data.js index 06f32d9..0b3f05b 100644 --- a/src/data.js +++ b/src/data.js @@ -2343,14 +2343,23 @@ function loadSessions() { if (!summary) { // Create a placeholder from stat + history of same sid (none here) let size = 0; - try { size = fs.statSync(filePath).size; } catch {} + let placeholderTs = 0; + try { + const stat = fs.statSync(filePath); + size = stat.size; + if (Number.isFinite(stat.mtimeMs) && stat.mtimeMs > 0) { + placeholderTs = Math.floor(stat.mtimeMs); + } else if (Number.isFinite(stat.ctimeMs) && stat.ctimeMs > 0) { + placeholderTs = Math.floor(stat.ctimeMs); + } + } catch {} sessions[sid] = { id: sid, tool: 'claude', project: '', project_short: '', - first_ts: 0, - last_ts: 0, + first_ts: placeholderTs, + last_ts: placeholderTs, messages: 0, first_message: '', has_detail: true, From 7b5ddd08e3ddf6d7f362beb3a407509d5bd4dd15 Mon Sep 17 00:00:00 2001 From: Dave from DS Date: Thu, 9 Apr 2026 12:43:43 +0300 Subject: [PATCH 06/14] Update src/data.js Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- src/data.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/data.js b/src/data.js index 0b3f05b..fbe38c0 100644 --- a/src/data.js +++ b/src/data.js @@ -1867,7 +1867,7 @@ function _saveGitRootDiskCache() { // (is_helper=true) all collapse into their own single bucket per project. function computeSessionGroupKey(s) { const proj = (s.project || 'unknown').trim(); - // Normalize first message: lowercase, collapse whitespace, truncate + // Normalize first message: collapse whitespace, trim, truncate let msg = (s.first_message || '').replace(/\s+/g, ' ').trim().slice(0, 200); if (s.is_helper) { // All helpers in the same project collapse into one group. From 0984fd14a099b5c446e1f00d74c0c84e72af460c Mon Sep 17 00:00:00 2001 From: Dave from DS Date: Thu, 9 Apr 2026 12:44:01 +0300 Subject: [PATCH 07/14] Update src/data.js Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- src/data.js | 28 +++++++++++++++++++++++----- 1 file changed, 23 insertions(+), 5 deletions(-) diff --git a/src/data.js b/src/data.js index fbe38c0..4d3f3b7 100644 --- a/src/data.js +++ b/src/data.js @@ -585,19 +585,37 @@ function getWarmingStatus() { // Flush all caches on process exit / signals so partial progress isn't lost let _flushHandlersInstalled = false; +let _flushInProgress = null; function _installFlushHandlers() { if (_flushHandlersInstalled) return; _flushHandlersInstalled = true; - const flushAll = () => { + const flushAllSync = () => { try { _saveParsedDiskCache(true); } catch {} try { _saveCostDiskCache(true); } catch {} try { if (typeof _saveDailyStatsDiskCache === 'function') _saveDailyStatsDiskCache(); } catch {} try { if (typeof _saveGitRootDiskCache === 'function') _saveGitRootDiskCache(); } catch {} - try { if (typeof _flushSqliteIngestBatch === 'function') _flushSqliteIngestBatch(); } catch {} }; - process.on('exit', flushAll); - process.on('SIGINT', () => { flushAll(); process.exit(0); }); - process.on('SIGTERM', () => { flushAll(); process.exit(0); }); + const flushAll = async () => { + flushAllSync(); + try { + if (typeof _flushSqliteIngestBatch === 'function') { + await _flushSqliteIngestBatch(); + } + } catch {} + }; + const handleSignal = async () => { + if (!_flushInProgress) { + _flushInProgress = flushAll(); + } + try { + await _flushInProgress; + } finally { + process.exit(0); + } + }; + process.on('exit', flushAllSync); + process.on('SIGINT', handleSignal); + process.on('SIGTERM', handleSignal); } _installFlushHandlers(); From a66cd0b06e2eb3e86607d33c1034c11ce567f278 Mon Sep 17 00:00:00 2001 From: Dave from DS Date: Thu, 9 Apr 2026 12:44:15 +0300 Subject: [PATCH 08/14] Update src/data.js Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- src/data.js | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/src/data.js b/src/data.js index 4d3f3b7..bc2fcc8 100644 --- a/src/data.js +++ b/src/data.js @@ -3565,15 +3565,16 @@ function buildOpencodeCostCache(sessions) { if (opencodeSessions.length === 0 || !fs.existsSync(OPENCODE_DB)) return cache; try { const batchRows = execFileSync('sqlite3', [ + '-json', OPENCODE_DB, `SELECT session_id, data FROM message WHERE json_extract(data, '$.role') = 'assistant' ORDER BY time_created` ], { encoding: 'utf8', timeout: 30000, windowsHide: true }).trim(); if (batchRows) { - for (const row of batchRows.split('\n')) { - const sepIdx = row.indexOf('|'); - if (sepIdx < 0) continue; - const sessId = row.slice(0, sepIdx); - const jsonStr = row.slice(sepIdx + 1); + const rows = JSON.parse(batchRows); + for (const row of rows) { + const sessId = row.session_id; + const jsonStr = row.data; + if (!sessId || typeof jsonStr !== 'string') continue; try { const msgData = JSON.parse(jsonStr); const t = msgData.tokens || {}; From e63b0fb9ab0afd217319ef3011837cc29e47f5e8 Mon Sep 17 00:00:00 2001 From: David Gornshtein Date: Thu, 9 Apr 2026 12:50:24 +0300 Subject: [PATCH 09/14] fix: address remaining Copilot review comments (#3, #5, #10) - #10: Move `-json` flag BEFORE DB_FILE in both `_exec()` and `_execAsync()`. sqlite3 CLI stops option parsing at the database filename, so `-json` after it was silently ignored (fell back to default text output on strict-POSIX systems). - #3: Add synchronous placeholder job in the leaderboard endpoint before the async cache lookup, same pattern as the analytics endpoint fix from earlier commits. Prevents concurrent requests from each starting their own `_runLeaderboardJob()`. - #5: Guard `_ensureSqliteBackfillRunning()` with a completed-phase check (`_sqliteBackfillStatus.phase === 'done'`). Without this, every `loadSessions()` call after the initial backfill finished would trigger another full directory scan. --- src/data.js | 4 ++++ src/server.js | 43 ++++++++++++++++++++++++++++++++++--------- src/sqlite-index.js | 11 ++++++++--- 3 files changed, 46 insertions(+), 12 deletions(-) diff --git a/src/data.js b/src/data.js index bc2fcc8..69c72d4 100644 --- a/src/data.js +++ b/src/data.js @@ -410,6 +410,10 @@ function getSqliteBackfillStatus() { function _ensureSqliteBackfillRunning() { if (_sqliteBackfillRunning) return; + // Don't re-scan after the initial backfill completed this process lifetime. + // New/changed files are ingested incrementally by the warmer; this full + // scan is only needed once on cold start. + if (_sqliteBackfillStatus.phase === 'done') return; let sqliteIndex; try { sqliteIndex = require('./sqlite-index'); } catch { return; } diff --git a/src/server.js b/src/server.js index 744b5a6..b9a2041 100644 --- a/src/server.js +++ b/src/server.js @@ -735,26 +735,51 @@ function startServer(host, port, openBrowser = true) { const sessions = loadSessions(); const filtered = includeHelpers ? sessions : sessions.filter(s => !s.is_helper); const fingerprint = _analyticsFingerprint(filtered, '', '', includeHelpers ? 'h1' : 'h0'); + // Install a synchronous placeholder job BEFORE the async cache + // read so concurrent requests for the same key don't each kick + // off their own _runLeaderboardJob. Only the request that owns + // the placeholder proceeds to start the real job. + const placeholderJob = { + state: 'running', + key: expectedKey, + result: null, + partialResult: null, + progress: { done: 0, total: filtered.length, phase: 'cache' }, + error: null, + startedAt: Date.now(), + finishedAt: null, + }; + _jobs.leaderboard = placeholderJob; sqliteIndex.getAggregateCacheAsync('leaderboard', fingerprint).then(cached => { if (served) return; if (cached && cached.result) { served = true; - _jobs.leaderboard = { - state: 'done', key: expectedKey, result: cached.result, partialResult: null, - progress: { done: filtered.length, total: filtered.length, phase: 'cached' }, - error: null, startedAt: cached.computedAt, finishedAt: cached.computedAt, - }; + if (_jobs.leaderboard === placeholderJob) { + _jobs.leaderboard = { + state: 'done', key: expectedKey, result: cached.result, partialResult: null, + progress: { done: filtered.length, total: filtered.length, phase: 'cached' }, + error: null, startedAt: cached.computedAt, finishedAt: cached.computedAt, + }; + } json(res, { status: 'done', ...cached.result, _cached: true }); } else { served = true; - _runLeaderboardJob(includeHelpers); - json(res, _jobResponse(_jobs.leaderboard)); + if (_jobs.leaderboard === placeholderJob) { + _runLeaderboardJob(includeHelpers); + json(res, _jobResponse(_jobs.leaderboard)); + } else { + json(res, _jobResponse(_jobs.leaderboard)); + } } }).catch(() => { if (served) return; served = true; - _runLeaderboardJob(includeHelpers); - json(res, _jobResponse(_jobs.leaderboard)); + if (_jobs.leaderboard === placeholderJob) { + _runLeaderboardJob(includeHelpers); + json(res, _jobResponse(_jobs.leaderboard)); + } else { + json(res, _jobResponse(_jobs.leaderboard)); + } }); } catch (e) { if (!served) { diff --git a/src/sqlite-index.js b/src/sqlite-index.js index 3f725eb..6bf0918 100644 --- a/src/sqlite-index.js +++ b/src/sqlite-index.js @@ -33,9 +33,12 @@ const _CMD_BUSY = '.timeout 30000'; function _exec(sql, opts) { opts = opts || {}; - const args = ['-cmd', _CMD_BUSY, DB_FILE]; + // sqlite3 CLI stops option parsing once it sees the database filename, so + // all flags (including -json) must come BEFORE DB_FILE or they're treated + // as part of the SQL input and silently ignored. + const args = ['-cmd', _CMD_BUSY]; if (opts.json) args.push('-json'); - args.push(sql); + args.push(DB_FILE, sql); const r = spawnSync('sqlite3', args, { encoding: 'utf8', maxBuffer: 64 * 1024 * 1024, @@ -58,8 +61,10 @@ function _execJson(sql, opts) { function _execAsync(sql, opts) { opts = opts || {}; return new Promise((resolve, reject) => { - const args = ['-cmd', _CMD_BUSY, DB_FILE]; + // Options must come BEFORE DB_FILE (see _exec comment) + const args = ['-cmd', _CMD_BUSY]; if (opts.json) args.push('-json'); + args.push(DB_FILE); const child = spawn('sqlite3', args, { stdio: ['pipe', 'pipe', 'pipe'], windowsHide: true, From 8bcb1479aa8ac2779bc0a2b83f84caca77b25f9a Mon Sep 17 00:00:00 2001 From: David Gornshtein Date: Thu, 9 Apr 2026 13:59:19 +0300 Subject: [PATCH 10/14] =?UTF-8?q?fix:=20cherry-pick=20upstream=20v6.15.11?= =?UTF-8?q?=20=E2=80=94=20iTerm2=20focus=20+=20tty=20normalize?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - select s (session) inside tab for correct iTerm2 focus - normalize tty: strip /dev/ prefix for reliable matching --- package.json | 2 +- src/terminals.js | 7 +++++-- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/package.json b/package.json index f872d88..d591182 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "codedash-app", - "version": "6.15.10", + "version": "6.15.11", "description": "Dashboard + CLI for Claude Code, Codex & OpenCode sessions. View, search, resume, convert, handoff between agents.", "bin": { "codedash": "./bin/cli.js" diff --git a/src/terminals.js b/src/terminals.js index 9020bf3..19a2b95 100644 --- a/src/terminals.js +++ b/src/terminals.js @@ -251,9 +251,11 @@ function focusTerminalByPid(pid) { return focusCmuxWorkspace(pid); } - // iTerm2: activate and select the right tab/window by tty + // iTerm2: activate and select the right tab/window/session by tty if (detectedTerminal === 'iTerm2' || !detectedTerminal) { try { + // Normalize tty: "ttys005" → "ttys005", "/dev/ttys005" → "ttys005" + const ttyNorm = ttyOut.replace('/dev/', ''); const script = ` tell application "iTerm" activate @@ -261,9 +263,10 @@ function focusTerminalByPid(pid) { repeat with t in tabs of w repeat with s in sessions of t set sessionTTY to tty of s - if sessionTTY contains "${ttyOut}" or "${ttyOut}" contains sessionTTY then + if sessionTTY contains "${ttyNorm}" then select w tell w to select t + tell t to select s return "found" end if end repeat From 2ab3d46128caf355646208d4bd98fe0625369fbf Mon Sep 17 00:00:00 2001 From: David Gornshtein Date: Thu, 9 Apr 2026 14:06:31 +0300 Subject: [PATCH 11/14] feat: vector search via all-MiniLM-L6-v2 + hybrid FTS5+semantic mode MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add `src/embeddings.js` — optional vector search using @huggingface/transformers (pure JS ONNX, no Python/torch needed). Model: Xenova/all-MiniLM-L6-v2, 384-dim embeddings, ~23 MB ONNX. Falls back gracefully when npm package isn't installed. - `/api/search?q=X&mode=text|semantic|hybrid` — three search modes: - `text`: FTS5 keyword MATCH (existing behavior, default fallback) - `semantic`: pure cosine similarity against pre-computed session embeddings - `hybrid`: FTS5 for recall (top 200) → vector re-rank for precision, combined score = 0.3×fts_matches + 0.7×similarity - SQLite `session_embeddings` table stores pre-computed embeddings per session. Populated during SQLite backfill phase 2 (after FTS5 ingest). Batched 32 at a time with setImmediate yields. - `/api/embeddings/status` — reports model availability, dim, count. - Frontend: Text / Hybrid / Semantic toggle buttons next to search bar. Default: hybrid. Re-runs search on mode change. - Cherry-pick upstream v6.15.11: iTerm2 focus fix (select session in tab, normalize tty /dev/ prefix). --- package-lock.json | 999 ++++++++++++++++++++++++++++++++++++++++ package.json | 3 + src/data.js | 36 ++ src/embeddings.js | 233 ++++++++++ src/frontend/app.js | 23 +- src/frontend/index.html | 5 + src/frontend/styles.css | 26 ++ src/server.js | 42 +- 8 files changed, 1359 insertions(+), 8 deletions(-) create mode 100644 package-lock.json create mode 100644 src/embeddings.js diff --git a/package-lock.json b/package-lock.json new file mode 100644 index 0000000..bebf26e --- /dev/null +++ b/package-lock.json @@ -0,0 +1,999 @@ +{ + "name": "codedash-app", + "version": "6.15.11", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "codedash-app", + "version": "6.15.11", + "license": "MIT", + "dependencies": { + "@huggingface/transformers": "^4.0.1" + }, + "bin": { + "codedash": "bin/cli.js" + }, + "engines": { + "node": ">=18" + } + }, + "node_modules/@emnapi/runtime": { + "version": "1.9.2", + "resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.9.2.tgz", + "integrity": "sha512-3U4+MIWHImeyu1wnmVygh5WlgfYDtyf0k8AbLhMFxOipihf6nrWC4syIm/SwEeec0mNSafiiNnMJwbza/Is6Lw==", + "license": "MIT", + "optional": true, + "dependencies": { + "tslib": "^2.4.0" + } + }, + "node_modules/@huggingface/jinja": { + "version": "0.5.6", + "resolved": "https://registry.npmjs.org/@huggingface/jinja/-/jinja-0.5.6.tgz", + "integrity": "sha512-MyMWyLnjqo+KRJYSH7oWNbsOn5onuIvfXYPcc0WOGxU0eHUV7oAYUoQTl2BMdu7ml+ea/bu11UM+EshbeHwtIA==", + "license": "MIT", + "engines": { + "node": ">=18" + } + }, + "node_modules/@huggingface/tokenizers": { + "version": "0.1.3", + "resolved": "https://registry.npmjs.org/@huggingface/tokenizers/-/tokenizers-0.1.3.tgz", + "integrity": "sha512-8rF/RRT10u+kn7YuUbUg0OF30K8rjTc78aHpxT+qJ1uWSqxT1MHi8+9ltwYfkFYJzT/oS+qw3JVfHtNMGAdqyA==", + "license": "Apache-2.0" + }, + "node_modules/@huggingface/transformers": { + "version": "4.0.1", + "resolved": "https://registry.npmjs.org/@huggingface/transformers/-/transformers-4.0.1.tgz", + "integrity": "sha512-tAQYEy+cnW0ku/NxBSjFXCymi+DZa1/JkoGf4McxjzO36CZZIL/J4TF6X7i/tzs75yTjshUDgsvSz03s2xym2A==", + "license": "Apache-2.0", + "dependencies": { + "@huggingface/jinja": "^0.5.6", + "@huggingface/tokenizers": "^0.1.3", + "onnxruntime-node": "1.24.3", + "onnxruntime-web": "1.25.0-dev.20260327-722743c0e2", + "sharp": "^0.34.5" + } + }, + "node_modules/@img/colour": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/@img/colour/-/colour-1.1.0.tgz", + "integrity": "sha512-Td76q7j57o/tLVdgS746cYARfSyxk8iEfRxewL9h4OMzYhbW4TAcppl0mT4eyqXddh6L/jwoM75mo7ixa/pCeQ==", + "license": "MIT", + "engines": { + "node": ">=18" + } + }, + "node_modules/@img/sharp-darwin-arm64": { + "version": "0.34.5", + "resolved": "https://registry.npmjs.org/@img/sharp-darwin-arm64/-/sharp-darwin-arm64-0.34.5.tgz", + "integrity": "sha512-imtQ3WMJXbMY4fxb/Ndp6HBTNVtWCUI0WdobyheGf5+ad6xX8VIDO8u2xE4qc/fr08CKG/7dDseFtn6M6g/r3w==", + "cpu": [ + "arm64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^18.17.0 || ^20.3.0 || >=21.0.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-darwin-arm64": "1.2.4" + } + }, + "node_modules/@img/sharp-darwin-x64": { + "version": "0.34.5", + "resolved": "https://registry.npmjs.org/@img/sharp-darwin-x64/-/sharp-darwin-x64-0.34.5.tgz", + "integrity": "sha512-YNEFAF/4KQ/PeW0N+r+aVVsoIY0/qxxikF2SWdp+NRkmMB7y9LBZAVqQ4yhGCm/H3H270OSykqmQMKLBhBJDEw==", + "cpu": [ + "x64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^18.17.0 || ^20.3.0 || >=21.0.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-darwin-x64": "1.2.4" + } + }, + "node_modules/@img/sharp-libvips-darwin-arm64": { + "version": "1.2.4", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-darwin-arm64/-/sharp-libvips-darwin-arm64-1.2.4.tgz", + "integrity": "sha512-zqjjo7RatFfFoP0MkQ51jfuFZBnVE2pRiaydKJ1G/rHZvnsrHAOcQALIi9sA5co5xenQdTugCvtb1cuf78Vf4g==", + "cpu": [ + "arm64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "darwin" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-darwin-x64": { + "version": "1.2.4", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-darwin-x64/-/sharp-libvips-darwin-x64-1.2.4.tgz", + "integrity": "sha512-1IOd5xfVhlGwX+zXv2N93k0yMONvUlANylbJw1eTah8K/Jtpi15KC+WSiaX/nBmbm2HxRM1gZ0nSdjSsrZbGKg==", + "cpu": [ + "x64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "darwin" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linux-arm": { + "version": "1.2.4", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-arm/-/sharp-libvips-linux-arm-1.2.4.tgz", + "integrity": "sha512-bFI7xcKFELdiNCVov8e44Ia4u2byA+l3XtsAj+Q8tfCwO6BQ8iDojYdvoPMqsKDkuoOo+X6HZA0s0q11ANMQ8A==", + "cpu": [ + "arm" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linux-arm64": { + "version": "1.2.4", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-arm64/-/sharp-libvips-linux-arm64-1.2.4.tgz", + "integrity": "sha512-excjX8DfsIcJ10x1Kzr4RcWe1edC9PquDRRPx3YVCvQv+U5p7Yin2s32ftzikXojb1PIFc/9Mt28/y+iRklkrw==", + "cpu": [ + "arm64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linux-ppc64": { + "version": "1.2.4", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-ppc64/-/sharp-libvips-linux-ppc64-1.2.4.tgz", + "integrity": "sha512-FMuvGijLDYG6lW+b/UvyilUWu5Ayu+3r2d1S8notiGCIyYU/76eig1UfMmkZ7vwgOrzKzlQbFSuQfgm7GYUPpA==", + "cpu": [ + "ppc64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linux-riscv64": { + "version": "1.2.4", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-riscv64/-/sharp-libvips-linux-riscv64-1.2.4.tgz", + "integrity": "sha512-oVDbcR4zUC0ce82teubSm+x6ETixtKZBh/qbREIOcI3cULzDyb18Sr/Wcyx7NRQeQzOiHTNbZFF1UwPS2scyGA==", + "cpu": [ + "riscv64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linux-s390x": { + "version": "1.2.4", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-s390x/-/sharp-libvips-linux-s390x-1.2.4.tgz", + "integrity": "sha512-qmp9VrzgPgMoGZyPvrQHqk02uyjA0/QrTO26Tqk6l4ZV0MPWIW6LTkqOIov+J1yEu7MbFQaDpwdwJKhbJvuRxQ==", + "cpu": [ + "s390x" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linux-x64": { + "version": "1.2.4", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-x64/-/sharp-libvips-linux-x64-1.2.4.tgz", + "integrity": "sha512-tJxiiLsmHc9Ax1bz3oaOYBURTXGIRDODBqhveVHonrHJ9/+k89qbLl0bcJns+e4t4rvaNBxaEZsFtSfAdquPrw==", + "cpu": [ + "x64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linuxmusl-arm64": { + "version": "1.2.4", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linuxmusl-arm64/-/sharp-libvips-linuxmusl-arm64-1.2.4.tgz", + "integrity": "sha512-FVQHuwx1IIuNow9QAbYUzJ+En8KcVm9Lk5+uGUQJHaZmMECZmOlix9HnH7n1TRkXMS0pGxIJokIVB9SuqZGGXw==", + "cpu": [ + "arm64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linuxmusl-x64": { + "version": "1.2.4", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linuxmusl-x64/-/sharp-libvips-linuxmusl-x64-1.2.4.tgz", + "integrity": "sha512-+LpyBk7L44ZIXwz/VYfglaX/okxezESc6UxDSoyo2Ks6Jxc4Y7sGjpgU9s4PMgqgjj1gZCylTieNamqA1MF7Dg==", + "cpu": [ + "x64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-linux-arm": { + "version": "0.34.5", + "resolved": "https://registry.npmjs.org/@img/sharp-linux-arm/-/sharp-linux-arm-0.34.5.tgz", + "integrity": "sha512-9dLqsvwtg1uuXBGZKsxem9595+ujv0sJ6Vi8wcTANSFpwV/GONat5eCkzQo/1O6zRIkh0m/8+5BjrRr7jDUSZw==", + "cpu": [ + "arm" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^18.17.0 || ^20.3.0 || >=21.0.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linux-arm": "1.2.4" + } + }, + "node_modules/@img/sharp-linux-arm64": { + "version": "0.34.5", + "resolved": "https://registry.npmjs.org/@img/sharp-linux-arm64/-/sharp-linux-arm64-0.34.5.tgz", + "integrity": "sha512-bKQzaJRY/bkPOXyKx5EVup7qkaojECG6NLYswgktOZjaXecSAeCWiZwwiFf3/Y+O1HrauiE3FVsGxFg8c24rZg==", + "cpu": [ + "arm64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^18.17.0 || ^20.3.0 || >=21.0.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linux-arm64": "1.2.4" + } + }, + "node_modules/@img/sharp-linux-ppc64": { + "version": "0.34.5", + "resolved": "https://registry.npmjs.org/@img/sharp-linux-ppc64/-/sharp-linux-ppc64-0.34.5.tgz", + "integrity": "sha512-7zznwNaqW6YtsfrGGDA6BRkISKAAE1Jo0QdpNYXNMHu2+0dTrPflTLNkpc8l7MUP5M16ZJcUvysVWWrMefZquA==", + "cpu": [ + "ppc64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^18.17.0 || ^20.3.0 || >=21.0.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linux-ppc64": "1.2.4" + } + }, + "node_modules/@img/sharp-linux-riscv64": { + "version": "0.34.5", + "resolved": "https://registry.npmjs.org/@img/sharp-linux-riscv64/-/sharp-linux-riscv64-0.34.5.tgz", + "integrity": "sha512-51gJuLPTKa7piYPaVs8GmByo7/U7/7TZOq+cnXJIHZKavIRHAP77e3N2HEl3dgiqdD/w0yUfiJnII77PuDDFdw==", + "cpu": [ + "riscv64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^18.17.0 || ^20.3.0 || >=21.0.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linux-riscv64": "1.2.4" + } + }, + "node_modules/@img/sharp-linux-s390x": { + "version": "0.34.5", + "resolved": "https://registry.npmjs.org/@img/sharp-linux-s390x/-/sharp-linux-s390x-0.34.5.tgz", + "integrity": "sha512-nQtCk0PdKfho3eC5MrbQoigJ2gd1CgddUMkabUj+rBevs8tZ2cULOx46E7oyX+04WGfABgIwmMC0VqieTiR4jg==", + "cpu": [ + "s390x" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^18.17.0 || ^20.3.0 || >=21.0.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linux-s390x": "1.2.4" + } + }, + "node_modules/@img/sharp-linux-x64": { + "version": "0.34.5", + "resolved": "https://registry.npmjs.org/@img/sharp-linux-x64/-/sharp-linux-x64-0.34.5.tgz", + "integrity": "sha512-MEzd8HPKxVxVenwAa+JRPwEC7QFjoPWuS5NZnBt6B3pu7EG2Ge0id1oLHZpPJdn3OQK+BQDiw9zStiHBTJQQQQ==", + "cpu": [ + "x64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^18.17.0 || ^20.3.0 || >=21.0.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linux-x64": "1.2.4" + } + }, + "node_modules/@img/sharp-linuxmusl-arm64": { + "version": "0.34.5", + "resolved": "https://registry.npmjs.org/@img/sharp-linuxmusl-arm64/-/sharp-linuxmusl-arm64-0.34.5.tgz", + "integrity": "sha512-fprJR6GtRsMt6Kyfq44IsChVZeGN97gTD331weR1ex1c1rypDEABN6Tm2xa1wE6lYb5DdEnk03NZPqA7Id21yg==", + "cpu": [ + "arm64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^18.17.0 || ^20.3.0 || >=21.0.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linuxmusl-arm64": "1.2.4" + } + }, + "node_modules/@img/sharp-linuxmusl-x64": { + "version": "0.34.5", + "resolved": "https://registry.npmjs.org/@img/sharp-linuxmusl-x64/-/sharp-linuxmusl-x64-0.34.5.tgz", + "integrity": "sha512-Jg8wNT1MUzIvhBFxViqrEhWDGzqymo3sV7z7ZsaWbZNDLXRJZoRGrjulp60YYtV4wfY8VIKcWidjojlLcWrd8Q==", + "cpu": [ + "x64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^18.17.0 || ^20.3.0 || >=21.0.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linuxmusl-x64": "1.2.4" + } + }, + "node_modules/@img/sharp-wasm32": { + "version": "0.34.5", + "resolved": "https://registry.npmjs.org/@img/sharp-wasm32/-/sharp-wasm32-0.34.5.tgz", + "integrity": "sha512-OdWTEiVkY2PHwqkbBI8frFxQQFekHaSSkUIJkwzclWZe64O1X4UlUjqqqLaPbUpMOQk6FBu/HtlGXNblIs0huw==", + "cpu": [ + "wasm32" + ], + "license": "Apache-2.0 AND LGPL-3.0-or-later AND MIT", + "optional": true, + "dependencies": { + "@emnapi/runtime": "^1.7.0" + }, + "engines": { + "node": "^18.17.0 || ^20.3.0 || >=21.0.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-win32-arm64": { + "version": "0.34.5", + "resolved": "https://registry.npmjs.org/@img/sharp-win32-arm64/-/sharp-win32-arm64-0.34.5.tgz", + "integrity": "sha512-WQ3AgWCWYSb2yt+IG8mnC6Jdk9Whs7O0gxphblsLvdhSpSTtmu69ZG1Gkb6NuvxsNACwiPV6cNSZNzt0KPsw7g==", + "cpu": [ + "arm64" + ], + "license": "Apache-2.0 AND LGPL-3.0-or-later", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": "^18.17.0 || ^20.3.0 || >=21.0.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-win32-ia32": { + "version": "0.34.5", + "resolved": "https://registry.npmjs.org/@img/sharp-win32-ia32/-/sharp-win32-ia32-0.34.5.tgz", + "integrity": "sha512-FV9m/7NmeCmSHDD5j4+4pNI8Cp3aW+JvLoXcTUo0IqyjSfAZJ8dIUmijx1qaJsIiU+Hosw6xM5KijAWRJCSgNg==", + "cpu": [ + "ia32" + ], + "license": "Apache-2.0 AND LGPL-3.0-or-later", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": "^18.17.0 || ^20.3.0 || >=21.0.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-win32-x64": { + "version": "0.34.5", + "resolved": "https://registry.npmjs.org/@img/sharp-win32-x64/-/sharp-win32-x64-0.34.5.tgz", + "integrity": "sha512-+29YMsqY2/9eFEiW93eqWnuLcWcufowXewwSNIT6UwZdUUCrM3oFjMWH/Z6/TMmb4hlFenmfAVbpWeup2jryCw==", + "cpu": [ + "x64" + ], + "license": "Apache-2.0 AND LGPL-3.0-or-later", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": "^18.17.0 || ^20.3.0 || >=21.0.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@protobufjs/aspromise": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/@protobufjs/aspromise/-/aspromise-1.1.2.tgz", + "integrity": "sha512-j+gKExEuLmKwvz3OgROXtrJ2UG2x8Ch2YZUxahh+s1F2HZ+wAceUNLkvy6zKCPVRkU++ZWQrdxsUeQXmcg4uoQ==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/base64": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/@protobufjs/base64/-/base64-1.1.2.tgz", + "integrity": "sha512-AZkcAA5vnN/v4PDqKyMR5lx7hZttPDgClv83E//FMNhR2TMcLUhfRUBHCmSl0oi9zMgDDqRUJkSxO3wm85+XLg==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/codegen": { + "version": "2.0.4", + "resolved": "https://registry.npmjs.org/@protobufjs/codegen/-/codegen-2.0.4.tgz", + "integrity": "sha512-YyFaikqM5sH0ziFZCN3xDC7zeGaB/d0IUb9CATugHWbd1FRFwWwt4ld4OYMPWu5a3Xe01mGAULCdqhMlPl29Jg==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/eventemitter": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/@protobufjs/eventemitter/-/eventemitter-1.1.0.tgz", + "integrity": "sha512-j9ednRT81vYJ9OfVuXG6ERSTdEL1xVsNgqpkxMsbIabzSo3goCjDIveeGv5d03om39ML71RdmrGNjG5SReBP/Q==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/fetch": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/@protobufjs/fetch/-/fetch-1.1.0.tgz", + "integrity": "sha512-lljVXpqXebpsijW71PZaCYeIcE5on1w5DlQy5WH6GLbFryLUrBD4932W/E2BSpfRJWseIL4v/KPgBFxDOIdKpQ==", + "license": "BSD-3-Clause", + "dependencies": { + "@protobufjs/aspromise": "^1.1.1", + "@protobufjs/inquire": "^1.1.0" + } + }, + "node_modules/@protobufjs/float": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/@protobufjs/float/-/float-1.0.2.tgz", + "integrity": "sha512-Ddb+kVXlXst9d+R9PfTIxh1EdNkgoRe5tOX6t01f1lYWOvJnSPDBlG241QLzcyPdoNTsblLUdujGSE4RzrTZGQ==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/inquire": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/@protobufjs/inquire/-/inquire-1.1.0.tgz", + "integrity": "sha512-kdSefcPdruJiFMVSbn801t4vFK7KB/5gd2fYvrxhuJYg8ILrmn9SKSX2tZdV6V+ksulWqS7aXjBcRXl3wHoD9Q==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/path": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/@protobufjs/path/-/path-1.1.2.tgz", + "integrity": "sha512-6JOcJ5Tm08dOHAbdR3GrvP+yUUfkjG5ePsHYczMFLq3ZmMkAD98cDgcT2iA1lJ9NVwFd4tH/iSSoe44YWkltEA==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/pool": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/@protobufjs/pool/-/pool-1.1.0.tgz", + "integrity": "sha512-0kELaGSIDBKvcgS4zkjz1PeddatrjYcmMWOlAuAPwAeccUrPHdUqo/J6LiymHHEiJT5NrF1UVwxY14f+fy4WQw==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/utf8": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/@protobufjs/utf8/-/utf8-1.1.0.tgz", + "integrity": "sha512-Vvn3zZrhQZkkBE8LSuW3em98c0FwgO4nxzv6OdSxPKJIEKY2bGbHn+mhGIPerzI4twdxaP8/0+06HBpwf345Lw==", + "license": "BSD-3-Clause" + }, + "node_modules/@types/node": { + "version": "25.5.2", + "resolved": "https://registry.npmjs.org/@types/node/-/node-25.5.2.tgz", + "integrity": "sha512-tO4ZIRKNC+MDWV4qKVZe3Ql/woTnmHDr5JD8UI5hn2pwBrHEwOEMZK7WlNb5RKB6EoJ02gwmQS9OrjuFnZYdpg==", + "license": "MIT", + "dependencies": { + "undici-types": "~7.18.0" + } + }, + "node_modules/adm-zip": { + "version": "0.5.17", + "resolved": "https://registry.npmjs.org/adm-zip/-/adm-zip-0.5.17.tgz", + "integrity": "sha512-+Ut8d9LLqwEvHHJl1+PIHqoyDxFgVN847JTVM3Izi3xHDWPE4UtzzXysMZQs64DMcrJfBeS/uoEP4AD3HQHnQQ==", + "license": "MIT", + "engines": { + "node": ">=12.0" + } + }, + "node_modules/boolean": { + "version": "3.2.0", + "resolved": "https://registry.npmjs.org/boolean/-/boolean-3.2.0.tgz", + "integrity": "sha512-d0II/GO9uf9lfUHH2BQsjxzRJZBdsjgsBiW4BvhWk/3qoKwQFjIDVN19PfX8F2D/r9PCMTtLWjYVCFrpeYUzsw==", + "deprecated": "Package no longer supported. Contact Support at https://www.npmjs.com/support for more info.", + "license": "MIT" + }, + "node_modules/define-data-property": { + "version": "1.1.4", + "resolved": "https://registry.npmjs.org/define-data-property/-/define-data-property-1.1.4.tgz", + "integrity": "sha512-rBMvIzlpA8v6E+SJZoo++HAYqsLrkg7MSfIinMPFhmkorw7X+dOXVJQs+QT69zGkzMyfDnIMN2Wid1+NbL3T+A==", + "license": "MIT", + "dependencies": { + "es-define-property": "^1.0.0", + "es-errors": "^1.3.0", + "gopd": "^1.0.1" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/define-properties": { + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/define-properties/-/define-properties-1.2.1.tgz", + "integrity": "sha512-8QmQKqEASLd5nx0U1B1okLElbUuuttJ/AnYmRXbbbGDWh6uS208EjD4Xqq/I9wK7u0v6O08XhTWnt5XtEbR6Dg==", + "license": "MIT", + "dependencies": { + "define-data-property": "^1.0.1", + "has-property-descriptors": "^1.0.0", + "object-keys": "^1.1.1" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/detect-libc": { + "version": "2.1.2", + "resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz", + "integrity": "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==", + "license": "Apache-2.0", + "engines": { + "node": ">=8" + } + }, + "node_modules/detect-node": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/detect-node/-/detect-node-2.1.0.tgz", + "integrity": "sha512-T0NIuQpnTvFDATNuHN5roPwSBG83rFsuO+MXXH9/3N1eFbn4wcPjttvjMLEPWJ0RGUYgQE7cGgS3tNxbqCGM7g==", + "license": "MIT" + }, + "node_modules/es-define-property": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/es-define-property/-/es-define-property-1.0.1.tgz", + "integrity": "sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/es-errors": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/es-errors/-/es-errors-1.3.0.tgz", + "integrity": "sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/es6-error": { + "version": "4.1.1", + "resolved": "https://registry.npmjs.org/es6-error/-/es6-error-4.1.1.tgz", + "integrity": "sha512-Um/+FxMr9CISWh0bi5Zv0iOD+4cFh5qLeks1qhAopKVAJw3drgKbKySikp7wGhDL0HPeaja0P5ULZrxLkniUVg==", + "license": "MIT" + }, + "node_modules/escape-string-regexp": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/escape-string-regexp/-/escape-string-regexp-4.0.0.tgz", + "integrity": "sha512-TtpcNJ3XAzx3Gq8sWRzJaVajRs0uVxA2YAkdb1jm2YkPz4G6egUFAyA3n5vtEIZefPk5Wa4UXbKuS5fKkJWdgA==", + "license": "MIT", + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/flatbuffers": { + "version": "25.9.23", + "resolved": "https://registry.npmjs.org/flatbuffers/-/flatbuffers-25.9.23.tgz", + "integrity": "sha512-MI1qs7Lo4Syw0EOzUl0xjs2lsoeqFku44KpngfIduHBYvzm8h2+7K8YMQh1JtVVVrUvhLpNwqVi4DERegUJhPQ==", + "license": "Apache-2.0" + }, + "node_modules/global-agent": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/global-agent/-/global-agent-3.0.0.tgz", + "integrity": "sha512-PT6XReJ+D07JvGoxQMkT6qji/jVNfX/h364XHZOWeRzy64sSFr+xJ5OX7LI3b4MPQzdL4H8Y8M0xzPpsVMwA8Q==", + "license": "BSD-3-Clause", + "dependencies": { + "boolean": "^3.0.1", + "es6-error": "^4.1.1", + "matcher": "^3.0.0", + "roarr": "^2.15.3", + "semver": "^7.3.2", + "serialize-error": "^7.0.1" + }, + "engines": { + "node": ">=10.0" + } + }, + "node_modules/globalthis": { + "version": "1.0.4", + "resolved": "https://registry.npmjs.org/globalthis/-/globalthis-1.0.4.tgz", + "integrity": "sha512-DpLKbNU4WylpxJykQujfCcwYWiV/Jhm50Goo0wrVILAv5jOr9d+H+UR3PhSCD2rCCEIg0uc+G+muBTwD54JhDQ==", + "license": "MIT", + "dependencies": { + "define-properties": "^1.2.1", + "gopd": "^1.0.1" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/gopd": { + "version": "1.2.0", + "resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz", + "integrity": "sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/guid-typescript": { + "version": "1.0.9", + "resolved": "https://registry.npmjs.org/guid-typescript/-/guid-typescript-1.0.9.tgz", + "integrity": "sha512-Y8T4vYhEfwJOTbouREvG+3XDsjr8E3kIr7uf+JZ0BYloFsttiHU0WfvANVsR7TxNUJa/WpCnw/Ino/p+DeBhBQ==", + "license": "ISC" + }, + "node_modules/has-property-descriptors": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/has-property-descriptors/-/has-property-descriptors-1.0.2.tgz", + "integrity": "sha512-55JNKuIW+vq4Ke1BjOTjM2YctQIvCT7GFzHwmfZPGo5wnrgkid0YQtnAleFSqumZm4az3n2BS+erby5ipJdgrg==", + "license": "MIT", + "dependencies": { + "es-define-property": "^1.0.0" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/json-stringify-safe": { + "version": "5.0.1", + "resolved": "https://registry.npmjs.org/json-stringify-safe/-/json-stringify-safe-5.0.1.tgz", + "integrity": "sha512-ZClg6AaYvamvYEE82d3Iyd3vSSIjQ+odgjaTzRuO3s7toCdFKczob2i0zCh7JE8kWn17yvAWhUVxvqGwUalsRA==", + "license": "ISC" + }, + "node_modules/long": { + "version": "5.3.2", + "resolved": "https://registry.npmjs.org/long/-/long-5.3.2.tgz", + "integrity": "sha512-mNAgZ1GmyNhD7AuqnTG3/VQ26o760+ZYBPKjPvugO8+nLbYfX6TVpJPseBvopbdY+qpZ/lKUnmEc1LeZYS3QAA==", + "license": "Apache-2.0" + }, + "node_modules/matcher": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/matcher/-/matcher-3.0.0.tgz", + "integrity": "sha512-OkeDaAZ/bQCxeFAozM55PKcKU0yJMPGifLwV4Qgjitu+5MoAfSQN4lsLJeXZ1b8w0x+/Emda6MZgXS1jvsapng==", + "license": "MIT", + "dependencies": { + "escape-string-regexp": "^4.0.0" + }, + "engines": { + "node": ">=10" + } + }, + "node_modules/object-keys": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/object-keys/-/object-keys-1.1.1.tgz", + "integrity": "sha512-NuAESUOUMrlIXOfHKzD6bpPu3tYt3xvjNdRIQ+FeT0lNb4K8WR70CaDxhuNguS2XG+GjkyMwOzsN5ZktImfhLA==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/onnxruntime-common": { + "version": "1.24.3", + "resolved": "https://registry.npmjs.org/onnxruntime-common/-/onnxruntime-common-1.24.3.tgz", + "integrity": "sha512-GeuPZO6U/LBJXvwdaqHbuUmoXiEdeCjWi/EG7Y1HNnDwJYuk6WUbNXpF6luSUY8yASul3cmUlLGrCCL1ZgVXqA==", + "license": "MIT" + }, + "node_modules/onnxruntime-node": { + "version": "1.24.3", + "resolved": "https://registry.npmjs.org/onnxruntime-node/-/onnxruntime-node-1.24.3.tgz", + "integrity": "sha512-JH7+czbc8ALA819vlTgcV+Q214/+VjGeBHDjX81+ZCD0PCVCIFGFNtT0V4sXG/1JXypKPgScQcB3ij/hk3YnTg==", + "hasInstallScript": true, + "license": "MIT", + "os": [ + "win32", + "darwin", + "linux" + ], + "dependencies": { + "adm-zip": "^0.5.16", + "global-agent": "^3.0.0", + "onnxruntime-common": "1.24.3" + } + }, + "node_modules/onnxruntime-web": { + "version": "1.25.0-dev.20260327-722743c0e2", + "resolved": "https://registry.npmjs.org/onnxruntime-web/-/onnxruntime-web-1.25.0-dev.20260327-722743c0e2.tgz", + "integrity": "sha512-8PXdZy4Ekhg10CLg+cFFt39b4tFDGMRJB6lGjnQL6eA+2boUQYDymZ0gtxiS+H6oIWoCjQp/ziyirvFbaFKfiw==", + "license": "MIT", + "dependencies": { + "flatbuffers": "^25.1.24", + "guid-typescript": "^1.0.9", + "long": "^5.2.3", + "onnxruntime-common": "1.24.0-dev.20251116-b39e144322", + "platform": "^1.3.6", + "protobufjs": "^7.2.4" + } + }, + "node_modules/onnxruntime-web/node_modules/onnxruntime-common": { + "version": "1.24.0-dev.20251116-b39e144322", + "resolved": "https://registry.npmjs.org/onnxruntime-common/-/onnxruntime-common-1.24.0-dev.20251116-b39e144322.tgz", + "integrity": "sha512-BOoomdHYmNRL5r4iQ4bMvsl2t0/hzVQ3OM3PHD0gxeXu1PmggqBv3puZicEUVOA3AtHHYmqZtjMj9FOfGrATTw==", + "license": "MIT" + }, + "node_modules/platform": { + "version": "1.3.6", + "resolved": "https://registry.npmjs.org/platform/-/platform-1.3.6.tgz", + "integrity": "sha512-fnWVljUchTro6RiCFvCXBbNhJc2NijN7oIQxbwsyL0buWJPG85v81ehlHI9fXrJsMNgTofEoWIQeClKpgxFLrg==", + "license": "MIT" + }, + "node_modules/protobufjs": { + "version": "7.5.4", + "resolved": "https://registry.npmjs.org/protobufjs/-/protobufjs-7.5.4.tgz", + "integrity": "sha512-CvexbZtbov6jW2eXAvLukXjXUW1TzFaivC46BpWc/3BpcCysb5Vffu+B3XHMm8lVEuy2Mm4XGex8hBSg1yapPg==", + "hasInstallScript": true, + "license": "BSD-3-Clause", + "dependencies": { + "@protobufjs/aspromise": "^1.1.2", + "@protobufjs/base64": "^1.1.2", + "@protobufjs/codegen": "^2.0.4", + "@protobufjs/eventemitter": "^1.1.0", + "@protobufjs/fetch": "^1.1.0", + "@protobufjs/float": "^1.0.2", + "@protobufjs/inquire": "^1.1.0", + "@protobufjs/path": "^1.1.2", + "@protobufjs/pool": "^1.1.0", + "@protobufjs/utf8": "^1.1.0", + "@types/node": ">=13.7.0", + "long": "^5.0.0" + }, + "engines": { + "node": ">=12.0.0" + } + }, + "node_modules/roarr": { + "version": "2.15.4", + "resolved": "https://registry.npmjs.org/roarr/-/roarr-2.15.4.tgz", + "integrity": "sha512-CHhPh+UNHD2GTXNYhPWLnU8ONHdI+5DI+4EYIAOaiD63rHeYlZvyh8P+in5999TTSFgUYuKUAjzRI4mdh/p+2A==", + "license": "BSD-3-Clause", + "dependencies": { + "boolean": "^3.0.1", + "detect-node": "^2.0.4", + "globalthis": "^1.0.1", + "json-stringify-safe": "^5.0.1", + "semver-compare": "^1.0.0", + "sprintf-js": "^1.1.2" + }, + "engines": { + "node": ">=8.0" + } + }, + "node_modules/semver": { + "version": "7.7.4", + "resolved": "https://registry.npmjs.org/semver/-/semver-7.7.4.tgz", + "integrity": "sha512-vFKC2IEtQnVhpT78h1Yp8wzwrf8CM+MzKMHGJZfBtzhZNycRFnXsHk6E5TxIkkMsgNS7mdX3AGB7x2QM2di4lA==", + "license": "ISC", + "bin": { + "semver": "bin/semver.js" + }, + "engines": { + "node": ">=10" + } + }, + "node_modules/semver-compare": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/semver-compare/-/semver-compare-1.0.0.tgz", + "integrity": "sha512-YM3/ITh2MJ5MtzaM429anh+x2jiLVjqILF4m4oyQB18W7Ggea7BfqdH/wGMK7dDiMghv/6WG7znWMwUDzJiXow==", + "license": "MIT" + }, + "node_modules/serialize-error": { + "version": "7.0.1", + "resolved": "https://registry.npmjs.org/serialize-error/-/serialize-error-7.0.1.tgz", + "integrity": "sha512-8I8TjW5KMOKsZQTvoxjuSIa7foAwPWGOts+6o7sgjz41/qMD9VQHEDxi6PBvK2l0MXUmqZyNpUK+T2tQaaElvw==", + "license": "MIT", + "dependencies": { + "type-fest": "^0.13.1" + }, + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/sharp": { + "version": "0.34.5", + "resolved": "https://registry.npmjs.org/sharp/-/sharp-0.34.5.tgz", + "integrity": "sha512-Ou9I5Ft9WNcCbXrU9cMgPBcCK8LiwLqcbywW3t4oDV37n1pzpuNLsYiAV8eODnjbtQlSDwZ2cUEeQz4E54Hltg==", + "hasInstallScript": true, + "license": "Apache-2.0", + "dependencies": { + "@img/colour": "^1.0.0", + "detect-libc": "^2.1.2", + "semver": "^7.7.3" + }, + "engines": { + "node": "^18.17.0 || ^20.3.0 || >=21.0.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-darwin-arm64": "0.34.5", + "@img/sharp-darwin-x64": "0.34.5", + "@img/sharp-libvips-darwin-arm64": "1.2.4", + "@img/sharp-libvips-darwin-x64": "1.2.4", + "@img/sharp-libvips-linux-arm": "1.2.4", + "@img/sharp-libvips-linux-arm64": "1.2.4", + "@img/sharp-libvips-linux-ppc64": "1.2.4", + "@img/sharp-libvips-linux-riscv64": "1.2.4", + "@img/sharp-libvips-linux-s390x": "1.2.4", + "@img/sharp-libvips-linux-x64": "1.2.4", + "@img/sharp-libvips-linuxmusl-arm64": "1.2.4", + "@img/sharp-libvips-linuxmusl-x64": "1.2.4", + "@img/sharp-linux-arm": "0.34.5", + "@img/sharp-linux-arm64": "0.34.5", + "@img/sharp-linux-ppc64": "0.34.5", + "@img/sharp-linux-riscv64": "0.34.5", + "@img/sharp-linux-s390x": "0.34.5", + "@img/sharp-linux-x64": "0.34.5", + "@img/sharp-linuxmusl-arm64": "0.34.5", + "@img/sharp-linuxmusl-x64": "0.34.5", + "@img/sharp-wasm32": "0.34.5", + "@img/sharp-win32-arm64": "0.34.5", + "@img/sharp-win32-ia32": "0.34.5", + "@img/sharp-win32-x64": "0.34.5" + } + }, + "node_modules/sprintf-js": { + "version": "1.1.3", + "resolved": "https://registry.npmjs.org/sprintf-js/-/sprintf-js-1.1.3.tgz", + "integrity": "sha512-Oo+0REFV59/rz3gfJNKQiBlwfHaSESl1pcGyABQsnnIfWOFt6JNj5gCog2U6MLZ//IGYD+nA8nI+mTShREReaA==", + "license": "BSD-3-Clause" + }, + "node_modules/tslib": { + "version": "2.8.1", + "resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz", + "integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==", + "license": "0BSD", + "optional": true + }, + "node_modules/type-fest": { + "version": "0.13.1", + "resolved": "https://registry.npmjs.org/type-fest/-/type-fest-0.13.1.tgz", + "integrity": "sha512-34R7HTnG0XIJcBSn5XhDd7nNFPRcXYRZrBB2O2jdKqYODldSzBAqzsWoZYYvduky73toYS/ESqxPvkDf/F0XMg==", + "license": "(MIT OR CC0-1.0)", + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/undici-types": { + "version": "7.18.2", + "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-7.18.2.tgz", + "integrity": "sha512-AsuCzffGHJybSaRrmr5eHr81mwJU3kjw6M+uprWvCXiNeN9SOGwQ3Jn8jb8m3Z6izVgknn1R0FTCEAP2QrLY/w==", + "license": "MIT" + } + } +} diff --git a/package.json b/package.json index d591182..dd45de6 100644 --- a/package.json +++ b/package.json @@ -32,5 +32,8 @@ "license": "MIT", "engines": { "node": ">=18" + }, + "dependencies": { + "@huggingface/transformers": "^4.0.1" } } diff --git a/src/data.js b/src/data.js index 69c72d4..5317dcc 100644 --- a/src/data.js +++ b/src/data.js @@ -498,6 +498,42 @@ function _ensureSqliteBackfillRunning() { } await _flushSqliteIngestBatch(); + // Phase 2: compute embeddings for sessions that don't have them yet + // (requires optional @huggingface/transformers npm package) + try { + const embeddings = require('./embeddings'); + if (embeddings.isAvailable()) { + _sqliteBackfillStatus.phase = 'computing embeddings'; + embeddings.ensureEmbeddingTable(); + const existingCount = embeddings.getEmbeddingCount(); + // Get sessions without embeddings + const allSessionRows = sqliteIndex._execJson(`SELECT id, first_message FROM sessions WHERE id NOT IN (SELECT session_id FROM session_embeddings) AND first_message != '' LIMIT 5000`); + if (allSessionRows.length > 0) { + _sqliteBackfillStatus.total = allSessionRows.length; + _sqliteBackfillStatus.done = 0; + const BATCH = 32; + for (let i = 0; i < allSessionRows.length; i += BATCH) { + const batch = allSessionRows.slice(i, i + BATCH); + const texts = batch.map(r => (r.first_message || '').slice(0, 512)); + try { + const embs = await embeddings.embedBatch(texts); + const rows = batch.map((r, j) => ({ + session_id: r.id, + embedding: embs[j], + model: embeddings.MODEL_ID, + })); + embeddings.storeEmbeddings(rows); + } catch (e) { + // Model download/init error — skip embeddings + break; + } + _sqliteBackfillStatus.done = Math.min(i + BATCH, allSessionRows.length); + await new Promise(r => setImmediate(r)); + } + } + } + } catch {} + _sqliteBackfillStatus.phase = 'done'; _sqliteBackfillStatus.finishedAt = Date.now(); } catch (e) { diff --git a/src/embeddings.js b/src/embeddings.js new file mode 100644 index 0000000..2f22695 --- /dev/null +++ b/src/embeddings.js @@ -0,0 +1,233 @@ +// Vector search via sentence embeddings (optional — requires @huggingface/transformers). +// +// Uses all-MiniLM-L6-v2 (384-dim, ~23 MB ONNX) for semantic similarity search +// across session first_messages and message content. Falls back gracefully if +// the npm package isn't installed. +// +// Architecture: +// 1. Pre-compute embeddings for each session's first_message during SQLite +// backfill and store them as JSON arrays in the `session_embeddings` table. +// 2. On search: embed the query, load candidate embeddings from SQLite, +// compute cosine similarity, return top-K. +// 3. Hybrid mode: FTS5 for recall (top 200) → vector re-rank for precision. + +const { spawnSync } = require('child_process'); +const path = require('path'); +const os = require('os'); + +// ── Lazy-load transformers.js ────────────────────────────── +let _pipeline = null; +let _extractor = null; +let _loading = false; +let _loadPromise = null; +let _available = null; // null = unknown, true/false after first check + +const MODEL_ID = 'Xenova/all-MiniLM-L6-v2'; +const EMBEDDING_DIM = 384; + +async function _ensureModel() { + if (_extractor) return _extractor; + if (_loadPromise) return _loadPromise; + + _loadPromise = (async () => { + try { + const { pipeline } = await import('@huggingface/transformers'); + _extractor = await pipeline('feature-extraction', MODEL_ID, { + device: 'cpu', + dtype: 'fp32', + }); + _available = true; + return _extractor; + } catch (e) { + _available = false; + _loadPromise = null; + throw new Error('Vector search unavailable: ' + e.message); + } + })(); + + return _loadPromise; +} + +function isAvailable() { + if (_available !== null) return _available; + try { + require.resolve('@huggingface/transformers'); + _available = true; // module exists, model may still need download + return true; + } catch { + _available = false; + return false; + } +} + +// ── Embedding computation ─────────────────────────────────── + +async function embed(text) { + const extractor = await _ensureModel(); + const result = await extractor(text, { pooling: 'mean', normalize: true }); + return Array.from(result.data); +} + +async function embedBatch(texts, batchSize) { + batchSize = batchSize || 32; + const extractor = await _ensureModel(); + const all = []; + for (let i = 0; i < texts.length; i += batchSize) { + const batch = texts.slice(i, i + batchSize); + const results = await extractor(batch, { pooling: 'mean', normalize: true }); + // results.data is a flat Float32Array of shape [batch, dim] + for (let j = 0; j < batch.length; j++) { + const start = j * EMBEDDING_DIM; + all.push(Array.from(results.data.slice(start, start + EMBEDDING_DIM))); + } + } + return all; +} + +// ── Cosine similarity ─────────────────────────────────────── + +function cosineSimilarity(a, b) { + let dot = 0; + for (let i = 0; i < a.length; i++) dot += a[i] * b[i]; + // Vectors are already L2-normalized by the pipeline, so dot = cosine + return dot; +} + +// ── SQLite storage ────────────────────────────────────────── + +function _sqliteIndex() { + return require('./sqlite-index'); +} + +function ensureEmbeddingTable() { + const sq = _sqliteIndex(); + sq._exec(` + CREATE TABLE IF NOT EXISTS session_embeddings ( + session_id TEXT PRIMARY KEY, + embedding TEXT NOT NULL, + model TEXT NOT NULL, + computed_at INTEGER NOT NULL + ); + CREATE INDEX IF NOT EXISTS idx_emb_model ON session_embeddings(model); + `); +} + +function storeEmbeddings(rows) { + // rows: [{session_id, embedding (array), model}] + if (!rows || rows.length === 0) return; + const sq = _sqliteIndex(); + ensureEmbeddingTable(); + + const now = Date.now(); + const parts = ['BEGIN;']; + for (const r of rows) { + const sid = r.session_id.replace(/'/g, "''"); + const emb = JSON.stringify(r.embedding); + const model = (r.model || MODEL_ID).replace(/'/g, "''"); + parts.push(`INSERT OR REPLACE INTO session_embeddings (session_id, embedding, model, computed_at) VALUES ('${sid}', '${emb}', '${model}', ${now});`); + } + parts.push('COMMIT;'); + sq._exec(parts.join('\n'), { timeout: 60000 }); +} + +function loadAllEmbeddings() { + const sq = _sqliteIndex(); + ensureEmbeddingTable(); + const rows = sq._execJson(`SELECT session_id, embedding FROM session_embeddings WHERE model = '${MODEL_ID.replace(/'/g, "''")}'`); + return rows.map(r => ({ + session_id: r.session_id, + embedding: JSON.parse(r.embedding), + })); +} + +function getEmbeddingCount() { + const sq = _sqliteIndex(); + ensureEmbeddingTable(); + const rows = sq._execJson(`SELECT COUNT(*) AS n FROM session_embeddings`); + return (rows[0] || {}).n || 0; +} + +// ── Semantic search ───────────────────────────────────────── + +async function semanticSearch(query, limit) { + limit = limit || 20; + const queryEmb = await embed(query); + const all = loadAllEmbeddings(); + + if (all.length === 0) return []; + + // Compute similarities + const scored = all.map(r => ({ + session_id: r.session_id, + score: cosineSimilarity(queryEmb, r.embedding), + })); + + // Sort by descending similarity, return top-K + scored.sort((a, b) => b.score - a.score); + return scored.slice(0, limit); +} + +// ── Hybrid search: FTS5 recall → vector re-rank ───────────── + +async function hybridSearch(query, limit) { + limit = limit || 20; + const sq = _sqliteIndex(); + + // Phase 1: FTS5 text search for recall (broad set) + const ftsResults = sq.search(query, 200); + + if (!isAvailable() || ftsResults.length === 0) { + // No vector model — return FTS5 results as-is + return ftsResults.map(r => ({ ...r, search_type: 'text' })); + } + + // Phase 2: embed query + let queryEmb; + try { + queryEmb = await embed(query); + } catch { + return ftsResults.map(r => ({ ...r, search_type: 'text' })); + } + + // Phase 3: load embeddings for FTS5-matched sessions + const ftsSessionIds = [...new Set(ftsResults.map(r => r.session_id))]; + const all = loadAllEmbeddings(); + const embMap = new Map(all.map(r => [r.session_id, r.embedding])); + + // Phase 4: score and re-rank + const scored = ftsSessionIds.map(sid => { + const emb = embMap.get(sid); + const similarity = emb ? cosineSimilarity(queryEmb, emb) : 0; + const ftsSnippets = ftsResults.filter(r => r.session_id === sid); + return { + session_id: sid, + similarity, + fts_matches: ftsSnippets.length, + // Combined score: FTS match count + semantic similarity + combined_score: (ftsSnippets.length * 0.3) + (similarity * 0.7), + matches: ftsSnippets.slice(0, 3).map(r => ({ + role: r.role, + snippet: (r.snippet || '').replace(/<>/g, ''), + })), + search_type: emb ? 'hybrid' : 'text', + }; + }); + + scored.sort((a, b) => b.combined_score - a.combined_score); + return scored.slice(0, limit); +} + +module.exports = { + isAvailable, + embed, + embedBatch, + cosineSimilarity, + ensureEmbeddingTable, + storeEmbeddings, + loadAllEmbeddings, + getEmbeddingCount, + semanticSearch, + hybridSearch, + EMBEDDING_DIM, + MODEL_ID, +}; diff --git a/src/frontend/app.js b/src/frontend/app.js index 87eaf97..88cf73c 100644 --- a/src/frontend/app.js +++ b/src/frontend/app.js @@ -826,18 +826,33 @@ async function toggleExpand(sessionId, project, btn) { var deepSearchCache = {}; var deepSearchTimeout = null; +var searchMode = 'hybrid'; // text|semantic|hybrid + +function setSearchMode(mode) { + searchMode = mode; + document.querySelectorAll('.search-mode-btn').forEach(function(b) { + b.classList.toggle('active', b.dataset.mode === mode); + }); + // Re-run current search with new mode + deepSearchCache = {}; + if (searchQuery && searchQuery.length >= 3) { + deepSearch(searchQuery); + } +} async function deepSearch(query) { if (!query || query.length < 3) return; - if (deepSearchCache[query]) { - applyDeepSearchResults(deepSearchCache[query]); + var cacheKey = searchMode + ':' + query; + if (deepSearchCache[cacheKey]) { + applyDeepSearchResults(deepSearchCache[cacheKey]); return; } try { - var resp = await fetch('/api/search?q=' + encodeURIComponent(query)); + var url = '/api/search?q=' + encodeURIComponent(query) + '&mode=' + searchMode; + var resp = await fetch(url); var results = await resp.json(); - deepSearchCache[query] = results; + deepSearchCache[cacheKey] = results; applyDeepSearchResults(results); } catch {} } diff --git a/src/frontend/index.html b/src/frontend/index.html index c8b3fd9..a22c9e1 100644 --- a/src/frontend/index.html +++ b/src/frontend/index.html @@ -111,6 +111,11 @@
+
+ + + +