Skip to content

perf: optimize log list/export with 3-layer metadata loading#8

Merged
ntwrknrd merged 2 commits intomainfrom
perf/log-list-optimization
Jan 11, 2026
Merged

perf: optimize log list/export with 3-layer metadata loading#8
ntwrknrd merged 2 commits intomainfrom
perf/log-list-optimization

Conversation

@ntwrknrd
Copy link
Copy Markdown
Owner

Problem

nssh log list and nssh log export freeze for 30+ seconds when there are many recording files or large recordings.

Root Cause

ReadCastMetadata() was reading the entire cast file to find the final timestamp for duration calculation. With large recordings (multi-hour sessions), this meant reading hundreds of MB per file.

Solution

Implemented a 3-layer optimization strategy:

Layer 1: Index-first lookup

  • Check .index.json sidecar file first (already exists, ~1KB)
  • Contains StartedAt/FinishedAt timestamps - no file parsing needed

Layer 2: Tail-read fallback

  • For files without index, read last 64KB instead of entire file
  • Parse backwards to find final timestamp
  • Reduces O(file_size) → O(1)

Layer 3: Lazy loading with mtime pre-sort

  • New ListCastFilesWithMtime() collects paths + mtimes cheaply
  • Sort by mtime before loading any metadata
  • IterSessionRecordsLimit() only parses top N files when --last is used

Performance Impact

Scenario Before After
nssh log list --last 5 Loads all sessions Loads only 5
Files with .index.json Full file scan ~1KB JSON read
Files without index Full file scan 64KB tail read

Testing

  • make build
  • make test
  • go vet ./...

Fix 30+ second freezes when running 'nssh log list' or 'nssh log export'
with many or large recording files.

Layer 1: Index-first lookup
- ReadCastMetadata now checks .index.json sidecar first
- Index already contains StartedAt/FinishedAt - no file parsing needed

Layer 2: Tail-read fallback
- For files without index, read last 64KB instead of entire file
- Parses backwards to find final timestamp
- Reduces O(file_size) to O(1)

Layer 3: Lazy loading with mtime pre-sort
- New ListCastFilesWithMtime collects paths + mtimes cheaply
- Sort by mtime before loading any metadata
- IterSessionRecordsLimit only parses top N files when --last is used

Performance impact:
- 'nssh log list --last 5': loads 5 sessions instead of all
- Files with .index.json: instant metadata lookup
- Files without index: 64KB read vs full file scan
@ntwrknrd ntwrknrd merged commit ebd8878 into main Jan 11, 2026
13 checks passed
@ntwrknrd ntwrknrd deleted the perf/log-list-optimization branch January 11, 2026 05:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant