perf: optimize log list/export with 3-layer metadata loading#8
Merged
perf: optimize log list/export with 3-layer metadata loading#8
Conversation
Fix 30+ second freezes when running 'nssh log list' or 'nssh log export' with many or large recording files. Layer 1: Index-first lookup - ReadCastMetadata now checks .index.json sidecar first - Index already contains StartedAt/FinishedAt - no file parsing needed Layer 2: Tail-read fallback - For files without index, read last 64KB instead of entire file - Parses backwards to find final timestamp - Reduces O(file_size) to O(1) Layer 3: Lazy loading with mtime pre-sort - New ListCastFilesWithMtime collects paths + mtimes cheaply - Sort by mtime before loading any metadata - IterSessionRecordsLimit only parses top N files when --last is used Performance impact: - 'nssh log list --last 5': loads 5 sessions instead of all - Files with .index.json: instant metadata lookup - Files without index: 64KB read vs full file scan
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
nssh log listandnssh log exportfreeze for 30+ seconds when there are many recording files or large recordings.Root Cause
ReadCastMetadata()was reading the entire cast file to find the final timestamp for duration calculation. With large recordings (multi-hour sessions), this meant reading hundreds of MB per file.Solution
Implemented a 3-layer optimization strategy:
Layer 1: Index-first lookup
.index.jsonsidecar file first (already exists, ~1KB)StartedAt/FinishedAttimestamps - no file parsing neededLayer 2: Tail-read fallback
Layer 3: Lazy loading with mtime pre-sort
ListCastFilesWithMtime()collects paths + mtimes cheaplyIterSessionRecordsLimit()only parses top N files when--lastis usedPerformance Impact
nssh log list --last 5Testing
make build✓make test✓go vet ./...✓