Skip to content

feat: unwrap search results for human-readable table output#250

Merged
tomaz-lc merged 1 commit intocli-v2from
feat-search-unwrap-results
Mar 13, 2026
Merged

feat: unwrap search results for human-readable table output#250
tomaz-lc merged 1 commit intocli-v2from
feat-search-unwrap-results

Conversation

@tomaz-lc
Copy link
Copy Markdown
Contributor

@tomaz-lc tomaz-lc commented Mar 13, 2026

Details

Improves the search run and search saved-run commands by unwrapping raw API SearchResult wrapper objects into a clean, human-readable table of events. Previously, the table output showed raw API metadata (searchResultId, nextToken, type, stats) with actual event data collapsed into [1546 items], making it unusable for interactive use.

Now in table mode, events from all pages are flattened into a single table with smart column selection (priority columns first, noisy routing metadata dropped, capped at 15 columns). Machine-readable formats (json, yaml, csv, jsonl) pass through the raw API response unchanged.

stderr/stdout separation

All progress messages, event counts, and stats summaries are printed to stderr, keeping stdout clean for data. This means limacharlie search run ... > results.txt captures only the table/data, while progress and stats appear in the terminal. This applies to:

  • Progress messages during search execution (query_id, page fetching, waiting for results)
  • Event count footer (e.g., (1,546 events))
  • Stats summary (matched, scanned, bytes, time, cost)

New flags

  • --raw: Show raw API result objects without unwrapping (restores old behavior)
  • --expand: Show each event as a full pretty-printed JSON block with a --- timestamp | stream | event_type --- header, useful for investigating individual events in detail

Ctrl+C handling

Pressing Ctrl+C during a search automatically sends a DELETE request to cancel the query on the server, freeing resources. A "Canceling search query..." message is shown when in interactive mode.

Before (raw API wrapper objects)
searchResultId    created                      type      stats                   nextToken    rows
12345             2026-03-13T16:43:08.467      events    {7 keys}               token-2      [1546 items]
12346             2026-03-13T16:43:08.467      facets    {7 keys}                            [4 items]
12347             2026-03-13T16:43:08.467      timeline  {7 keys}                            [24 items]
After (unwrapped event table)
time                 stream    routing.event_type    routing.hostname    event.FILE_PATH       event.COMMAND_LINE
2026-03-13 15:43:08  event     NEW_PROCESS           web-01              /usr/bin/curl          curl https://example.com
2026-03-13 15:43:09  event     DNS_REQUEST           web-01              -                      -
2026-03-13 15:43:10  event     NEW_PROCESS           db-02               /usr/bin/psql          psql -h localhost
...
(1,546 events)
Stats: matched: 1,546, scanned: 50,000, bytes: 2.1 GB, time: 4.2s, cost: $0.0050
After with --expand (full JSON per event)
--- 2026-03-13 15:43:08 | event | NEW_PROCESS ---
{
  "routing": {
    "event_type": "NEW_PROCESS",
    "hostname": "web-01",
    ...
  },
  "event": {
    "FILE_PATH": "/usr/bin/curl",
    "COMMAND_LINE": "curl https://example.com"
  }
}
--- 2026-03-13 15:43:09 | event | DNS_REQUEST ---
{
  ...
}

Changes

limacharlie/commands/search.py

  • _flatten_event_row(): Flattens SearchResultRow into single-level dict with human-readable timestamps
  • _unwrap_search_results(): Separates results by type (events/facets/timeseries), only takes stats from "events" type
  • _format_stats_summary(): Formats stats line with cumulative stats preferred, human-readable byte sizes
  • _limit_event_columns(): Smart column selection - priority columns first, drop noisy routing metadata, cap at 15
  • _format_expanded_events(): Formats each event as a JSON block with timestamp/stream/type header
  • _make_progress_fn(): Returns stderr callback for interactive terminals (TTY + not quiet)
  • _output_search_results(): Routes to table/expand/raw format, prints event count and stats to stderr
  • run and saved-run commands: Added --raw, --expand flags, progress_fn, KeyboardInterrupt handling

limacharlie/sdk/search.py

  • execute(): Added progress_fn callback, tracks pages/events/elapsed time
  • _cancel_query(): Sends DELETE on all exit paths (normal, error, Ctrl+C)
  • KeyboardInterrupt handled separately from Exception (not wrapped in SearchError)

tests/unit/test_search_output.py

  • 91 tests across 9 test classes covering all new functionality

Blast radius / isolation

  • Only affects search run and search saved-run CLI commands
  • SDK Search.execute() gains progress_fn parameter (backward compatible, defaults to None)
  • No changes to other commands, API contracts, or data formats
  • Machine-readable output formats completely unchanged

Performance characteristics

  • No additional API calls (same pagination, same polling)
  • Column limiting is O(rows * columns), negligible for typical result sets
  • Event flattening is a single pass over rows

Notable contracts / APIs

  • Search.execute() now accepts progress_fn: Callable[[str], None] | None - callback for progress messages
  • Search._cancel_query() - always called in finally block to clean up server resources
  • No wire format or API contract changes

@tomaz-lc tomaz-lc force-pushed the feat-search-unwrap-results branch from ec890e1 to babf6aa Compare March 13, 2026 17:42
@tomaz-lc tomaz-lc added the enhancement New feature or request label Mar 13, 2026
@tomaz-lc tomaz-lc marked this pull request as draft March 13, 2026 17:44
@tomaz-lc tomaz-lc requested a review from maximelb March 13, 2026 17:47
Search run/saved-run commands now display flattened event rows instead
of raw SearchResult wrapper objects in table mode.  Events from all
pages are merged into a single table with smart column selection.
Machine-readable formats (json, yaml, csv, jsonl) pass through the raw
API response unchanged.

Changes:
- Unwrap events: flatten mtd + nested data using dot notation
- Smart column limiting: max 15 columns, priority fields first, noisy
  routing metadata dropped.  --wide bypasses the limit.
- Progress feedback to stderr in interactive mode: query_id on start,
  page/events/elapsed during pagination, waiting status during polls
- Cumulative stats summary using server-aggregated cumulativeStats
- Stats from events results only (not facets/timeline)
- Clean Ctrl+C: sends DELETE to cancel query on server, prints status
- Handle null rows/facets/timeseries in API response (JSON null)
- Flush stderr for immediate progress display
- --raw flag to get old behavior in table mode
- 79 tests covering all new functionality

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
maximelb
maximelb previously approved these changes Mar 13, 2026
@tomaz-lc tomaz-lc force-pushed the feat-search-unwrap-results branch from babf6aa to c22f04e Compare March 13, 2026 17:51
@tomaz-lc tomaz-lc marked this pull request as ready for review March 13, 2026 19:29
@tomaz-lc tomaz-lc merged commit dc25653 into cli-v2 Mar 13, 2026
1 check passed
@tomaz-lc tomaz-lc deleted the feat-search-unwrap-results branch March 13, 2026 19:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants