Skip to content

Stream NDJSON parsing to reduce memory pressure for large test suites #30

@tsavo-at-pieces

Description

@tsavo-at-pieces

Context

TestResultsUtil.parseTestResultsJson() in lib/src/cli/utils/test_results_util.dart uses file.readAsLinesSync() which loads the entire NDJSON file into memory as a List<String>. Combined with jsonDecode creating a Map per line, memory usage spikes for large test suites.

Current mitigations (already in place)

  • Failure list capped at 50 entries (line 92)
  • Per-field truncation: error (8KB), stack trace (6KB), print output (6KB)
  • Malformed JSON circuit breaker (5 warnings then suppressed)
  • stdout/stderr capture bounded at 2 MiB

Improvement

Replace readAsLinesSync() with streaming line-by-line:

final stream = file.openRead()
    .transform(utf8.decoder)
    .transform(const LineSplitter());
await for (final line in stream) {
  // parse line
}

This requires making parseTestResultsJson async — a small refactor since callers are already async.

Impact

Only affects repos with 10,000+ tests generating very large NDJSON files. Current mitigations make this low priority.

Identified during PR #29 review.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions