Add Process.ReadAllLines synchronous API

[Copilot]

Description

Implements Process.ReadAllLines(TimeSpan? timeout = default) — a synchronous, single-threaded multiplexing API that reads interleaved lines from both stdout and stderr without deadlocks.

API approved in #125838 (comment).

using Process process = Process.Start(startInfo);
foreach (ProcessOutputLine line in process.ReadAllLines())
{
    if (line.StandardError)
        Console.ForegroundColor = ConsoleColor.Red;
    Console.WriteLine(line.Content);
    Console.ResetColor();
}

Cross-platform core (Process.Multiplexing.cs)

• ReadAllLines() validates state eagerly via ValidateReadAllState, passes Encoding instances per stream to the platform-specific ReadPipesToLines iterator (which creates Decoder instances locally so they can be replaced on encoding detection)
• Shared char-level helpers: DecodeAndAppendChars (incremental byte→char decoding via Decoder), ParseLinesFromCharBuffer (scans for '\r' or '\n' via IndexOfAny, handles '\r\n' as a single terminator), EmitRemainingCharsAsLine (EOF partial line)
• DecodeAndAppendChars accepts ref int charStartIndex and always compacts the char buffer (moving unconsumed data to the front) before considering growth when charStartIndex > 0, ensuring RentLargerBuffer only copies the unconsumed tail rather than the entire buffer including the consumed prefix
• DecodeAndAppendChars has no early return guard — the while (charEndIndex + charCount > charBuffer.Length) buffer growth loop is trivially skipped when charCount == 0, and GetChars is still called so partial bytes buffered internally by the Decoder are preserved
• DecodeBytesAndParseLines accumulates the initial bytes of each stream into a small 4-byte bomAccum buffer before calling SkipPreambleOrDetectEncoding, ensuring that a UTF-32 LE BOM (FF FE 00 00) is never misclassified as a UTF-16 LE BOM (FF FE) when the first OS read delivers fewer than 4 bytes — a new FlushBomAccumulation helper resolves any accumulated bytes at EOF
• SkipPreambleOrDetectEncoding operates at the byte level and is always called with at least 4 accumulated bytes (or all available bytes at EOF), mimicking StreamReader's two-phase approach: first checks whether the bytes match the current encoding's preamble (like StreamReader.IsPreamble), then falls back to BOM-based encoding detection (like StreamReader.DetectEncoding) supporting UTF-8, UTF-16 LE/BE, and UTF-32 LE/BE — if a different encoding is detected, the Decoder is replaced accordingly
• Line parsing operates on char[] buffers using Encoding.GetDecoder() per stream, correctly handling multi-byte encodings (UTF-16, UTF-32) and partial sequences across reads — byte-level 0x0A scanning is not used
• RentLargerBuffer<T> is a single generic helper used for both byte[] and char[] buffer growth (no duplication)
• CompactOrGrowCharBuffer was removed — DecodeAndAppendChars already handles compacting and growing lazily on the next decode call

Windows (Process.Multiplexing.Windows.cs)

• ReadPipesToLines iterator using overlapped I/O with WaitAny multiplexing
• Stores NativeOverlapped* as nint to avoid pointer fields in iterator state machine; casts back inside scoped unsafe blocks (C# 13 allows unsafe in iterators, but yield return must be in safe context)
• Error event at index 0 in wait handle array ensures stderr is processed first when both streams have data
• No DangerousAddRef/DangerousRelease needed — all Windows syscalls accept SafeHandle directly

Unix (Process.Multiplexing.Unix.cs)

• ReadPipesToLines iterator using poll(2) with non-blocking reads
• Uses a single shared byte buffer for both stdout and stderr reads (only one read is performed at a time), while maintaining separate char buffers per stream for incremental decoding state
• PollForPipeActivity helper accepts Span<Interop.PollEvent> and encapsulates PreparePollFds + timeout check + inline poll call (single triggered variable) + error/EINTR handling, shared by both ReadPipesToLines and ReadPipes to eliminate code duplication
• ReadPipes (non-iterator) uses stackalloc for the poll event buffer to avoid heap allocation, while the iterator-based ReadPipesToLines passes a heap-allocated array (implicitly converted to Span)
• HandlePipeLineRead helper encapsulates the read → decode → preamble/encoding check → parse lines flow
• The finally block restores each file descriptor to blocking mode (DangerousSetIsNonBlocking(fd, 0)) before calling DangerousRelease(), ensuring the handles are cleaned up in the correct state

Tests

• Extended existing streaming tests to validate both sync and async line-reading paths
• ReadAllLines_WorksWithNonDefaultEncodings theory tests UTF-8, UTF-16, and UTF-32 encodings for both sync and async paths, asserting identical output — test logic is inlined into the test method using CreateProcess with static lambdas (no separate RemotelyInvokable helper methods)
• ReadAllLines_WorksWithMultiByteCharacters theory tests multi-byte character handling with UTF-8, UTF-16, and UTF-32, writing raw encoded bytes split at the byte-array midpoint (which lands inside a multi-byte character for all three encodings) with an explicit Flush() between the two halves — this deterministically exercises decoder state across reads and would fail if partial-sequence handling were broken
• ReadAllLines_HandlesMixedLineEndings theory tests all three line-terminator styles (\r\n, bare \n, bare \r) and an unterminated final chunk in the same stream, asserting that ParseLinesFromCharBuffer and EOF flushing produce the correct four lines
• ReadAllLines_HandlesPartialBomAcrossReads theory tests that a UTF-32 LE BOM split across two separately-flushed writes (FF FE then 00 00 + content) is correctly accumulated and detected rather than misclassified as a UTF-16 LE BOM — exercises the bomAccum deferred-detection path
• ReadAllLines_ReadsVeryLongLines theory tests buffer growth by producing lines over 8192 characters on both stdout and stderr
• EnumerateLines test helper populates List<string> capturedOutput, List<string> capturedError directly
• Added using System.IO; to ProcessStreamingTests.cs to resolve a build error caused by StreamWriter and Stream references in the inlined test helpers

Build fix

• Updated ToTimeoutMilliseconds call to ProcessUtils.ToTimeoutMilliseconds to match upstream refactor on main that moved this method from Process to ProcessUtils

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-diagnostics-process
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

Adds a new synchronous Process.ReadAllLines(TimeSpan? timeout = default) API that multiplexes stdout/stderr and yields interleaved ProcessOutputLine instances without deadlocks, plus updates tests and the ref assembly.

Changes:

• Introduces Process.ReadAllLines(...) and shared buffer-to-line parsing helpers.
• Implements platform-specific synchronous multiplexing iterators for Windows (overlapped I/O + WaitAny) and Unix (poll(2) + non-blocking reads).
• Extends existing streaming tests to validate both sync and async line-reading paths.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/libraries/System.Diagnostics.Process/tests/ProcessStreamingTests.cs	Expands ReadAllLinesAsync test coverage to also exercise the new sync API.
src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Multiplexing.cs	Adds ReadAllLines(...) entrypoint and shared helpers for parsing/compacting line buffers.
src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Multiplexing.Windows.cs	Adds Windows-specific synchronous line multiplexing iterator using overlapped I/O.
src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Multiplexing.Unix.cs	Adds Unix-specific synchronous line multiplexing iterator using poll and non-blocking reads.
src/libraries/System.Diagnostics.Process/ref/System.Diagnostics.Process.cs	Adds the ReadAllLines(...) signature to the reference assembly.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

src/libraries/System.Diagnostics.Process/tests/ProcessStreamingTests.cs:213

• The new sync path adds incremental decoding/line-splitting logic that’s meant to support multi-byte encodings (UTF-16/UTF-32, partial sequences across reads, BOM). The updated tests don’t currently exercise StandardOutputEncoding/StandardErrorEncoding variations; adding a theory that runs both sync and async with non-default encodings would help prevent regressions in the decoder/line parsing behavior.

        public async Task ReadAllLines_ReadsBothOutputAndError(string standardOutput, string standardError, bool useAsync)
        {
            using Process process = StartLinePrintingProcess(
                string.IsNullOrEmpty(standardOutput) ? DontPrintAnything : standardOutput,
                string.IsNullOrEmpty(standardError) ? DontPrintAnything : standardError);

[Copilot]

@copilot please invoke the code-review skill and post your analysis/results as a comment on this PR

🤖 Copilot Code Review — PR #127106

Note

This review was AI-generated using the code-review skill with multi-model analysis (Claude Sonnet 4, GPT-5.3-Codex, Gemini 2.5 Pro).

Holistic Assessment

Motivation: The PR implements the synchronous counterpart (ReadAllLines) to the already-shipped ReadAllLinesAsync. The API is approved in issue #125838 (api-approved label). The motivation is strong: deadlock-free synchronous reading of interleaved stdout/stderr from a single thread.

Approach: Platform-specific multiplexing (overlapped I/O + WaitAny on Windows, poll(2) on Unix) with shared char-level line parsing and byte-level BOM/encoding detection mimicking StreamReader. Follows established patterns from existing ReadPipes/ReadAllText/ReadAllBytes in the same files.

Summary: ⚠️ Needs Human Review. The code is well-structured, follows codebase patterns, and handles resource cleanup correctly on both platforms. One low-probability edge case in encoding detection warrants maintainer attention.

---

Detailed Findings

✅ API Approval — Matches approved shape exactly

The ref assembly signature public IEnumerable<ProcessOutputLine> ReadAllLines(TimeSpan? timeout = default) matches the approved API shape from the API review comment exactly.

✅ Resource Management — Correct on both platforms

• Windows: outputDone/errorDone flags initialized to true before try; finally calls CancelPendingIOIfNeeded (which does CancelIoEx + GetOverlappedResult(bWait: true)) before freeing NativeOverlapped via NativeMemory.Free. Events, pins, and pooled buffers all cleaned up. Safe even on early break.
• Unix: DangerousAddRef/DangerousRelease properly guarded with bool flags. All ArrayPool buffers returned in finally. Single shared byte buffer correctly used since only one read occurs at a time.

✅ Encoding & Line Parsing — Correct char-level operation

• DecodeAndAppendChars has no early return, so the growth loop is trivially skipped when charCount == 0, and GetChars is always called — preserving partial byte sequences in the decoder's internal state. Buffer compaction happens before growth, ensuring RentLargerBuffer only copies the unconsumed tail.
• ParseLinesFromCharBuffer correctly handles \r, \n, and \r\n. When \r is at the very end of available data, it correctly breaks out of the loop and waits for more data before deciding whether it's a standalone \r or part of \r\n.
• EmitRemainingCharsAsLine strips a trailing \r at EOF, matching StreamReader.ReadLine behavior.

✅ SkipPreambleOrDetectEncoding — Mimics StreamReader correctly

The method implements the two-phase approach from StreamReader: first checks for the current encoding's preamble (like IsPreamble), then falls back to BOM-based detection (like DetectEncoding) covering UTF-8, UTF-16 LE/BE, and UTF-32 LE/BE. When a different encoding is detected, both encoding and decoder are replaced. The byte patterns and detection order match StreamReader.DetectEncoding exactly.

⚠️ One-shot preamble check — Low probability edge case (advisory, not blocking)

SkipPreambleOrDetectEncoding is called exactly once on the first read (controlled by preambleChecked flag). StreamReader.IsPreamble supports progressive checking across multiple reads (if the first read returns fewer bytes than the preamble length). For process pipes with a 4096-byte read buffer, the first read will virtually always contain at least 4 bytes (the maximum BOM length), so a split BOM is extremely unlikely. However, this is a behavioral divergence from StreamReader. This is advisory — I don't think it blocks merge given the practical impossibility of a <4 byte first pipe read, but a human reviewer should be aware.

✅ Windows Overlapped IO — Correct lifecycle management

• NativeOverlapped* stored as nint to avoid pointer fields in the iterator state machine — correct workaround for C# iterator limitations.
• CancelPendingIOIfNeeded correctly calls CancelIoEx followed by GetOverlappedResult(bWait: true) to ensure the IO completes before freeing the OVERLAPPED, per MSDN requirements.
• Error event at index 0 in WaitAny array ensures stderr is serviced first when both signals arrive simultaneously.

✅ Unix poll — Correct non-blocking multiplexing

• PollForPipeActivity correctly encapsulates PreparePollFds + timeout check + poll(2) + EINTR handling. Accepts Span<PollEvent> so ReadPipes can use stackalloc while the iterator uses a heap-allocated array.
• HandlePipeLineRead properly handles bytesRead > 0 (decode+parse), bytesRead == 0 (EOF flush), and bytesRead < 0 (EAGAIN, retry).

✅ Test Coverage — Comprehensive

Tests cover: basic output/error, interleaved output, large output (1000 lines), very long lines (8192 chars, exercising buffer growth), timeout/cancellation, early break, disposed process, encoding variations (UTF-8, UTF-16, UTF-32), multi-byte characters (CJK), and all tests run in both sync and async modes via [Theory] + useAsync. Test helpers are inlined as static lambdas via CreateProcess, improving readability.

💡 Suggestion — Mixed line ending test

No test exercises mixed line endings (\r\n + bare \n + bare \r in the same stream). The ParseLinesFromCharBuffer logic handles these correctly by inspection, but a targeted test would add confidence. This is a follow-up, not a blocker.

---

Models contributing: Claude Sonnet 4 (primary), GPT-5.3-Codex, Gemini 2.5 Pro. All three independently verified resource management, encoding correctness, and line parsing logic.

[Copilot]

@copilot fix the build, run the tests before pushing changes

...

Fixed in 8936fe6. Added using System.IO; to ProcessStreamingTests.cs — StreamWriter is in System.IO which was missing from the using directives.

[adamsitnik]

💡 Suggestion — Mixed line ending test

No test exercises mixed line endings (\r\n + bare \n + bare \r in the same stream). The ParseLinesFromCharBuffer logic handles these correctly by inspection, but a targeted test would add confidence. This is a follow-up, not a blocker.

@copilot address this feedback

Despite the comment about exercising decoder state across reads, this test writes a short line via StreamWriter.WriteLine, which is unlikely to deterministically split multi-byte sequences across read boundaries. That means it may not actually validate the incremental-decoding behavior it intends to cover. Consider forcing boundary splits (e.g., write encoded bytes directly in multiple writes/flushes that intentionally cut through a multi-byte sequence, or generate enough output to require multiple reads) so the test reliably fails if decoder state handling is broken.

@copilot address this feedback

@copilot make sure you build the process solution and run all tests before pushing any changes

[Copilot]

💡 Suggestion — Mixed line ending test

No test exercises mixed line endings (\r\n + bare \n + bare \r in the same stream). The ParseLinesFromCharBuffer logic handles these correc...

Both items addressed in d19dd3e:

1. Mixed line endings: Added ReadAllLines_HandlesMixedLineEndings theory that writes "lineA\r\nlineB\nlineC\rlineD" as raw UTF-8 bytes to stdout, covering all three terminator styles (\r\n, \n, bare \r) and an unterminated final chunk (emitted at EOF), asserting ["lineA", "lineB", "lineC", "lineD"].

2. Multi-byte decoder state: ReadAllLines_WorksWithMultiByteCharacters now writes raw encoded bytes split at encoded.Length / 2 with an explicit Flush() between the two halves. For all three encodings the midpoint lands inside a multi-byte character — e.g. for UTF-8 "hello_世界_stdout\n" (20 bytes) splits at byte 10, which is inside the second CJK character (E7|95 8C) — so the test reliably exercises decoder state between reads.

[tarekgh]

Added a couple of comments, LGTM otherwise.

[tmds]

@adamsitnik @tarekgh, @copilot struggles to understand the BOM/preamble handling which causes extra review cycles.

[adamsitnik]

@adamsitnik @tarekgh, @copilot struggles to understand the BOM/preamble handling which causes extra review cycles.

I agree, I will fix it myself soon

[adamsitnik]

@copilot please invoke the code-review skill and post your analysis/results as a comment on this PR

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

[adamsitnik]

@copilot please invoke the code-review skill and post your analysis/results as a comment on this PR

[github-actions]

🤖 Copilot Code Review — PR #127106

Note

This review was generated by Copilot using multi-model analysis (Claude Opus 4.6 primary, with GPT-5.3-Codex and Claude Opus 4.5 sub-agents).

Holistic Assessment

Motivation: The PR is well-justified — it implements the synchronous ReadAllLines API that was approved in #125838. This fills the gap between the existing ReadAllText/ReadAllBytes sync APIs and the ReadAllLinesAsync async API, giving users a simple foreach experience over interleaved stdout/stderr lines.

Approach: The implementation mirrors the platform-specific multiplexing strategy already used by ReadAllBytes/ReadAllText (Unix poll + non-blocking IO, Windows overlapped IO + WaitAny). Reusing the same infrastructure is the right call. Shared helpers for BOM detection, char decoding, and line parsing are well-factored and avoid duplication across platforms. The iterator pattern (yield return) with proper finally cleanup is the correct choice for an IEnumerable API that manages pooled resources.

Summary: ✅ LGTM. The implementation is thorough, correctly handles resource cleanup in iterators, properly detects BOM/encoding across partial reads, handles all line ending styles (\r, \n, \r\n), and the test coverage is comprehensive. The API shape matches the approved proposal exactly. No blocking issues found. A few minor suggestions below.

---

Detailed Findings

✅ API Approval — Matches approved shape

The ref/ change adds exactly public IEnumerable<ProcessOutputLine> ReadAllLines(TimeSpan? timeout = default) which matches the approved API shape from the api-approved issue #125838. No unapproved public surface added.

✅ Resource Safety — Correct cleanup in both iterator implementations

Both the Unix (Process.Multiplexing.Unix.cs:107-123) and Windows (Process.Multiplexing.Windows.cs:180-206) ReadPipesToLines iterators use try/finally blocks that properly clean up all resources when enumeration completes or is abandoned early:

• Unix: Releases DangerousAddRef refs, returns all 4 ArrayPool buffers
• Windows: Cancels pending IO via CancelPendingIOIfNeeded (which waits for completion with GetOverlappedResult(bWait: true)), frees NativeOverlapped* memory, disposes EventWaitHandles and PinnedGCHandles, returns all 4 ArrayPool buffers

This was confirmed by both sub-agent reviews (Claude Opus 4.5 and GPT-5.3-Codex).

✅ BOM/Preamble Detection — Correctly handles partial reads

The accumulation logic (unconsumedBytesCount) correctly waits until at least MaxEncodingBytesLength (4) bytes are available before calling SkipPreambleOrDetectEncoding, preventing a partial UTF-32 LE BOM (FF FE) from being misclassified as UTF-16 LE. The FlushDecoderAndEmitRemainingChars method handles the EOF case where fewer than 4 bytes were ever received. The ReadAllLines_HandlesPartialBomAcrossReads and ReadAllLines_LessThanFourBytes tests verify these edge cases.

✅ Line Parsing — Correct \r/\n/\r\n handling

ParseLinesFromCharBuffer correctly handles all three line ending styles and defers bare \r at buffer end until more data arrives (or EOF). EmitRemainingCharsAsLine strips trailing \r at EOF, matching StreamReader.ReadLine behavior. The ReadAllLines_HandlesMixedLineEndings test covers this comprehensively.

💡 Eager vs. Lazy Validation — Minor behavioral difference (not blocking)

ReadAllLines() validates eagerly (before enumeration starts), while ReadAllLinesAsync() validates lazily (on first MoveNextAsync). This is actually the recommended .NET pattern — sync iterator methods should validate eagerly to avoid deferred exceptions. The side effect of setting stream modes before enumeration is consistent with ReadAllText()/ReadAllBytes() which also call ValidateReadAllState() eagerly. No action needed.

💡 DecodeAndAppendChars double-scans bytes — Acceptable trade-off

DecodeAndAppendChars calls GetCharCount then GetChars, scanning the byte buffer twice. The alternative (Decoder.Convert loop) would be more complex for negligible gain — process pipe IO is the bottleneck, not in-memory decoding. Not a hot path concern.

💡 Test Coverage — Consider adding in follow-up

A few edge cases that could be tested in a follow-up:

• Asserting the documented stderr-first priority when both streams have data ready
• Process that writes only a bare \r to stdout

These are minor and do not block merge.

Generated by Code Review for issue #127106 · ◷

[adamsitnik]

LGTM, thank you for all the code reviews @tmds @tarekgh and @JeremyKuhne !