error: StreamTooLong when recompiling; duplicate source files in cache manifest

[rupurt]

Zig Version

0.11.0-dev.3771+128fd7dd0

Steps to Reproduce and Observed Behavior

https://github.com/rupurt/record-duckdb-extension

Steps to reproduce

nix develop -c $SHELL
nix run .#build-fast
nix run .#build-fast

Observed behavior

build.zig options:
- openssl-include-dir=/nix/store/4a0x3z29dhwid9w3wpc1vl05ykm5pqgs-openssl-3.0.9-dev/include
zig build-lib duckdb_static ReleaseFast native: error: error: StreamTooLong

zig build-lib duckdb_static ReleaseFast native: error: the following command exited with error code 1:
/nix/store/213vknkrkc5izl27z7xzs67rm8m6pvfx-zig-0.11.0-dev.3771+128fd7dd0/bin/zig build-lib @/home/alex/workspace/rupurt/record-duckdb-extension/zig-cache/args/12e63fc7cee3d195fbfe9ad73681e58c0e9a44720a0f0886f19d2df986096ddb
Build Summary: 35/38 steps succeeded; 1 failed (disable with --summary none)
install transitive failure
└─ install duckdb_static transitive failure
   └─ zig build-lib duckdb_static ReleaseFast native failure
error: the following build command failed with exit code 1:
/home/alex/workspace/rupurt/record-duckdb-extension/zig-cache/o/78b16d4b1e14050816deef8c0e0e9dee/build /nix/store/213vknkrkc5izl27z7xzs67rm8m6pvfx-zig-0.11.0-dev.3771+128fd7dd0/bin/zig /home/alex/workspace/rupurt/record-duckdb-extension /home/alex/workspace/rupurt/record-duckdb-extension/zig-cache /home/alex/.cache/zig -Dopenssl-include-dir=/nix/store/4a0x3z29dhwid9w3wpc1vl05ykm5pqgs-openssl-3.0.9-dev/include -Doptimize=ReleaseFast

Expected Behavior

I should be able to compile subsequent runs successfully

[matu3ba]

Can you reduce the problem further and provide an ideally minimal reproducible?
Also, please provide the zig command to reproduce the error to rule out nix being the source of the problem.

[rupurt]

Full command with same error:

zig build -Doptimize=ReleaseFast -Dopenssl-include-dir=/nix/store/gx7946h9y6qwcdil01ibf4hvm8vfc823-openssl-3.0.9-dev/include -Dbuild-shell

Unfortunately I don't know how to reduce the problem further. The example steps I gave in the repo produces the error every time. I'm running zig build ... twice.

I started looking through the zig build implementation and I feel like this could be a good starting point. It seems to hard code a max IO size limit of 10mb. I'm currently trying to figure out how to rebuild zig and use it in this codebase.

[emidoots]

I am encountering this frequently when trying to cross-compile mach-gpu-dawn for Windows, so I can at least confirm nix is not the only source of the problem.

Unfortunately I don't have an easy/minimal repro, I'll see if I can produce one later

[emidoots]

It is notable that in both my case and @rupurt's report:

• this only appears to happen on the second zig build, the first one is fine. If you blow away zig-cache and zig-out, the issue is worked around until the next build.
• both our failing commands are zig build-lib with enough args that a compiler response file zig build-lib @foo needs to be used.

[emidoots]

OK I've traced it down to the manifest file exceeding 50MB, here's a stack trace from a stage3 zig:

/Users/slimsag/Desktop/zig/build/stage3/bin/zig build-lib @/Users/slimsag/Desktop/hexops/mach/libs/gpu/zig-cache/args/6eb4a576449b3bb4f02df34922b46612d50202131eb23e0cbbf92134466cdd62 
zig build-lib dawn ReleaseFast x86_64-windows: error: error: StreamTooLong
/Users/slimsag/Desktop/zig/lib/std/io/reader.zig:85:21: 0x1025bed43 in readAllArrayListAligned__anon_42380 (zig)
                    return error.StreamTooLong;
                    ^
/Users/slimsag/Desktop/zig/lib/std/io/reader.zig:65:13: 0x1025bea03 in readAllArrayList (zig)
            return self.readAllArrayListAligned(null, array_list, max_append_size);
            ^
/Users/slimsag/Desktop/zig/lib/std/io/reader.zig:105:13: 0x1025bef33 in readAllAlloc (zig)
            try self.readAllArrayList(&array_list, max_size);
            ^
/Users/slimsag/Desktop/zig/lib/std/Build/Cache.zig:436:35: 0x1025c124b in hit (zig)
            const file_contents = try self.manifest_file.?.reader().readAllAlloc(gpa, manifest_file_size_max);
                                  ^
/Users/slimsag/Desktop/zig/src/Compilation.zig:1905:52: 0x10265cdcf in update (zig)
            const i = man.failed_file_index orelse return err;
                                                   ^
/Users/slimsag/Desktop/zig/src/main.zig:3420:21: 0x1026876ef in serve (zig)
                    try comp.update(main_progress_node);
                    ^
/Users/slimsag/Desktop/zig/src/main.zig:3234:13: 0x10251ecc3 in buildOutputType (zig)
            try serve(
            ^
/Users/slimsag/Desktop/zig/src/main.zig:271:9: 0x1024d0fdf in mainArgs (zig)
        return buildOutputType(gpa, arena, args, .{ .build = .Lib });
        ^
/Users/slimsag/Desktop/zig/src/main.zig:213:5: 0x1024ce7cf in main (zig)
    return mainArgs(gpa, arena, args);
    ^

zig build-lib dawn ReleaseFast x86_64-windows: error: the following command exited with error code 1:
/Users/slimsag/Desktop/zig/build/stage3/bin/zig build-lib @/Users/slimsag/Desktop/hexops/mach/libs/gpu/zig-cache/args/6eb4a576449b3bb4f02df34922b46612d50202131eb23e0cbbf92134466cdd62 

[emidoots]

I found the manifest file max is 50MB:

const manifest_file_size_max = 50 * 1024 * 1024;

And began inspecting the manifest files produced for my mach-gpu-dawn build, most are <150K, but two are not:

122M	./zig-cache/h/326381304e641965cd20afd8120b2761.txt
122M	./zig-cache/h/62e8fa63813e956aff18bd12659f7ffd.txt

I don't have a convenient place to upload files that large, but I could if it's helpful. I was able to introspect the contents of them a bit and I think I've found the cause? They are valid manifest files, they look like this:

9204 79469445 1687615676000000000 30ba36d2e76cc3f4eb662adecc256507 1 libc/include/any-windows-any/ole2.h
7852 79469433 1687615676000000000 ed27bfcd8ba39c420f3e66f0cf7f8b18 1 libc/include/any-windows-any/objbase.h
14373 79468632 1687615676000000000 e4cfc5527d471f9a0873d486fb40fdb6 1 libc/include/any-windows-any/combaseapi.h
9699 79470009 1687615676000000000 0ead0d629701e46c0acfac8339e10aa4 1 libc/include/any-windows-any/wtypesbase.h

There are 1.1 million lines:

cat ./zig-cache/h/326381304e641965cd20afd8120b2761.txt | wc -l
 1102133

~100k (18MB of the file) are from my project:

cat ./zig-cache/h/326381304e641965cd20afd8120b2761.txt | grep '/Users/slimsag/Desktop/hexops/mach' | wc -l
  102061

There appear to be millions of repeated, identical lines:

slimsag@hexops-ci gpu % cat ./zig-cache/h/326381304e641965cd20afd8120b2761.txt | gsort | wc -l
 1102133
slimsag@hexops-ci gpu % 
slimsag@hexops-ci gpu % cat ./zig-cache/h/326381304e641965cd20afd8120b2761.txt | gsort | uniq | wc -l
    4054

If I remove only the 100% duplicate lines, the manifest file is just 704K:

slimsag@hexops-ci gpu % cat ./zig-cache/h/326381304e641965cd20afd8120b2761.txt | gsort | uniq > tmpfo
slimsag@hexops-ci gpu % du -sh tmpfo 
704K	tmpfo

......soooo, why do manifest files have so many duplicates in them? any guesses @matu3ba ?

[emidoots]

The relevant manifest writing code is here:

zig/lib/std/Build/Cache.zig

Line 842 in 88284c1

for (self.files.items) |file| {

I suspect we need some deduplication logic here (or the files list itself should not have duplicates), not 100% sure but it might be a while before I have time to look into this further

[andrewrk]

There are 1.1 million lines:

Goodness!

Um, before adding deduplication logic I'd like to understand why there are duplicates in the first place. Maybe it will reveal another issue.

[emidoots]

update: I am still encountering this, both in my project from before and now in a new (also large C++) project.

Observations:

• Using 0.12.0-dev.1751+7fbbeae61
• It doesn't happen on every zig build. Just as before, the first build (when zig-cache and zig-out does not exist) is fine. Subsequent builds are sometimes fine, sometimes not.
• When StreamTooLong is observed, I always find a very large txt file in zig-cache:

 97M    zig-cache/h/649e59cf152d048f7b98c0d08ff65780.txt

Deleting just that one file works around the issue, until several more zig build are ran at which point it may or may not happen again.

Subsequent builds do not always produce a 97M txt file like that, hinting there may be some non-deterministic race condition which leads to the txt file expanding to that size? It does not appear to grow each zig build either.