Skip to content

Help? Working on Hydra release builds for master x64 Deb/RPM#1769

Closed
bhipple wants to merge 1 commit into
NixOS:masterfrom
bhipple:fixing-hydra-builds
Closed

Help? Working on Hydra release builds for master x64 Deb/RPM#1769
bhipple wants to merge 1 commit into
NixOS:masterfrom
bhipple:fixing-hydra-builds

Conversation

@bhipple
Copy link
Copy Markdown
Contributor

@bhipple bhipple commented Dec 30, 2017

Hydra has been broken since 4cde04f due to some strange behavior with the
sandboxing, and somewhere during that time a debug symbol problem also crept
into the RPM builds. Strangely, neither problem shows up on 32-bit builds.

This is hideous but it gets both Ubuntu and Fedora building again:
$ nix-build release.nix -A rpm_fedora25x86_64 -A deb_ubuntu1604x86_64

The first issue seems to be that /bin/bash doesn't get mounted in the chroot,
but I'm not sure why that is. I verified outside the sandbox call that it is
there and does exist. Not sure on the debug symbols issue. I'll keep digging
around but figured I'd show some hotfixes to demonstrate these are the only
problems and see if anyone else knows more?

@bhipple
Copy link
Copy Markdown
Contributor Author

bhipple commented Dec 30, 2017

Not really intending for this to be merged as-is (hoping someone has some insight into the VM tools/debug symbols/chroot issues and can enlighten me), though it would be nice to get the Hydra release build in the green again.

Hydra has been broken since 4cde04f due to some strange behavior with the
sandboxing, and somewhere during that time a debug symbol problem also crept
into the RPM builds. Strangely, neither problem shows up on 32-bit builds.

This is hideous but it gets both Ubuntu and Fedora building again:
  $ nix-build release.nix -A rpm_fedora25x86_64 -A deb_ubuntu1604x86_64
  <passes everything>

The first issue seems to be that /bin/bash doesn't get mounted in the chroot,
but I'm not sure why that is. I verified outside the sandbox call that it is
there and does exist. Not sure on the debug symbols issue. I'll keep digging
around but figured I'd show some hotfixes to demonstrate these are the only
problems and see if anyone else knows more?
@dtzWill
Copy link
Copy Markdown
Member

dtzWill commented Dec 31, 2017

Not sure what to do about the debuginfo problem, but your fix seems reasonable if it's a known issue with the tools shipping with released Fedora.

Searching around...

FWIW, here's a log snippet from the Fedora failure:

+ find /tmp/rpmout/BUILDROOT/nix-1.12pre5828_56591f6-2.fc25.x86_64 -name '*.la' -exec rm -f '{}' ';'
+ mkdir -p /tmp/rpmout/BUILDROOT/nix-1.12pre5828_56591f6-2.fc25.x86_64/nix/store
+ chmod 1775 /tmp/rpmout/BUILDROOT/nix-1.12pre5828_56591f6-2.fc25.x86_64/nix/store
+ for d in profiles gcroots
+ mkdir -p /tmp/rpmout/BUILDROOT/nix-1.12pre5828_56591f6-2.fc25.x86_64/nix/var/nix/profiles/per-user
+ chmod 1777 /tmp/rpmout/BUILDROOT/nix-1.12pre5828_56591f6-2.fc25.x86_64/nix/var/nix/profiles/per-user
+ for d in profiles gcroots
+ mkdir -p /tmp/rpmout/BUILDROOT/nix-1.12pre5828_56591f6-2.fc25.x86_64/nix/var/nix/gcroots/per-user
+ chmod 1777 /tmp/rpmout/BUILDROOT/nix-1.12pre5828_56591f6-2.fc25.x86_64/nix/var/nix/gcroots/per-user
+ chmod -x /tmp/rpmout/BUILDROOT/nix-1.12pre5828_56591f6-2.fc25.x86_64/etc/profile.d/nix.sh
+ rm -f /tmp/rpmout/BUILDROOT/nix-1.12pre5828_56591f6-2.fc25.x86_64/usr/share/doc/nix-doc-1.12pre5828_56591f6/README
+ rm -rf /tmp/rpmout/BUILDROOT/nix-1.12pre5828_56591f6-2.fc25.x86_64/etc/init
+ /usr/lib/rpm/find-debuginfo.sh --strict-build-id -m --run-dwz --dwz-low-mem-die-limit 10000000 --dwz-max-die-limit 110000000 /tmp/rpmout/BUILD/nix-1.12pre5828_56591f6
extracting debug info from /tmp/rpmout/BUILDROOT/nix-1.12pre5828_56591f6-2.fc25.x86_64/usr/libexec/nix/build-remote
extracting debug info from /tmp/rpmout/BUILDROOT/nix-1.12pre5828_56591f6-2.fc25.x86_64/usr/libexec/nix/buildenv
extracting debug info from /tmp/rpmout/BUILDROOT/nix-1.12pre5828_56591f6-2.fc25.x86_64/usr/lib64/libnixmain.so
extracting debug info from /tmp/rpmout/BUILDROOT/nix-1.12pre5828_56591f6-2.fc25.x86_64/usr/lib64/libnixexpr.so
extracting debug info from /tmp/rpmout/BUILDROOT/nix-1.12pre5828_56591f6-2.fc25.x86_64/usr/lib64/libnixstore.so
/usr/lib/rpm/find-debuginfo.sh: line 127:  3683 Bus error               eu-strip --remove-comment $r $g -f "$1" "$2"
error: Bad exit status from /var/tmp/rpm-tmp.r9d216 (%install)


RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.r9d216 (%install)
[  338.625208] reboot: Power down

@bhipple
Copy link
Copy Markdown
Contributor Author

bhipple commented Jan 3, 2018

I can confirm that the fix in #1770 ALSO fixes the RPM sandbox test cases! Good catch on /lib64 @dtzWill!

That just leaves the debug symbols for the RPMs. That ticket hints that eu-strip throws that error when it runs out of memory while separating debug symbols, and sure enough, if we increase our VM memory from 1028mb to 2048mb we get working RPM builds again for Fedora! Awesome! I opened #1773 which has this fix, and assuming that the extra VM memory isn't a problem for Hydra it should get our builds back in the green.

@bhipple bhipple closed this Jan 3, 2018
@dtzWill
Copy link
Copy Markdown
Member

dtzWill commented Jan 3, 2018

Oh, hooray! Good call on VM memory, looking forward to seeing some green builds! 😁

@vcunat
Copy link
Copy Markdown
Member

vcunat commented Jan 3, 2018

bus error? I suppose they use mmap...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants