GNU tests `factor` regressions

[tertsdiepraam]

On a recent PR unrelated to factor, a lot of factor tests are failing

Error: GNU test failed: tests/factor/t01. tests/factor/t01 is passing on 'main'. Maybe you have to rebase?
Error: GNU test failed: tests/factor/t03. tests/factor/t03 is passing on 'main'. Maybe you have to rebase?
Error: GNU test failed: tests/factor/t04. tests/factor/t04 is passing on 'main'. Maybe you have to rebase?
Error: GNU test failed: tests/factor/t06. tests/factor/t06 is passing on 'main'. Maybe you have to rebase?
Error: GNU test failed: tests/factor/t08. tests/factor/t08 is passing on 'main'. Maybe you have to rebase?
Error: GNU test failed: tests/factor/t11. tests/factor/t11 is passing on 'main'. Maybe you have to rebase?
Error: GNU test failed: tests/factor/t15. tests/factor/t15 is passing on 'main'. Maybe you have to rebase?
Error: GNU test failed: tests/factor/t16. tests/factor/t16 is passing on 'main'. Maybe you have to rebase?
Error: GNU test failed: tests/factor/t20. tests/factor/t20 is passing on 'main'. Maybe you have to rebase?

Same happened here.

[sylvestre]

The question is also if they were working before or not ? :)
cc @nbraud

[tertsdiepraam]

They seem to pass on main: https://uutils.github.io/coreutils-docs/user/test_coverage.html Is there something else going on?

[rivy]

Maybe related to recent CI changes that I made to support "debug" builds for GnuTest coverage?

I'll try to take a closer look at these logs.

Intermittent CI bugs, ah joy... 😕

[rivy]

Yep, the factor tests are failing with...

2022-02-21T04:40:23.6941999Z FAIL: tests/factor/t11
2022-02-21T04:40:23.6942064Z ======================
2022-02-21T04:40:23.6942069Z 
2022-02-21T04:40:23.6942196Z sha1sum: failed to open file: No such file or directory
2022-02-21T04:40:23.6942319Z factor: warning: 18446744073708551700: Broken pipe (os error 32)
... literally 100000+ lines of "Broken pipe (os error 32)" ...

My recent changes caused some sort of breakage within the tests themselves.
It looks like they are all broken, some were just broken for other reasons prior...
I'm digging further into it.

[rivy]

So, after looking at this, I think we should just disable all the dynamically generated factor tests within GNU.
They are just testing for factor output over a given range of inputs. I don't think they add anything we can't or aren't already doing with in-house tests. We already test in-house (admittedly, for smaller ranges) with much better precision and robustness.

And we've already cut out half of the dynamic GNU tests for taking too long to test; and, now, in "debug" builds, the rest are taking even more annoyingly long amounts of time to complete. That has lead to partitioning some tests for "release" only, complicating the build script for the GNU tests. And hacking the test scripts directly with the build script is proving too fragile.

Disabling them should stabilize the test oscillations. 🤞🏻

I've got a commit set with the change running in my local fork which looks more stable.

It will cut the TOTAL number to 590 and PASS to 215 (to GNU tests summary = TOTAL: 590 / PASS: 215 / FAIL: 291 / ERROR: 10 from approximately GNU tests summary = TOTAL: 611 / PASS: 234 / FAIL: 293 / ERROR: 10).

We can judiciously move them back in at a later date if/when desired (although I'm not sure that we need them).

@sylvestre , @tertsdiepraam , thoughts?

[tertsdiepraam]

Do you know why they fail exactly? I'm fine with disabling them if the failures don't indicate any regressions, because that makes them pretty much useless anyway.

[rivy]

From what I see in the logs, the tests are failing due to broken pipes to a failed sha1sum (sha1sum: failed to open file: No such file or directory) not regressions of factor, although I'm not sure why the sha1sum is failing. I'm trying to fully duplicate on an Ubuntu VM, but it's not proceeding quickly.

I'm almost 100% certain that it's not a factor regression because the same tests all pass in the first non-debug iteration of GnuTests. The small remaining doubt is that maybe the "debug" build is introducing or uncovering some flaw that's otherwise hidden, but, again, it looks like a sha1sum problem not a factor problem.

[tertsdiepraam]

Interesting, I've been seeing more broken pipe errors in the CI lately, for example with those flaky ls tests recently. I wonder if that's related.

[jfinkels]

This seems to still be happening frequently. It's kind of annoying because it creates a lot of spurious build failure notifications. See https://github.com/uutils/coreutils/runs/5637336986?check_suite_focus=true for a recent example.

[sylvestre]

So, after looking at this, I think we should just disable all the

dynamically generated factor tests within GNU.

I've got a commit set with the change running in my local fork which looks more stable.

Agreed do you still have this patch ?:) thanks S

…

Message ID: ***@***.***>

[rivy]

Sure.
I'll add a PR this weekend.

[rivy]

I've got a PR almost done. I'm looking at two errors before I post it...

Error: GNU test failed: tests/df/df-symlink. tests/df/df-symlink is passing on 'master'. Maybe you have to rebase?
Error: GNU test failed: tests/ls/readdir-mountpoint-inode. tests/ls/readdir-mountpoint-inode is passing on 'master'. Maybe you have to rebase?
Error: Process completed with exit code 255.