GNU tests/tail-2/F-headers is failing often in the GNU CI

[sylvestre]

Error: GNU test failed: tests/tail-2/F-headers. tests/tail-2/F-headers is passing on 'main'. Maybe you have to rebase?

The error is:

2022-08-01T07:32:10.8281270Z FAIL: tests/tail-2/F-headers
2022-08-01T07:32:10.8281507Z ============================
2022-08-01T07:32:10.8281643Z 
2022-08-01T07:32:10.8281881Z tail: cannot open 'b' for reading: No such file or directory
2022-08-01T07:32:10.8282294Z ./tests/tail-2/F-headers.sh: a: unexpected delay?
2022-08-01T07:32:10.8282700Z tail: cannot open 'a' for reading: No such file or directory
2022-08-01T07:32:10.8283128Z tail: cannot open 'b' for reading: No such file or directory
2022-08-01T07:32:10.8283395Z ==> b <==
2022-08-01T07:32:10.8283713Z FAIL tests/tail-2/F-headers.sh (exit status: 1)
2022-08-01T07:32:10.8283900Z 

@niyaznigmatullin tried to fix it in #3737

[niyaznigmatullin]

I think probably it's because inotify doesn't send an event sometimes. Probably, because virtual disk, or many tests running in parallel, I don't know how inotify works inside.

I think that's because, when I increased timeout from 12 seconds to 51 seconds it was still failing, so seems that event doesn't come at all.

Plus, in my fork I tried to run GNU tests workflow around 10 times (from main branch, with 12 seconds timeout), and it didn't fail even once. That's strange

[niyaznigmatullin]

I also tried to somehow make notify crate fail to deliver the events, but it did well when creating 15000 files.

https://github.com/niyaznigmatullin/notify-stress-test/actions

Found that one in notify-rs issues saying that on some systems it doesn't deliver all the messages.

[niyaznigmatullin]

I propose to try to increase timeout and see if it will still fail. PR #3737

[jhscheer]

I propose to try to increase timeout and see if it will still fail. PR #3737

The timeout is most probably not the problem. In my tests, the inotify event either triggered immediately or never.

I already spent a lot of time trying to fix this test for the CI. At this point the only thing I can think of is sprinkling some more sleep calls, e.g.: #3776

[niyaznigmatullin]

Yes, it didn't help. As the increase before..

When you said about sleep, I remembered in https://github.com/coreutils/coreutils/blob/master/src/tail.c#L1574 there is a special case handling when the event happened after tail already tried to open the file and print its content and before inotify watcher has been set up.

And I added some sleep just before notifier

fn follow(files: &mut FileHandling, settings: &mut Settings) -> UResult<()> {
    thread::sleep(Duration::from_millis(4000));
    let mut process = platform::ProcessChecker::new(settings.pid);

and the F-headers.sh test is failing now every time I run it.