zstd --fileliste windows_edited_filelist_with_empty_last_line.txt; fgets() result code == NULL "to early" due to empty line ending, low priority

[halifir]

zstd 1.4.8

When I use a fileliste edited with an Windows Editor als ASCII with CR+LF line endings and a last empty line,
i get the error
zstd -f -2 --fileliste windows_edited_filelist_with_empty_last_line.txt
Error : util.c, 317 : fgets(buf, (int) len, file)

When debugging zstd I saw, that the readLineFromChar is called
due the fact, that there is still something in the file (not feof() reached).

But the fgets result for the last line is NULL - and errno == 0. And then, after this fgets with result == NULL feof will be true.

To avoid this problem with my filelist, i changed the CONTROL (fgets) to CONTROL ((fgets != NULL) && (errno == 0)) like this, and a had success:

util.c
static size_t readLineFromFile(char* buf, size_t len, FILE* file)
{
...
    CONTROL( (fgets(buf, (int) len, file) != NULL) || (errno == 0)  ); // <<< last fgets for empty line in windows edited fileliste file produces a NULL and errno==0 


static int
readLinesFromFile(void* dst, size_t dstCapacity,
            const char* inputFileName)
{

    **while ( !feof(inputFile) ) {**

It is just an idea, to change util.c to improve robustness of zstd.

[Cyan4973]

Okay, so basically, --filelist should also support files finishing with one (or more?) additional empty line(s),
or even ignore empty lines anywhere in the list altogether.

[halifir]

{ by the way: zstd is soooo fast. great job. thank you. }

echo "filename1" >fileliste.txt
echo "filename2" >>fileliste.txt

is producing the empty line at end, because echo finish the line with cr/lf on windows.

I guess on UNIX without the "T" Text transformation in the stream, the last \n is counted in the filesize of file.

And this will help the feof() right before the fgets() to detect the end of file condition after text "filename2" in this example.

On Windows the fgets() Text converter may be counting chars (after fgets without \r, just \n) and
the feof() do not signal the end of file condition after reading "filename2".

I is the Text converter inside fgets that detects the end of file condition. So one may say feof() is wrong, on the other hand side Microsoft documented fgets(), that NULL from fgets is not always an error condition.
"NULL is returned to indicate an error or an end-of-file condition. "
https://docs.microsoft.com/de-de/cpp/c-runtime-library/reference/fgets-fgetws?view=msvc-160

The last empty line is no "ignore" condition, but just the difference in the feof() sensor working on Linux a little bit different than in the Windows Text Mode. (I read the Microsoft help like this: dont trust feof() in Text Mode and use fgets() == NULL to detect the end of file.)

The feof() that seems to be the "problem" which end a whil on Linux but not on Windows seems to be here - i will do some more testing tomorrow...

readLinesFromFile(void* dst, size_t dstCapacity,   const char* inputFileName)
`   while ( !feof(inputFile) ) {`   // does no detect end of file with text mode inputFile on my windows machine

For something completly different

I write some batch jobs.

If somebody would ask me for empty lines in the fillist, I would prefer a similar handling of --filellist like in the xargs usage. = ignore empty lines = but this is a very personal view and has nothing to do with the fgets(text mode in windows) handling and end of file detection.

! == # instead of the usual hashtag symbol for comments in bash, i use !, because the hashtag changes the fontsize... unexpected

! my usage for --filelist.txt
ls >filelist.txt # or dir >filelist.txt for cmd.exe on windows

! doing something with filelist.txt - de selecting file not to compress etc... probably with programs, probably manual
! and than compress. extraordinary fast with zstd

test -s filelist.txt && xargs zstd -q -T0 -2 -f -- <filelist.txt # works great with zstd on windows and linux - zooom! this is my solution, so I for myself don't need any fix

test -s filelist.txt && zstd -q --fileliste filellist.txt
! break with the Error : util.c, 317 : fgets(buf, (int) len, file)
! on my windows because of fgets() == NULL
! if i delete the last cr/lf than feof() send the end of file signal before fgets() and everything is working well

! short help:
test -s filelist.txt && # test if filelist.txt is not empty && and then start the cmd
xargs command and parameters < filelist.txt # a methode to start command with the parameters and xargs fill up the cmd + parameter line with additional arguments from filellist.txt until the command line is full. if filellist.,txt is long, xargs will start more than one command.

[Cyan4973]

fix currently proposed in #2620

I could not reproduce the proposed test using echo X > filelist method proposed here.
Instead, I had to manually create and edit a text file with notepad,
the resulting file would feature cr+lf end of lines (outcome verified with xxd).
The proposed fix is compatible with such files.