Error with dec=',', sep = 'auto' but detected as ','

[MichaelChirico]

Possibly related: #2750

fread('A,B,C\n1,+,4\n2,-,5\n3,-,6\n', dec=',', verbose=TRUE)

Stopped early on line 2. Expected 3 fields but found 3. Consider fill=TRUE and comment.char=. First discarded non-empty line: <<1,+,4>>

with dec=',', +,4 reads as +.4.

But we've already detected sep=','... not sure the correct behavior for manually specified dec=','.

In any case the error message is strange "Expected 3 fields but found 3"

[st-pasha]

When dec=',' and sep=',', we should not attempt to use the float parser at all, unless the field is quoted.
Otherwise we'll end up reading lines like 1,2,3,4 as two numbers 1.2 and 3.4, which is rather insane...

[MichaelChirico]

rather insane...

Actually found a file on some Spanish government website for which that's the correct parsing 😂 but yes, indeed insane as a general thing.

This issue is slightly different. We already block sep==dec:

data.table/src/fread.c

Line 1180 in ff3e7d4

if (args.sep == dec) STOP(_("sep == dec ('%c') is not allowed"), dec);

Here what happens is, since sep detection comes first, right now we aren't enforcing sep != dec when sep starts out as 'auto' but ends up as ','.

Maybe the best option is to change here:

data.table/src/fread.c

Line 1455 in ff3e7d4

char seps[]=",|;\t "; // default seps in order of preference. See ?fread.

to

char seps[] = dec == ',' ? "|;\t " : ",|;\t ";

i.e. if user has manually specified dec=',', prevent sep='auto' from finding sep=','