Should fix the tests that depends on certain locale

[shrektan]

For example, test 168.1, I have to set the locale to "C" in order to parse the datetimes correctly (otherwise they are all NAs) because my default locale is Chinese (Simplified)_People's Republic of China.936.

Another example is the test 1966, it simply throws an error on my computer complains the file name is invalid.

data.table/inst/tests/tests.Rraw

Lines 468 to 472 in 676c9b2

	# IDateTime conversion methods that ggplot2 uses (it calls as.data.frame method)
	datetimes = c("2011 NOV18 09:29:16", "2011 NOV18 10:42:40", "2011 NOV18 23:47:12",
	"2011 NOV19 01:06:01", "2011 NOV19 11:35:34", "2011 NOV19 11:51:09")
	DT = IDateTime(strptime(datetimes,"%Y %b%d %H:%M:%S"))
	test(168.1, DT[,as.data.frame(itime)], data.frame(V1=as.ITime(x<-c("09:29:16","10:42:40","23:47:12","01:06:01","11:35:34","11:51:09"))))

data.table/inst/tests/tests.Rraw

Lines 13012 to 13014 in 676c9b2

	f = tempfile("\u00f6"); cat("3.14", file = f)
	fn = enc2native(f); f8 = enc2utf8(f)
	test(1966.1, fread(fn), data.table(V1=3.14))

[shrektan]

I actually get tons of errors regarding to the tests for fread(). Most of them are about \r\n versus \n. For example:

Running test id 1445      Test 1445 ran without errors but failed check that x equals y:
> x = fread(testDir("doublequote_newline.csv"))[7:10] 
   A                                                             B
1: 1                                                             a
2: 1 embedded ""field""\r\nwith some embedded new\r\nlines as well
3: 2                                                  not this one
4: 1                                                             a
    [Key= Types=int,cha Classes=int,cha]
1:                                      
2:                                      
3:                                      
4:                                      
> y = data.table(A = c(1L, 1L, 2L, 1L), B = c("a", "embedded \"\"field\"\"\nwith some embedded new\nlines as well",      "not this one", "a")) 
   A                                                         B
1: 1                                                         a
2: 1 embedded ""field""\nwith some embedded new\nlines as well
3: 2                                              not this one
4: 1                                                         a
    [Key= Types=int,cha Classes=int,cha]
1:                                      
2:                                      
3:                                      
4:                                      
Column 'B': 1 string mismatch

I rarely use fread() thus have no idea of the cause. In addition, I'm quite curious why it can pass the autotest running in the Windows machine.

The full test output on Windows 10 (the locale is: Chinese (Simplified)_China.936)

> test.data.table()
Running C:/app/R/R-3.5.3/library/data.table/tests/tests.Rraw 

**** Suggested package nanotime is not installed. Tests using it will be skipped.

Running test id 1445      Test 1445 ran without errors but failed check that x equals y:
> x = fread(testDir("doublequote_newline.csv"))[7:10] 
   A                                                             B
1: 1                                                             a
2: 1 embedded ""field""\r\nwith some embedded new\r\nlines as well
3: 2                                                  not this one
4: 1                                                             a
    [Key= Types=int,cha Classes=int,cha]
1:                                      
2:                                      
3:                                      
4:                                      
> y = data.table(A = c(1L, 1L, 2L, 1L), B = c("a", "embedded \"\"field\"\"\nwith some embedded new\nlines as well",      "not this one", "a")) 
   A                                                         B
1: 1                                                         a
2: 1 embedded ""field""\nwith some embedded new\nlines as well
3: 2                                              not this one
4: 1                                                         a
    [Key= Types=int,cha Classes=int,cha]
1:                                      
2:                                      
3:                                      
4:                                      
Column 'B': 1 string mismatch
Running test id 1585.1      Test 1585.1 ran without errors but failed check that x equals y:
> x = f1(testDir("536_fread_fill_1.txt")) 
First 6 of 3 (type 'list'):   a b  c
1 1 2 qq
2 1 2 qq
3 1 2 qq
4 1 2 qq
5 4 5   
6 1 2 qq
> y = f2(testDir("536_fread_fill_1.txt"))[-28, ] 
First 6 of 3 (type 'list'):    a  b  c
1  1  2 qq
2  1  2 qq
3  1  2 qq
4  1  2 qq
5  4  5   
6 NA NA   
Running test id 1585.3      Test 1585.3 ran without errors but failed check that x equals y:
> x = f1(testDir("536_fread_fill_2.txt")) 
First 6 of 3 (type 'list'):   a b  c
1 1 2 qq
2 1 2 qq
3 1 2 qq
4 1 2 qq
5 4 5   
6 1 2 qq
> y = f2(testDir("536_fread_fill_2.txt")) 
First 6 of 3 (type 'list'):    a  b  c
1  1  2 qq
2  1  2 qq
3  1  2 qq
4  1  2 qq
5  4  5   
6 NA NA   
Running test id 1585.5      Test 1585.5 ran without errors but failed check that x equals y:
> x = f1(testDir("536_fread_fill_3_extreme.txt")) 
First 6 of 3 (type 'list'):    a
1  1
2 NA
3 NA
4 NA
5  2
6  3
                                                                                        b
1 first,,,,,,,,,,,,,,,,\r\n\r\nsecond,,,,,,,\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\nthird
2                                                                                        
3                                                                                        
4                                                                                        
5                                                                                     foo
6                                                                                        
   c
1  2
2 NA
3 NA
4 NA
5 NA
6 NA
> y = f2(testDir("536_fread_fill_3_extreme.txt"))[-9, ] 
First 6 of 3 (type 'list'):    a                                                               b  c
1  1 first,,,,,,,,,,,,,,,,\n\nsecond,,,,,,,\n\n\n\n\n\n\n\n\n\nthird  2
2 NA                                                                 NA
3 NA                                                                 NA
4 NA                                                                 NA
5  2                                                             foo NA
6  3                                                                 NA
Running test id 1585.6      Test 1585.6 ran without errors but failed check that x equals y:
> x = f1(testDir("536_fread_fill_3_extreme.txt"), b = TRUE) 
First 6 of 3 (type 'list'):   a
1 1
2 2
3 3
                                                                                        b
1 first,,,,,,,,,,,,,,,,\r\n\r\nsecond,,,,,,,\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\nthird
2                                                                                     foo
3                                                                                        
   c
1  2
2 NA
3 NA
> y = f2(testDir("536_fread_fill_3_extreme.txt"), b = TRUE) 
First 6 of 3 (type 'list'):   a                                                               b  c
1 1 first,,,,,,,,,,,,,,,,\n\nsecond,,,,,,,\n\n\n\n\n\n\n\n\n\nthird  2
2 2                                                             foo NA
3 3                                                                 NA
Running test id 1585.7      Test 1585.7 ran without errors but failed check that x equals y:
> x = f1(testDir("536_fread_fill_4.txt")) 
First 6 of 3 (type 'list'):    a  b  c
1 NA NA   
2  1  2 qq
3  1  2 qq
4  1  2 qq
5  1  2 qq
6  4  5   
> y = f2(testDir("536_fread_fill_4.txt"))[-29, ] 
First 6 of 3 (type 'list'):    a  b  c
1 NA NA   
2  1  2 qq
3  1  2 qq
4  1  2 qq
5  1  2 qq
6  4  5   
Running test id 1590.16      Test 1590.16 ran without errors but failed check that x equals y:
> x = forderv(c(x2, x1, x1, x2)) 
First 6 of 4 (type 'integer'): [1] 2 3 1 4
> y = INT(1, 4, 2, 3) 
First 6 of 4 (type 'integer'): [1] 1 4 2 3
Mean relative difference: 0.4
Running test id 1621.1      Test 1621.1 ran without errors but failed check that x equals y:
> x = fread(testDir("issue_1116_fread_few_lines.txt"), logical01 = FALSE) 
    x                     y  [Key= Types=int,cha Classes=int,cha]
 1: 1 a\r\n,,,,,,,,,,\r\n_b                                      
 2: 1 a\r\n,,,,,,,,,,\r\n_b                                      
---                                                              
43: 1 a\r\n,,,,,,,,,,\r\n_b                                      
44: 1 a\r\n,,,,,,,,,,\r\n_b                                      
> y = setDT(read.delim(testDir("issue_1116_fread_few_lines.txt"), stringsAsFactors = FALSE,      sep = ",", check.names = FALSE)) 
    x                 y  [Key= Types=int,cha Classes=int,cha]
 1: 1 a\n,,,,,,,,,,\n_b                                      
 2: 1 a\n,,,,,,,,,,\n_b                                      
---                                                          
43: 1 a\n,,,,,,,,,,\n_b                                      
44: 1 a\n,,,,,,,,,,\n_b                                      
Column 'y': 44 string mismatches
Running test id 1621.2      Test 1621.2 ran without errors but failed check that x equals y:
> x = fread(testDir("issue_1116_fread_few_lines_2.txt"), logical01 = FALSE) 
    x                       y  [Key= Types=int,cha Classes=int,cha]
 1: 1 a\r\n\r\n,,,,,,,,\r\n_b                                      
 2: 1 a\r\n\r\n,,,,,,,,\r\n_b                                      
---                                                                
43: 1 a\r\n\r\n,,,,,,,,\r\n_b                                      
44: 1 a\r\n\r\n,,,,,,,,\r\n_b                                      
> y = setDT(read.delim(testDir("issue_1116_fread_few_lines_2.txt"),      stringsAsFactors = FALSE, sep = ",", check.names = FALSE)) 
    x                 y  [Key= Types=int,cha Classes=int,cha]
 1: 1 a\n\n,,,,,,,,\n_b                                      
 2: 1 a\n\n,,,,,,,,\n_b                                      
---                                                          
43: 1 a\n\n,,,,,,,,\n_b                                      
44: 1 a\n\n,,,,,,,,\n_b                                      
Column 'y': 44 string mismatches
Running test id 1869.7      Test 1869.7 didn't produce correct output:
Expected: <<Copying file in RAM.*file is very unusual.*one single column, ends with 2 or more end-of-line.*and is a multiple of 4096>>
Observed: <<omp_get_num_procs()==4\nR_DATATABLE_NUM_PROCS_PERCENT=="" (default 50)\nR_DATATABLE_NUM_THREADS==""\nomp_get_thread_limit()==2147483647\nomp_get_max_threads()==4\nOMP_THREAD_LIMIT==""\nOMP_NUM_THREADS==""\ndata.table is using 2 threads. This is set on startup, and by setDTthreads(). See ?setDTthreads.\nRestoreAfterFork==true\nInput contains no \n. Taking this to be a filename to open\n[01] Check arguments\n  Using 2 threads (omp_get_max_threads()=4, nth=2)\n  NAstrings = [<<NA>>]\n  None of the NAstrings look like numbers.\n  show progress = 1\n  0/1 column will be read as integer\n[02] Opening the file\n  Opening file C:/app/R/R-3.5.3/library/data.table/tests/onecol4096.csv\n  File opened, size = 4.256KB (4358 bytes).\n  Memory mapped ok\n[03] Detect and skip BOM\n[04] Arrange mmap to be \0 terminated\n  \n has been found in the input and different lines can end with different line endings (e.g. mixed \n and \r\n in one file). This is common and ideal.\n[05] Skipping initial rows if needed\n  Positioned on line 1 starting: <<A>>\n[06] Detect separator, quoting rule, and ncolumns\n  Detecting sep automatically ...\n  No sep and quote rule found a block of 2x2 or greater. Single column input.\n  Detected 1 columns on line 1. This line is either column names or first data row. Line starts as: <<A>>\n  Quote rule picked = 0\n  fill=false and the most number of columns found is 1\n  1-column file ends with 2 or more end-of-line. Restoring last eol using extra byte in cow page.\n[07] Detect column types, good nrow estimate and whether first row is column names\n  Number of sampling jump points = 1 because (4357 bytes from row 1 to eof) / (2 * 1669 jump0size) == 1\n  Type codes (jump 000)    : A  Quote rule 0\n  Type codes (jump 001)    : A  Quote rule 0\n  'header' determined to be true because all columns are type string and a better guess is not possible\n  =====\n  Sampled 153 rows (handled \n inside quoted fields) at 2 jump points\n  Bytes from first data row on line 2 to the end of last row: 4354\n  Line length: mean=16.46 sd=2.85 min=1 max=18\n  Estimated number of rows: 4354 / 16.46 = 265\n  Initial alloc = 404 rows (265 + 52%) using bytes/max(mean-2*sd,min) clamped between [1.1*estn, 2.0*estn]\n  =====\n[08] Assign column names\n[09] Apply user overrides on column types\n  After 0 type and 0 drop user overrides : A\n[10] Allocate memory for the datatable\n  Allocating 1 column slots (1 - 0 dropped) with 404 rows\n[11] Read the data\n  jumps=[0..1), chunk_size=1048576, total_size=4354\nRead 261 rows x 1 columns from 4.256KB (4358 bytes) file in 00:00.000 wall clock time\n[12] Finalizing the datatable\n  Type counts:\n         1 : string    'A'\n=============================\n   0.000s (  0%) Memory map 0.000GB file\n   0.000s (  0%) sep='�' ncol=1 and header detection\n   0.000s (  0%) Column type detection using 153 sample rows\n   0.000s (  0%) Allocation of 404 rows x 1 cols (0.000GB) of which 261 ( 65%) rows used\n   0.000s (  0%) Reading 1 chunks (0 swept) of 1.000MB (each chunk 261 rows) using 1 threads\n   +    0.000s (  0%) Parse to row-major thread buffers (grown 0 times)\n   +    0.000s (  0%) Transpose\n   +    0.000s (  0%) Waiting\n   0.000s (  0%) Rereading 0 columns due to out-of-sample type exceptions\n   0.000s        Total\n                  A\n 1: FooBarBazQux000\n 2: FooBarBazQux001\n 3:                \n 4: FooBarBazQux245\n 5:                \n 6: FooBarBazQux254\n 7:    FooBarBazQux\n 8:     FooBarBaz12\n 9: FooBarBazQux256\n10:                \n11:                \n12:                >>
Running test id 1965      Error in file(file, ifelse(append, "a", "w")) : 
  cannot open the connection

[shrektan]

I see... It's because Git auto converted the \n to \r\n on Windows (the option is enabled by default)...

[MichaelChirico]

@shrektan please try gain on latest master, this should have been covered in #3553 I think?

[2005m]

When running data.table::test.data.table on windows I have 6 errors. My local are set to English_United Kingdom.1252 . Errors are related to test 1445, 1585.1, 1585.3, 1585.5, 1585.6 and 1585.7. As @shrektan said , it is related to \n for test 1445. Should we unclude a if(.Platform$OS.type == "windows") in the test file?

[MichaelChirico]

The English-v-native part will become a lot easier after #3935 is eventually merged. EOL issues are another thing...

[2005m]

Test 1445 seems to work with R 3.6.1 but not 3.5.3 on windows.

[shrektan]

As metioned above, if you are using Git to clone the repo to your Windows computer, you will find tests failures. This is because when Git clones the repo, it automatically convert the \n to \r\n. It means if you download the zip file directly from GitHub and re-run the tests, you may find the error related to \n -> \r\n won't occur. However, there're still errors due to locales (at least on my computer).

[jangorecki]

I am getting the same issue as mentioned above.

R version 3.6.3 (2020-02-29) -- "Holding the Windsock"
Copyright (C) 2020 The R Foundation for Statistical Computing
Platform: x86_64-w64-mingw32/x64 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> require(data.table)
Loading required package: data.table
> 
> test.data.table()  # runs the main test suite of 5,000+ tests in /inst/tests/tests.Rraw
getDTthreads(verbose=TRUE):
  omp_get_num_procs()            2
  R_DATATABLE_NUM_PROCS_PERCENT  unset (default 50)
  R_DATATABLE_NUM_THREADS        unset
  omp_get_thread_limit()         2147483647
  omp_get_max_threads()          2
  OMP_THREAD_LIMIT               unset
  OMP_NUM_THREADS                unset
  RestoreAfterFork               true
  data.table is using 1 threads. See ?setDTthreads.
test.data.table() running: C:/GitLab-Runner/builds/jangorecki/win-ci/data.table.Rcheck/data.table/tests/tests.Rraw 

**** Suggested package bit64 is not installed. Tests using it will be skipped.


**** Suggested package xts is not installed. Tests using it will be skipped.


**** Suggested package nanotime is not installed. Tests using it will be skipped.


**** Suggested package R.utils is not installed. Tests using it will be skipped.


**** Suggested package yaml is not installed. Tests using it will be skipped.

Test 1445 ran without errors but failed check that x equals y:
> x = fread(testDir("doublequote_newline.csv"))[7:10] 
   A                                                             B
1: 1                                                             a
2: 1 embedded ""field""\r\nwith some embedded new\r\nlines as well
3: 2                                                  not this one
4: 1                                                             a
    [Key= Types=int,cha Classes=int,cha]
1:                                      
2:                                      
3:                                      
4:                                      
> y = data.table(A = c(1L, 1L, 2L, 1L), B = c("a", "embedded \"\"field\"\"\nwith some embedded new\nlines as well",      "not this one", "a")) 
   A                                                         B
1: 1                                                         a
2: 1 embedded ""field""\nwith some embedded new\nlines as well
3: 2                                              not this one
4: 1                                                         a
    [Key= Types=int,cha Classes=int,cha]
1:                                      
2:                                      
3:                                      
4:                                      
Column 'B': 1 string mismatch
Test 1585.1 ran without errors but failed check that x equals y:
> x = f1(testDir("536_fread_fill_1.txt")) 
First 3 of 3 (type 'list'): 
> y = f2(testDir("536_fread_fill_1.txt"))[-28, ] 
First 3 of 3 (type 'list'): 
Test 1585.3 ran without errors but failed check that x equals y:
> x = f1(testDir("536_fread_fill_2.txt")) 
First 3 of 3 (type 'list'): 
> y = f2(testDir("536_fread_fill_2.txt")) 
First 3 of 3 (type 'list'): 
Test 1585.5 ran without errors but failed check that x equals y:
> x = f1(testDir("536_fread_fill_3_extreme.txt")) 
First 3 of 3 (type 'list'): 
> y = f2(testDir("536_fread_fill_3_extreme.txt"))[-9, ] 
First 3 of 3 (type 'list'): 
Test 1585.6 ran without errors but failed check that x equals y:
> x = f1(testDir("536_fread_fill_3_extreme.txt"), b = TRUE) 
First 3 of 3 (type 'list'): 
> y = f2(testDir("536_fread_fill_3_extreme.txt"), b = TRUE) 
First 3 of 3 (type 'list'): 
Test 1585.7 ran without errors but failed check that x equals y:
> x = f1(testDir("536_fread_fill_4.txt")) 
First 3 of 3 (type 'list'): 
> y = f2(testDir("536_fread_fill_4.txt"))[-29, ] 
First 3 of 3 (type 'list'): 
Test 1760 not run because this session either has no OpenMP or has been limited to one thread (e.g. under UBSAN and ASAN)

Tue Apr 14 22:57:28 2020  endian==little, sizeof(long double)==16, sizeof(pointer)==8, TZ=Africa/Casablanca, locale='LC_COLLATE=C;LC_CTYPE=English_United States.1252;LC_MONETARY=English_United States.1252;LC_NUMERIC=C;LC_TIME=English_United States.1252', l10n_info()='MBCS=FALSE; UTF-8=FALSE; Latin-1=TRUE; codepage=1252', getDTthreads()='omp_get_num_procs()==2; R_DATATABLE_NUM_PROCS_PERCENT==unset (default 50); R_DATATABLE_NUM_THREADS==unset; omp_get_thread_limit()==2147483647; omp_get_max_threads()==2; OMP_THREAD_LIMIT==unset; OMP_NUM_THREADS==unset; RestoreAfterFork==true; data.table is using 1 threads. See ?setDTthreads.'
Error in test.data.table() : 
  6 errors out of 8954. Search tests/tests.Rraw for test numbers: 1445, 1585.1, 1585.3, 1585.5, 1585.6, 1585.7.
Execution halted

[jangorecki]

using git config --global core.autocrlf false resolved my issue

[jangorecki]

@shrektan could you please check if #4431 will resolve all problems here? not sure but fresh git clone may be needed to verify that (to avoid windows newline conversion):

mkdir tmp
cd tmp
git clone --depth=1 -o upstream https://github.com/Rdatatable/data.table.git
cd data.table
git fetch upstream ci2:ci2
git checkout ci2
make build && make install && make test

[shrektan]

1445 still produces the error.

 12 errors out of 8955. Search tests/tests.Rraw for test numbers: 1445, 1585.1, 1585.3, 1585.5, 1585.6, 1585.7, 1966.1, 1966.2, 1966.3, 1966.4, 1966.5, 1966.6.

@jangorecki I expect it will fail, actually.

I see you changed the ".gitattributes" in the branch ci2. But when I execute git clone, I think the .gitattributes in the master branch (which is nothing) will be used. So checkout "ci2" doesn't affect anything later...