test262 are not match unix testing result on windows

[lygstate]

JerryScript revision

9676500

Build platform

C:\Users\lygstate>python -c "import platform; print(platform.platform())"
Windows-10-10.0.19041-SP0

Build steps

For ES 5.1

PS C:\work\study\languages\typescript\jerryscript> python tools/run-tests.py --test262
Build command:
cmd \
        /S \
        /C \
        C:\work\study\languages\typescript\jerryscript\tools\build.py \
        --profile=es5.1 \
        --builddir=C:\work\study\languages\typescript\jerryscript\build\tests\test262_tests \
        --install=C:\work\study\languages\typescript\jerryscript\build\tests\test262_tests\local
Test command:
TZ='America/Los_Angeles' \
cmd \
        /S \
        /C \
        C:\work\study\languages\typescript\jerryscript\tools\runners/run-test-suite-test262.py \
        --engine \
        C:\work\study\languages\typescript\jerryscript\build\tests\test262_tests\local\bin\jerry.exe --test262-object \
        --test-dir \
        C:\work\study\languages\typescript\jerryscript\tests/test262 \
        --es51
.................................................. Executed 5000 tests.
.................................................. Executed 10000 tests.
.................
=== Summary ===
 - Ran 11725 tests
 - Passed 11557 tests (98.6%)
 - Failed 168 tests (1.4%)

For ES6

PS C:\work\study\languages\typescript\jerryscript> python tools/run-tests.py --test262-es2015 update
Build command:
cmd \
        /S \
        /C \
        C:\work\study\languages\typescript\jerryscript\tools\build.py \
        --profile=es.next \
        --line-info=on \
        --error-messages=on \
        --builddir=C:\work\study\languages\typescript\jerryscript\build\tests\test262_tests_es2015 \   
        --install=C:\work\study\languages\typescript\jerryscript\build\tests\test262_tests_es2015\local
Test command:
TZ='America/Los_Angeles' \
cmd \
        /S \
        /C \
        C:\work\study\languages\typescript\jerryscript\tools\runners/run-test-suite-test262.py \
        --engine \
        C:\work\study\languages\typescript\jerryscript\build\tests\test262_tests_es2015\local\bin\jerry.exe --test262-object \
        --test-dir \
        C:\work\study\languages\typescript\jerryscript\tests/test262 \
        --es2015 \
        update
.................................................. Executed 5000 tests.
.................................................. Executed 10000 tests.
.................................................. Executed 15000 tests.
ERROR:root:Error disposing temp file: [WinError 32] 另一个程序正在使用此文件，进程无法访问。: 'C:\\Users\\lygstate\\AppData\\Local\\Temp\\test262-out-kkxx3goy'
ERROR:root:Error disposing temp file: [WinError 32] 另一个程序正在使用此文件，进程无法访问。: 'C:\\Users\\lygstate\\AppData\\Local\\Temp\\test262-out-x6oy5j1s'
ERROR:root:Error disposing temp file: [WinError 32] 另一个程序正在使用此文件，进程无法访问。: 'C:\\Users\\lygstate\\AppData\\Local\\Temp\\test262-err-nj697qhg'
ERROR:root:Error disposing temp file: [WinError 32] 另一个程序正在使用此文件，进程无法访问。: 'C:\\Users\\lygstate\\AppData\\Local\\Temp\\test262-err-ymdteaqd'
ERROR:root:Error disposing temp file: [WinError 32] 另一个程序正在使用此文件，进程无法访问。: 'C:\\Users\\lygstate\\AppData\\Local\\Temp\\test262-out-5b09cyb1'
ERROR:root:Error disposing temp file: [WinError 32] 另一个程序正在使用此文件，进程无法访问。: 'C:\\Users\\lygstate\\AppData\\Local\\Temp\\test262-out-10s2ozab'
ERROR:root:Error disposing temp file: [WinError 32] 另一个程序正在使用此文件，进程无法访问。: 'C:\\Users\\lygstate\\AppData\\Local\\Temp\\test262-err-dzy30erw'
ERROR:root:Error disposing temp file: [WinError 32] 另一个程序正在使用此文件，进程无法访问。: 'C:\\Users\\lygstate\\AppData\\Local\\Temp\\test262-err-wfrmjsp4'
.................................................. Executed 20000 tests.
.................................................. Executed 25000 tests.
ERROR:root:Error disposing temp file: [WinError 32] 另一个程序正在使用此文件，进程无法访问。: 'C:\\Users\\lygstate\\AppData\\Local\\Temp\\test262-out-qu8dgc37'
ERROR:root:Error disposing temp file: [WinError 32] 另一个程序正在使用此文件，进程无法访问。: 'C:\\Users\\lygstate\\AppData\\Local\\Temp\\test262-out-e_0fjy1l'
ERROR:root:Error disposing temp file: [WinError 32] 另一个程序正在使用此文件，进程无法访问。: 'C:\\Users\\lygstate\\AppData\\Local\\Temp\\test262-err-k3yhbr7v'
ERROR:root:Error disposing temp file: [WinError 32] 另一个程序正在使用此文件，进程无法访问。: 'C:\\Users\\lygstate\\AppData\\Local\\Temp\\test262-err-41uefzcl'
.....................................
=== Summary ===
 - Ran 28709 tests
 - Passed 28070 tests (97.8%)
 - Failed 639 tests (2.2%)

=== Summary - updating excludelist ===

New failing tests added to the excludelist
  built-ins/String/raw/special-characters.js
  language/expressions/template-literal/tv-line-continuation.js
  language/expressions/template-literal/tv-line-terminator-sequence.js

Excludelist was updated succesfully.

ESNEXT built-ins,annexB,harness,intl402

PS C:\work\study\languages\typescript\jerryscript> python tools/run-tests.py --test262-esnext update --test262-test-list built-ins,annexB,harness,intl402
Build command:
cmd \
        /S \
        /C \
        C:\work\study\languages\typescript\jerryscript\tools\build.py \
        --profile=es.next \
        --line-info=on \
        --error-messages=on \
        --mem-heap=20480 \
        --builddir=C:\work\study\languages\typescript\jerryscript\build\tests\test262_tests_esnext \
        --install=C:\work\study\languages\typescript\jerryscript\build\tests\test262_tests_esnext\local
Test command:
TZ='America/Los_Angeles' \
cmd \
        /S \
        /C \
        C:\work\study\languages\typescript\jerryscript\tools\runners/run-test-suite-test262.py \
        --engine \
        C:\work\study\languages\typescript\jerryscript\build\tests\test262_tests_esnext\local\bin\jerry.exe --test262-object \
        --test-dir \
        C:\work\study\languages\typescript\jerryscript\tests/test262 \
        --esnext \
        update \
        --test262-test-list \
        built-ins,annexB,harness,intl402
Cloning into 'C:\work\study\languages\typescript\jerryscript\tests/test262\esnext'...
remote: Enumerating objects: 99, done.
remote: Counting objects: 100% (99/99), done.
remote: Compressing objects: 100% (76/76), done.
Receiving objects: 100% (316202/316202), 89.99 MiB | 3.70 MiB/s, done.103 eceiving objects: 100% (316202/316202), 88.11 MiB | 6.56 MiB/s

Resolving deltas: 100% (268501/268501), done.
Updating files: 100% (46022/46022), done.
Note: switching to '281eb10b2844929a7c0ac04527f5b42ce56509fd'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 281eb10b28 Intl.DateTimeFormat.{formatRange,formatRangeToParts} should throw TypeError instead of RangeError if either of parameter is undefined (#2685)
.................................................. Executed 10000 tests.
.................................................. Executed 15000 tests.
.................................................. Executed 20000 tests.
.................................................. Executed 25000 tests.
.................................................. Executed 30000 tests.
..........................................
=== Summary ===
 - Ran 34297 tests
 - Passed 28527 tests (83.2%)
 - Failed 5770 tests (16.8%)

=== Summary - updating excludelist ===

New failing tests added to the excludelist
  built-ins/String/raw/special-characters.js

Excludelist was updated succesfully.

ESNEXT language

PS C:\work\study\languages\typescript\jerryscript> python tools/run-tests.py --test262-esnext update --test262-test-list language
Build command:
cmd \
        /S \
        /C \
        C:\work\study\languages\typescript\jerryscript\tools\build.py \
        --profile=es.next \
        --line-info=on \
        --error-messages=on \
        --mem-heap=20480 \
        --builddir=C:\work\study\languages\typescript\jerryscript\build\tests\test262_tests_esnext \
        --install=C:\work\study\languages\typescript\jerryscript\build\tests\test262_tests_esnext\local
Test command:
TZ='America/Los_Angeles' \
cmd \
        /S \
        /C \
        C:\work\study\languages\typescript\jerryscript\tools\runners/run-test-suite-test262.py \
        --engine \
        C:\work\study\languages\typescript\jerryscript\build\tests\test262_tests_esnext\local\bin\jerry.exe --test262-object \
        --test-dir \
        C:\work\study\languages\typescript\jerryscript\tests/test262 \
        --esnext \
        update \
        --test262-test-list \
        language
.................................................. Executed 5000 tests.
.................................................. Executed 10000 tests.
.................................................. Executed 15000 tests.
.................................................. Executed 20000 tests.
.................................................. Executed 25000 tests.
.................................................. Executed 30000 tests.
.................................................. Executed 35000 tests.
.................................................. Executed 40000 tests.
........
=== Summary ===
 - Ran 40835 tests
 - Passed 30743 tests (75.3%)
 - Failed 10092 tests (24.7%)

=== Summary - updating excludelist ===

New failing tests added to the excludelist
  language/expressions/template-literal/tv-line-continuation.js
  language/expressions/template-literal/tv-line-terminator-sequence.js
  language/literals/string/line-continuation-double.js
  language/literals/string/line-continuation-single.js

Excludelist was updated succesfully.

[ossy-szeged]

I haven't seen any extra failure on Windows with ES5.1, ES2015 and ES.Next version of test262.
Tested on the latest master ( 7b00db4 ) in cmd.exe not in PS.

(update: tested in PS too, got the same result)

There is only one es2015 test hang (infinite loop) in strict and non-strict mode, but it isn't related
to Windows, but a spec change - language/statements/for-of/iterator-next-reference.js.

[lygstate]

I haven't seen any extra failure on Windows with ES5.1, ES2015 and ES.Next version of test262.
Tested on the latest master ( 7b00db4 ) in cmd.exe not in PS.

(update: tested in PS too, got the same result)

There is only one es2015 test hang (infinite loop) in strict and non-strict mode, but it isn't related
to Windows, but a spec change - language/statements/for-of/iterator-next-reference.js.

It's matched now, I found the cause why not match, because my locale encoding are GBK, and cause python encoding/decoding failure. Fixed by #4390