Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 69 additions & 4 deletions .github/workflows/gh-actions.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,9 @@ jobs:
with:
python-version: '2.7' # needed by jerry-debugger
- run: $RUNNER -q --jerry-tests
- run: $RUNNER -q --jerry-tests --build-debug
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now each job takes almost 2 times longer, since it's running the tests in release and debug mode.

I'd suggest to create <original_job_name>_Debug` jobs instead of burden each job with running it in debug mode as well. GH action can run 20 jobs in parallel so take it as an advantage.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/jerryscript-project/jerryscript/runs/1832072314?check_suite_focus=true
This is the test result, which test need split?
Or all need to be split?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

@rerobika rerobika Feb 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, most of them are okay within one job, but I'd definitely split these:
image

These show the biggest slowdowns.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#4389 fixes ES5_1, others OK to me

- run: $RUNNER -q --jerry-debugger
- run: $RUNNER -q --jerry-debugger --build-debug

Linux_x86_cpointer-32bit_Build_Correctness_Tests:
runs-on: ubuntu-latest
Expand All @@ -39,14 +41,23 @@ jobs:
- run: sudo apt update
- run: sudo apt install gcc-multilib
- run: $RUNNER -q --jerry-tests --buildoptions=--compile-flag=-m32,--cpointer-32bit=on
- run: $RUNNER -q --jerry-tests --buildoptions=--compile-flag=-m32,--cpointer-32bit=on --build-debug

OSX_x86-64_Build_Correctness_Unit_Tests:
runs-on: macos-latest
steps:
- uses: actions/checkout@v2
- run: $RUNNER -q --jerry-tests
# FIXME: regression-test-issue-3785.js currently are deadloop on OSX.
- run: $RUNNER -q --jerry-tests --skip-list=regression-test-issue-3785.js
- run: $RUNNER -q --unittests

OSX_x86-64_Build_Correctness_Unit_Tests_Debug:
runs-on: macos-latest
steps:
- uses: actions/checkout@v2
- run: $RUNNER -q --jerry-tests --build-debug
- run: $RUNNER -q --unittests --build-debug

Linux_x86-64_Build_Option_Tests:
runs-on: ubuntu-latest
steps:
Expand All @@ -60,45 +71,56 @@ jobs:
steps:
- uses: actions/checkout@v2
- run: $RUNNER --test262
- run: $RUNNER --test262 --build-debug

Conformance_Tests_ES2015:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- run: $RUNNER --test262-es2015=update
- run: $RUNNER --test262-es2015=update --build-debug
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Debug artifact isn't uploaded, but it should.

- uses: actions/upload-artifact@v2
if: success() || failure()
with:
name: Test262-ES2015-results
path: build/tests/test262_tests_es2015/local/bin/test262.report
path: |
build/tests/test262_tests_es2015/local/bin/test262.report
build/tests/test262_tests_es2015-debug/local/bin/test262.report

Conformance_Tests_ESNext_A:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- run: $RUNNER --test262-esnext=update --test262-test-list=built-ins,annexB,harness,intl402
- run: $RUNNER --test262-esnext=update --test262-test-list=built-ins,annexB,harness,intl402 --build-debug
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Debug artifact isn't uploaded, but it should.

- uses: actions/upload-artifact@v2
if: success() || failure()
with:
name: Test262-ESNext-results-A
path: build/tests/test262_tests_esnext/local/bin/test262.report
path: |
build/tests/test262_tests_esnext/local/bin/test262.report
build/tests/test262_tests_esnext-debug/local/bin/test262.report

Conformance_Tests_ESNext_B:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- run: $RUNNER --test262-esnext=update --test262-test-list=language
- run: $RUNNER --test262-esnext=update --test262-test-list=language --build-debug
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Debug artifact isn't uploaded, but it should.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't understand this statement.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One line down you can see: "uses: actions/upload-artifact@v2" which upload build/tests/test262_tests_esnext/local/bin/test262.report as artifact to be able to download by the user and can analyze detailed test results. If we run debug tests inside the same job, we will need to upload those output too.

- uses: actions/upload-artifact@v2
if: success() || failure()
with:
name: Test262-ESNext-results-B
path: build/tests/test262_tests_esnext/local/bin/test262.report
path: |
build/tests/test262_tests_esnext/local/bin/test262.report
build/tests/test262_tests_esnext-debug/local/bin/test262.report

Unit_Tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- run: $RUNNER -q --unittests
- run: $RUNNER -q --unittests --build-debug

ASAN_Tests:
runs-on: ubuntu-latest
Expand All @@ -113,6 +135,19 @@ jobs:
--buildoptions=--stack-limit=0,--compile-flag=-fsanitize=address,--compile-flag=-m32,--compile-flag=-fno-omit-frame-pointer,--compile-flag=-fno-common,--compile-flag=-O2,--debug,--system-allocator=on,--linker-flag=-fuse-ld=gold
--skip-list=parser-oom.js,parser-oom2.js,stack-limit.js,regression-test-issue-2190.js,regression-test-issue-2258-2963.js,regression-test-issue-2448.js,regression-test-issue-2905.js,regression-test-issue-3785.js

ASAN_Tests_Debug:
runs-on: ubuntu-latest
env:
ASAN_OPTIONS: detect_stack_use_after_return=1:check_initialization_order=true:strict_init_order=true
steps:
- uses: actions/checkout@v2
- run: sudo apt update
- run: sudo apt install gcc-multilib
- run: >-
$RUNNER -q --jerry-tests --build-debug
--buildoptions=--stack-limit=0,--compile-flag=-fsanitize=address,--compile-flag=-m32,--compile-flag=-fno-omit-frame-pointer,--compile-flag=-fno-common,--compile-flag=-O2,--debug,--system-allocator=on,--linker-flag=-fuse-ld=gold
--skip-list=parser-oom.js,parser-oom2.js,stack-limit.js,regression-test-issue-2190.js,regression-test-issue-2258-2963.js,regression-test-issue-2448.js,regression-test-issue-2905.js,regression-test-issue-3785.js

UBSAN_Tests:
runs-on: ubuntu-latest
env:
Expand All @@ -125,6 +160,10 @@ jobs:
$RUNNER -q --jerry-tests
--buildoptions=--compile-flag=-fsanitize=undefined,--compile-flag=-m32,--compile-flag=-fno-omit-frame-pointer,--compile-flag=-fno-common,--debug,--system-allocator=on,--linker-flag=-fuse-ld=gold
--skip-list=parser-oom.js,parser-oom2.js
- run: >-
$RUNNER -q --jerry-tests --build-debug
--buildoptions=--compile-flag=-fsanitize=undefined,--compile-flag=-m32,--compile-flag=-fno-omit-frame-pointer,--compile-flag=-fno-common,--debug,--system-allocator=on,--linker-flag=-fuse-ld=gold
--skip-list=parser-oom.js,parser-oom2.js

Linux_ARMv7l_Tests:
runs-on: ubuntu-latest
Expand All @@ -139,6 +178,19 @@ jobs:
$RUNNER -q --jerry-tests
--buildoptions=--toolchain=cmake/toolchain_linux_armv7l.cmake,--linker-flag=-static

Linux_ARMv7l_Tests_Debug:
runs-on: ubuntu-latest
env:
RUNTIME: qemu-arm-static
TIMEOUT: 300
steps:
- uses: actions/checkout@v2
- run: sudo apt update
- run: sudo apt install gcc-arm-linux-gnueabihf libc6-dev-armhf-cross qemu-user-static
- run: >-
$RUNNER -q --jerry-tests --build-debug
--buildoptions=--toolchain=cmake/toolchain_linux_armv7l.cmake,--linker-flag=-static

Linux_AArch64_Tests:
runs-on: ubuntu-latest
env:
Expand All @@ -152,6 +204,19 @@ jobs:
$RUNNER -q --jerry-tests
--buildoptions=--toolchain=cmake/toolchain_linux_aarch64.cmake,--linker-flag=-static

Linux_AArch64_Tests_Debug:
runs-on: ubuntu-latest
env:
RUNTIME: qemu-aarch64-static
TIMEOUT: 300
steps:
- uses: actions/checkout@v2
- run: sudo apt update
- run: sudo apt install gcc-aarch64-linux-gnu libc6-dev-armhf-cross qemu-user-static
- run: >-
$RUNNER -q --jerry-tests --build-debug
--buildoptions=--toolchain=cmake/toolchain_linux_aarch64.cmake,--linker-flag=-static

MbedOS5_K64F_Build_Test:
runs-on: ubuntu-18.04 # needed due to ppa:team-gcc-arm-embedded/ppa
steps:
Expand Down
2 changes: 0 additions & 2 deletions tests/test262-es6-excludelist.xml
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,6 @@
<test id="annexB/B.2.3.7.js"><reason></reason></test>
<test id="annexB/B.2.3.8.js"><reason></reason></test>
<test id="annexB/B.2.3.9.js"><reason></reason></test>
<test id="built-ins/decodeURIComponent/S15.1.3.2_A2.5_T1.js"><reason></reason></test>
<test id="built-ins/decodeURI/S15.1.3.1_A2.5_T1.js"><reason></reason></test>
<test id="built-ins/GeneratorPrototype/next/context-constructor-invocation.js"><reason></reason></test>
<test id="built-ins/Promise/all/species-get-error.js"><reason>Promise.all no longer uses @@species</reason></test>
<test id="built-ins/Promise/race/species-get-error.js"><reason>Promise.race no longer uses @@species</reason></test>
Expand Down
46 changes: 18 additions & 28 deletions tools/run-tests.py
Original file line number Diff line number Diff line change
Expand Up @@ -58,59 +58,44 @@ def skip_if(condition, desc):
JERRY_UNITTESTS_OPTIONS = [
Options('unittests-es.next',
OPTIONS_COMMON + OPTIONS_UNITTESTS + OPTIONS_PROFILE_ESNEXT),
Options('unittests-es.next-debug',
OPTIONS_COMMON + OPTIONS_UNITTESTS + OPTIONS_PROFILE_ESNEXT + OPTIONS_DEBUG),
Options('doctests-es.next',
OPTIONS_COMMON + OPTIONS_DOCTESTS + OPTIONS_PROFILE_ESNEXT),
Options('doctests-es.next-debug',
OPTIONS_COMMON + OPTIONS_DOCTESTS + OPTIONS_PROFILE_ESNEXT + OPTIONS_DEBUG),
Options('unittests-es5.1',
OPTIONS_COMMON + OPTIONS_UNITTESTS + OPTIONS_PROFILE_ES51),
Options('unittests-es5.1-debug',
OPTIONS_COMMON + OPTIONS_UNITTESTS + OPTIONS_PROFILE_ES51 + OPTIONS_DEBUG),
Options('doctests-es5.1',
OPTIONS_COMMON + OPTIONS_DOCTESTS + OPTIONS_PROFILE_ES51),
Options('doctests-es5.1-debug',
OPTIONS_COMMON + OPTIONS_DOCTESTS + OPTIONS_PROFILE_ES51 + OPTIONS_DEBUG),
Options('unittests-es5.1-debug-init-fini',
OPTIONS_COMMON + OPTIONS_UNITTESTS + OPTIONS_PROFILE_ES51 + OPTIONS_DEBUG
Options('unittests-es5.1-init-fini',
OPTIONS_COMMON + OPTIONS_UNITTESTS + OPTIONS_PROFILE_ES51
+ ['--cmake-param=-DFEATURE_INIT_FINI=ON'],
skip=skip_if((sys.platform == 'win32'), 'FEATURE_INIT_FINI build flag isn\'t supported on Windows,' +
' because Microsoft Visual C/C++ Compiler doesn\'t support' +
' library constructors and destructors.')),
Options('unittests-es5.1-debug-math',
OPTIONS_COMMON + OPTIONS_UNITTESTS + OPTIONS_PROFILE_ES51 + OPTIONS_DEBUG
Options('unittests-es5.1-math',
OPTIONS_COMMON + OPTIONS_UNITTESTS + OPTIONS_PROFILE_ES51
+ ['--jerry-math=on']),
]

# Test options for jerry-tests
JERRY_TESTS_OPTIONS = [
Options('jerry_tests-es.next-debug',
OPTIONS_COMMON + OPTIONS_PROFILE_ESNEXT + OPTIONS_DEBUG + OPTIONS_STACK_LIMIT + OPTIONS_GC_MARK_LIMIT
Options('jerry_tests-es.next',
Copy link
Contributor Author

@lygstate lygstate Feb 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ossy-szeged Hi, it's this build configuration.
On master, it's only test jerry_tests-es.next-debug, not the jerry_tests-es build configuration
The jerry_tests-es.next build configuration cause the OSX failure.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, good to know that we haven't run esnext release jerry tests on the CI ever. Bad news.
In this case I prefer --skip-list=regression-test-issue-3785.js for OSX release jerry-test command
until #4530 is fixed properly.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I was think it's leaved not running intentionally

OPTIONS_COMMON + OPTIONS_PROFILE_ESNEXT + OPTIONS_STACK_LIMIT + OPTIONS_GC_MARK_LIMIT
+ OPTIONS_MEM_STRESS),
Options('jerry_tests-es5.1',
OPTIONS_COMMON + OPTIONS_PROFILE_ES51 + OPTIONS_STACK_LIMIT + OPTIONS_GC_MARK_LIMIT),
Options('jerry_tests-es5.1-snapshot',
OPTIONS_COMMON + OPTIONS_PROFILE_ES51 + OPTIONS_SNAPSHOT + OPTIONS_STACK_LIMIT + OPTIONS_GC_MARK_LIMIT,
['--snapshot']),
Options('jerry_tests-es5.1-debug',
OPTIONS_COMMON + OPTIONS_PROFILE_ES51 + OPTIONS_DEBUG + OPTIONS_STACK_LIMIT + OPTIONS_GC_MARK_LIMIT
+ OPTIONS_MEM_STRESS),
Options('jerry_tests-es5.1-debug-snapshot',
OPTIONS_COMMON + OPTIONS_PROFILE_ES51 + OPTIONS_SNAPSHOT + OPTIONS_DEBUG + OPTIONS_STACK_LIMIT
+ OPTIONS_GC_MARK_LIMIT, ['--snapshot']),
Options('jerry_tests-es5.1-debug-cpointer_32bit',
OPTIONS_COMMON + OPTIONS_PROFILE_ES51 + OPTIONS_DEBUG + OPTIONS_STACK_LIMIT + OPTIONS_GC_MARK_LIMIT
Options('jerry_tests-es5.1-cpointer_32bit',
OPTIONS_COMMON + OPTIONS_PROFILE_ES51 + OPTIONS_STACK_LIMIT + OPTIONS_GC_MARK_LIMIT
+ ['--cpointer-32bit=on', '--mem-heap=1024']),
Options('jerry_tests-es5.1-debug-external_context',
OPTIONS_COMMON + OPTIONS_PROFILE_ES51 + OPTIONS_DEBUG + OPTIONS_STACK_LIMIT + OPTIONS_GC_MARK_LIMIT
Options('jerry_tests-es5.1-external_context',
OPTIONS_COMMON + OPTIONS_PROFILE_ES51 + OPTIONS_STACK_LIMIT + OPTIONS_GC_MARK_LIMIT
+ ['--external-context=on']),
]

# Test options for test262
TEST262_TEST_SUITE_OPTIONS = [
Options('test262_tests', OPTIONS_PROFILE_ES51),
Options('test262_tests-debug', OPTIONS_PROFILE_ES51 + OPTIONS_DEBUG)
]

# Test options for test262-es2015
Expand All @@ -127,7 +112,7 @@ def skip_if(condition, desc):
# Test options for jerry-debugger
DEBUGGER_TEST_OPTIONS = [
Options('jerry_debugger_tests',
OPTIONS_DEBUG + ['--jerry-debugger=on'])
['--jerry-debugger=on'])
]

# Test options for buildoption-test
Expand Down Expand Up @@ -207,6 +192,8 @@ def get_arguments():
help='Run license check')
parser.add_argument('--check-magic-strings', action='store_true',
help='Run "magic string source code generator should be executed" check')
parser.add_argument('--build-debug', action='store_true',
help='Build debug version jerryscript')
parser.add_argument('--jerry-debugger', action='store_true',
help='Run jerry-debugger tests')
parser.add_argument('--jerry-tests', action='store_true',
Expand Down Expand Up @@ -266,6 +253,10 @@ def report_skip(job):

def create_binary(job, options):
build_args = job.build_args[:]
build_dir_path = os.path.join(options.outdir, job.name)
if options.build_debug:
build_args.extend(OPTIONS_DEBUG)
build_dir_path = os.path.join(options.outdir, job.name + '-debug')
if options.buildoptions:
for option in options.buildoptions.split(','):
if option not in build_args:
Expand All @@ -275,7 +266,6 @@ def create_binary(job, options):
build_cmd.append(settings.BUILD_SCRIPT)
build_cmd.extend(build_args)

build_dir_path = os.path.join(options.outdir, job.name)
build_cmd.append('--builddir=%s' % build_dir_path)

install_dir_path = os.path.join(build_dir_path, 'local')
Expand Down Expand Up @@ -472,7 +462,7 @@ def run_unittests(options):
break

if sys.platform == 'win32':
if "--debug" in job.build_args:
if options.build_debug:
build_config = "Debug"
else:
build_config = "MinSizeRel"
Expand Down
2 changes: 1 addition & 1 deletion tools/runners/test262-harness.py
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@


# The timeout of each test case
TEST262_CASE_TIMEOUT = 5
TEST262_CASE_TIMEOUT = 180


def yaml_load(string):
Expand Down