Skip to content

Modern Testing#2456

Merged
kabeor merged 24 commits intocapstone-engine:nextfrom
Rot127:yaml-testing
Aug 31, 2024
Merged

Modern Testing#2456
kabeor merged 24 commits intocapstone-engine:nextfrom
Rot127:yaml-testing

Conversation

@Rot127
Copy link
Copy Markdown
Collaborator

@Rot127 Rot127 commented Aug 25, 2024 •
edited
Loading

Your checklist for this pull request

  • I've documented or updated the documentation of every API function and struct this PR changes.
  • I've added tests that prove my fix is effective or that my feature works (if possible)

Detailed description

Supersedes #2456

Modernize testing.

The new unit testing is build around yaml files. They can be consumed by the C test engine (cstest) or any bindings easily.

Statistics

The new cstest logs technically two statistics. One by cmocka and one by cstest.
The reason is:

  1. We might want to group tests in the future (by arch or file for example) so we need a global statistics.
  2. Errors of incorrect yaml parsing are not tracked by cmocka.
  3. Also, cmocka doesn't provide the detailed statistics to us. So we cannot print a nice summary at the end.

Changes

  • New cstest C-implementation
  • New py_cstest Python-implementation
  • Moves almost all tests to yaml files.
  • Old tests which can't be moved (because they are effectively integration tests) are kept.
  • Add numerous test jobs in the workflow
  • Adds several CMake test targets
  • Add LoongArch Python bindings
  • Several bug fixes discovered during testing.

Current test coverage of Auto-Sync <ARCH>Mapping.c files by running cstest test/:

file coverage covered
arch/LoongArch/LoongArchMapping.c 67.29% 144 / 214
arch/PowerPC/PPCMapping.c 72.78% 254 / 349
arch/AArch64/AArch64Mapping.c 60.47% 918 / 1518
arch/TriCore/TriCoreMapping.c 45.00% 45 / 100
arch/Alpha/AlphaMapping.c 86.08% 68 / 79
arch/ARM/ARMMapping.c 61.68% 792 / 1284
arch/HPPA/HPPAMapping.c 29.85% 20 / 67

After merge

  • Add windows build to CI

Test plan

Added and all green

Closing issues

closes #1984
closes #2405
closes #2383
closes #2383

Rot127 added 8 commits August 25, 2024 05:34
@Rot127
Replace old cstest with new implementation.
4bf76ff
@Rot127
Remove and partially move legacy tests.
6a0b2cb 
- Not removed are the legacy integration tests.
- Detail and asm tests are now in the yaml test files.
@Rot127
Rewrite MCUpdater
cb87af1 
The MCUpdater is completely rewritten to generate yaml test files
from the LLVM MC tests.

Also the compatibility header got a test case in CMake.
@Rot127
Add new tests files.
4979f80
@Rot127
Remove unused files.
349fa34
@Rot127
Fix several bugs and clang-tidy warnings detected by the new tests.
7119e07
@Rot127
Add py_cstest and LoongArch Python bindings.
17ce43d
@Rot127
Update docs and Github workflows.
c423c60
@Rot127 Rot127 force-pushed the yaml-testing branch 3 times, most recently from c5bd850 to c9c98ad Compare August 26, 2024 11:00
@Rot127 Rot127 force-pushed the yaml-testing branch 3 times, most recently from 864c9dd to d5bdcba Compare August 26, 2024 11:45
kabeor
kabeor reviewed Aug 27, 2024
View reviewed changes
Comment thread .github/workflows/CITest.yml Outdated
@Rot127
Copy link
Copy Markdown
Collaborator Author

Rot127 commented Aug 28, 2024

@kabeor I fixed the Auto-Sync workflow. Please take another look.

@Rot127 Rot127 mentioned this pull request Aug 28, 2024
Merged
2 tasks
Rot127 added 2 commits August 29, 2024 06:22
@Rot127
Renaming py_cstest -> cstest_py
114df29
@Rot127
Don't generate tests from assembly tests.
f37e676 
We cannot toggle between alias or non aias assembly currently.
So valid tests fail because the non alias asm text is compared.
@Rot127 Rot127 mentioned this pull request Aug 31, 2024
Merged
2 tasks
kabeor
kabeor approved these changes Aug 31, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@kabeor kabeor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool!

@kabeor kabeor merged commit 191db14 into capstone-engine:next Aug 31, 2024
@Rot127 Rot127 deleted the yaml-testing branch September 1, 2024 09:19
This was referenced Sep 3, 2024
Closed
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kabeor kabeor kabeor approved these changes

+1 more reviewer

@XVilka XVilka XVilka requested changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

AArch64 Arch Alpha Arch ARM Arch Auto-Sync-files Auto-Sync BPF Arch CS-core-files auto-sync Documentation EVM Arch Github-files Github related files HPPA Arch LLVM-generated-files LoongArch Arch M68K Arch M680X Arch Mips Arch MOS65XX Arch PowerPC Arch Python Bindings RISCV Arch SH Arch Sparc Arch SystemZ Arch TMS320C64x Arch TriCore Arch WASM Arch X86 Arch XCore Arch

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

test_riscv.py failing test_tms320c64x.c does not abort when not enabled Modernize Capstone testing

3 participants

@Rot127 @XVilka @kabeor