Python package building rework

[Antelox]

Your checklist for this pull request

• I've documented or updated the documentation of every API function and struct this PR changes.
• I've added tests that prove my fix is effective or that my feature works (if possible)

Detailed description

• Refactored setup.py to remove hacks regarding packaging of wheels for different platforms, improve and cleanup the code
• Updated README.txt
• Removed old Makefile and build_wheel.sh scripts
• Created a new workflow that takes care of building and testing python packages for different platforms/architectures/python versions
• Fixed cstest_py positional argument
• Fixed root path detection in integration_tests.py
• Added run_tests.py script to run all the python unittests
• Added options to specify build type and threads to be used during building
• Refreshed the list of files to be copied into src folder for sdist archive

Test plan

There are 6 tests failing for both Linux and Windows x86 arch.

INFO  - Run test: TestInput { arch: mos65xx, options: ['CS_OPT_DETAIL', 'CS_MODE_MOS65XX_W65C02', 'CS_OPT_SYNTAX_MOTOROLA'], addr: 1000, bytes: [ 0x07,0x12,0x27,0x12,0x47,0x12,0x67,0x12,0x87,0x12,0xa7,0x12,0xc7,0x12,0xe7,0x12,0x10,0xfe,0x0f,0x12,0xfd,0x4f,0x12,0xfd,0x8f,0x12,0xfd,0xcf,0x12,0xfd ] }
WARNING - CS_ARCH: Capstone doesn't have the attribute 'mos65xx'
ERROR - Normalized asm-text doesn't match:
decoded:  'bbr0 $12, (null)f0fb60df'
expected: 'bbr0 $12, $1012'

INFO  - TestResult.FAILED

INFO  - Run test: TestInput { arch: mos65xx, options: ['CS_OPT_DETAIL', 'CS_MODE_MOS65XX_65816_LONG_MX', 'CS_OPT_SYNTAX_MOTOROLA'], addr: 1000, bytes: [ 0xa9,0x34,0x12,0xad,0x34,0x12,0xbd,0x34,0x12,0xb9,0x34,0x12,0xaf,0x56,0x34,0x12,0xbf,0x56,0x34,0x12,0xa5,0x12,0xb5,0x12,0xb2,0x12,0xa1,0x12,0xb1,0x12,0xa7,0x12,0xb7,0x12,0xa3,0x12,0xb3,0x12,0xc2,0x00,0xe2,0x00,0x54,0x34,0x12,0x44,0x34,0x12,0x02,0x12 ] }
ERROR - Normalized asm-text doesn't match:
decoded:  'mvn $12, (null)f0fb60df'
expected: 'mvn $12, $34'

INFO  - TestResult.FAILED

INFO  - Run test: TestInput { arch: CS_ARCH_ARM, options: ['CS_MODE_THUMB', 'CS_OPT_DETAIL'], addr: 0, bytes: [ 0x18,0xf8,0x03,0x1e ] }
ERROR - regs_write_count: 2 != 1
INFO  - TestResult.FAILED

INFO  - Run test: TestInput { arch: CS_ARCH_ARM, options: ['CS_MODE_THUMB', 'CS_MODE_MCLASS', 'CS_OPT_DETAIL'], addr: 0, bytes: [ 0x98,0xfc,0x4e,0x08 ] }
ERROR - imm: 4029476313 != 90
INFO  - TestResult.FAILED

INFO  - Run test: TestInput { arch: CS_ARCH_ARM, options: ['CS_MODE_THUMB', 'CS_MODE_V8', 'CS_OPT_DETAIL'], addr: 0, bytes: [ 0x94,0xfd,0x46,0x48 ] }
ERROR - imm: 4029476493 != 270
INFO  - TestResult.FAILED

Closing issues

I suppose it should close:

• closes Clean up or rework Python package building #2445
• closes Missing pkgconfig.mk and .in files in v6.0.0a1 sdist archive #2548
• closes Failed to install from pip3 #1701
• closes python bindings seem to install deprecated egg distribution instead of wheel #1292
• closes install capstone in windows. create capstone.lib search for capstone.dll #1796

[Rot127]

Looks way way cleaner! Thanks a lot.
I start the CI once for all the jobs. But in general we have to discuss with @kabeor about the possible CI runtimes with this one.

Also, do I see this right? You test only on 3.8 and 3.9 here?
Testing on 3.8 + 3.13 (or whatever is the oldest/newest) is better IMHO.

[Rot127]

The failing tests are possibly related to #2416. Both are clearly 32bit system issues.

[Rot127]

Very nice! Thanks a lot for the edits. Just some more small things please.

Also please do a release in your fork on this branch. So we can test if the release workflow works.

Only the actual publish/pypip upload step should fail.

Thanks a lot for cleaning up all this.

[Rot127]

@kabeor This PR opens the question if Capstone supports x86 systems. The tests fail on 32bit systems (at least Windows). I think it is a good idea to support it. Because there are also modern 32bit CPU architectures out there. So having at least a single x86 job is good reduces the chance of bugs also on the other archtectures.

[Rot127]

Something is still wrong with the naming:
https://github.com/capstone-engine/capstone/actions/runs/11813575581/job/32923353685?pr=2538

[Antelox]

Something is still wrong with the naming: https://github.com/capstone-engine/capstone/actions/runs/11813575581/job/32923353685?pr=2538

That is ugly I know, but it's like that because that job is skipped hence the matrix doesn't get created and finally the placeholders don't get replaced

[Rot127]

This one is weird: https://github.com/capstone-engine/capstone/actions/runs/11818841771/job/32927697780?pr=2538
Happens also for this job. Let's see after the review of the others if it is fixed.

[Rot127]

Last little nitpick. Then it looks good to me.

I check the build wheels later this day. And will check a last time after kabeor is done.

[Antelox]

This one is weird: https://github.com/capstone-engine/capstone/actions/runs/11818841771/job/32927697780?pr=2538 Happens also for this job. Let's see after the review of the others if it is fixed.

Should be internal GitHub networking issue

[Rot127]

@aquynh Can you please check how many runner minutes are used currently?
According to these Github docs all latest Linux, Windows and MacOS runners (even an M1 instance) are free.

I can't see if this is also the case for all architectures (like Linux + AArch64, Windows + AArch64 or Windows i686). But if all those are free, we should absolutely enable some more runners.

[Antelox]

I can't see if this is also the case for all architectures (like Linux + AArch64, Windows + AArch64 or Windows i686). But if all those are free, we should absolutely enable some more runners.

Those are not native runners so it's not the case.
Here you can check what are the specs of the runners. Basically all are x64 plus macos-latest one that is AppleSilicon (ARM).

[Rot127]

@kabeor The run in case of tag creation: https://github.com/Antelox/capstone/actions/runs/11839185045

[Rot127]

Please write in the commit message what the fix was for (next time).
Also, provide a link to the Action run of a release with the latest commit. So kabeor can quickly check.

[Rot127]

Binaries look good as well:

for f in $(ls ./*/capstone/lib/*); do file $f; done
./capstone-6.0.0a1-cp310-cp310-macosx_10_9_universal2/capstone/lib/libcapstone.dylib: Mach-O universal binary with 2 architectures: [x86_64:\012- Mach-O 64-bit x86_64 dynamically linked shared library, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|NO_REEXPORTED_DYLIBS>] [\012- arm64]
./capstone-6.0.0a1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64/capstone/lib/libcapstone.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, BuildID[sha1]=d139511e718f699bc4ff70698f6e5cd7d2c9d26d, not stripped
./capstone-6.0.0a1-cp310-cp310-win_amd64/capstone/lib/capstone.dll: PE32+ executable (DLL) (GUI) x86-64, for MS Windows, 6 sections
./capstone-6.0.0a1-cp310-cp310-win_arm64/capstone/lib/capstone.dll: PE32+ executable (DLL) (GUI) Aarch64, for MS Windows, 6 sections
./capstone-6.0.0a1-cp39-cp39-macosx_10_9_x86_64/capstone/lib/libcapstone.dylib: Mach-O 64-bit x86_64 dynamically linked shared library, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|NO_REEXPORTED_DYLIBS>
./capstone-6.0.0a1-cp39-cp39-macosx_11_0_arm64/capstone/lib/libcapstone.dylib: Mach-O 64-bit arm64 dynamically linked shared library, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|NO_REEXPORTED_DYLIBS>
./capstone-6.0.0a1-cp39-cp39-musllinux_1_2_x86_64/capstone/lib/libcapstone.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=fd9af6ba25a55df236d41a04d6d4776f7b585cab, with debug_info, not stripped
./capstone-6.0.0a1-cp39-cp39-win_amd64/capstone/lib/capstone.dll: PE32+ executable (DLL) (GUI) x86-64, for MS Windows, 6 sections

[kabeor]

LGTM, Thank you!