Building python bindings on `next` doesn't seem to use correct dylib

[rchtsang]

See #1935 for original issue, whose underlying issue was resolved in #1949.

On Mac M1 Pro, macOS Sonoma 14.2.1.

Python 3.11, next branch @ 215eae8

The original issue was that dissassembling Thumb branch/call immediate instructions did not calculate the correct immediate value.

For instance, in the following example:

>>> from capstone import *
>>> from capstone.arm_const import *
>>> cs = Cs(CS_ARCH_ARM, CS_MODE_THUMB)
>>> insn = next(cs.disasm(b"\xff\xf7\xad\xff", 4))
>>> insn
<CsInsn 0x4 [fff7adff]: bl #0xffffff62>

The correct result (as can be verified with cstool) is 0xffffff5e:

$ cstool -d thumb "\xff\xf7\xad\xff"
 0  ff f7 ad ff  bl	#0xffffff5e
	ID: 16 (bl)
	op_count: 1
		operands[0].type: IMM = -0xa2
	Registers read: pc
	Registers modified: lr pc
	Groups: branch_relative thumb call jump 

Since cstool works, it would seem that there is an issue with building the python bindings, though I haven't managed to locate it.

Seems like it might be related to and/or solved by #2223, though I was using 3.11 to avoid the issues mentioned there.

Building python bindings following these steps:

$ mamba create -n tmp-env # create a virtual environment to install capstone in
$ mamba activate tmp-env
$ mamba install -c conda-forge python=3.11 pip -y
$ cd capstone
$ ./make.sh # build libcapstone.5.dylib
$ cd bindings/python
$ sudo make clean
$ sudo make install3 # build and install python bindings into active pip

Which puts capstone 5.0.0.post1 in my pip environment (which should ostensibly not matter, because the original issue should've been fixed with v5).

[peace-maker]

Watch out for having capstone installed multiple times. Once system-wide and once for the user. The user install might still use the system libraries. Check which path was used to load the library by the capstone._path variable and compare it to where make install3 copied the bindings to. You might have to uninstall it multiple times using [sudo] pip before installing from source.

import capstone
print('python code:', capstone.__path__)
print(capstone.debug())
print('c library path:', capstone._path)

[Rot127]

@rchtsang Could you check if it is due to multiple Capstone installations?

[rchtsang]

Sorry, yes I just checked.

Running from a conda environment:

$ cd bindings/python
$ make install3
$ pip list
DEPRECATION: Loading egg at /opt/homebrew/Caskroom/miniforge/base/envs/capstone/lib/python3.11/site-packages/capstone-5.0.0.post1-py3.11.egg is deprecated. pip 24.3 will enforce this behaviour change. A possible replacement is to use pip for package installation.. Discussion can be found at https://github.com/pypa/pip/issues/12330
Package    Version
---------- -----------
capstone   5.0.0.post1
capstone   5.0.0.post1
capstone   5.0.0.post1
pip        24.0
setuptools 69.1.0
wheel      0.42.0

then checking the installation as follows:

>>> import capstone
>>> print(capstone.__path__)
['/Users/rtsang/Documents/Research/projects/embedded-emu/capstone/bindings/python/capstone']
>>> print(capstone.debug())
python-standard-aarch64-alpha-arm-bpf-evm-m680x-m68k-mips-mos65xx-ppc-riscv-sh-sparc-sysz-tms320c64x-tricore-wasm-xcore-x86-c5.0-b5.0
>>> print(capstone._path)
/Users/rtsang/Documents/Research/projects/embedded-emu/capstone/bindings/python/capstone/lib

The capstone library path is pointing to .../bindings/python/capstone/lib/libcapstone.dylib.

From what I can tell, this path should be right, but diff-ing the dylib file from the built version does show that it is different:

$ diff capstone/lib/libcapstone.dylib ../../libcapstone.5.dylib 
Binary files capstone/lib/libcapstone.dylib and ../../libcapstone.5.dylib differ

Though admittedly, there could be any number of reasons for this since idk if the binary changes when copied.

So I don't think I have a multiple install problem.

I do have capstone also installed via brew, but that is the latest version and does not have the problem I was seeing with #1935 as I am still getting when I install to virtual environment using make install3.

I did manage to manually specify the correct dylib for make install3 by copying my locally built libcapstone.5.dylib into bindings/python/prebuilt after looking through the build_libraries function of setup.py:

>>> from capstone import *
>>> from capstone.arm_const import *
>>> cs = Cs(CS_ARCH_ARM, CS_MODE_THUMB)
>>> insn = next(cs.disasm(b"\xff\xf7\xad\xff", 0))
>>> insn
<CsInsn 0x0 [fff7adff]: bl 0xffffff5e>

However, when it is not specified, the default dylib being built or used by build_libraries is different and I can't tell why at first glance.

[Rot127]

Did you installed the v5.0.1 with brew? Or the head? Because #1949 is only in the next branch.

[rchtsang]

when I installed to the conda environment it was with next, I don't think the brew dylibs are being used.

[rchtsang]

Ok, huh, odd. I just tried sudo make install3 instead of just make install3 and it seemed to work. Maybe that was in the build instructions somewhere and I missed it, but then I don't know why it would still build a version of the bindings without sudo. Plus I'd really rather not have to use sudo if I don't need to.

Also, this may need it's own issue, but when it's installed to pip, it shows up 3 times on pip list for whatever reason:

$ sudo make install3
$ pip list
DEPRECATION: Loading egg at /opt/homebrew/Caskroom/miniforge/base/envs/capstone/lib/python3.11/site-packages/capstone-5.0.0.post1-py3.11.egg is deprecated. pip 24.3 will enforce this behaviour change. A possible replacement is to use pip for package installation.. Discussion can be found at https://github.com/pypa/pip/issues/12330
Package    Version
---------- -----------
capstone   5.0.0.post1
capstone   5.0.0.post1
capstone   5.0.0.post1
pip        24.0
setuptools 69.1.0
wheel      0.42.0

[Rot127]

Yeah, the build instructions need some love as well. But there are a few things which need some care, when it comes to the Python bindings.

Is this issue solved with this?
We will release v5.0.2 (hopefully) soon, and with it, the fix of the original problem should be part of the release version.
Can you please close the issue if there is nothing else.