Change CI to create Debian Package to Release#2521
Change CI to create Debian Package to Release#2521kabeor merged 2 commits intocapstone-engine:nextfrom AndrewQuijano:debian
Conversation
Rot127
left a comment
There was a problem hiding this comment.
Awesome! Please address the comments. We can test it by making a release in our own forks. And download and install the artifacts.
I think it should contain the static and shared lib file. You might need to build Capstone twice with cmake. Once for the static, once for creating the shared one. |
|
Current diff looks like this, I suspect I can ignore the /sys stuff and I suspect I can ignore the stuff under /usr/local/lib/cmake/ |
|
Oh and @Rot127, assuming all this works as intended, could you make a capstone 5.0.3 Debian package or a 5.0.4 Debian package? The tool I use PANDA, for now, seems to only support version 5. The goal is to replace this with a one line to download the Debian package and install it. Essentially, I do need a libcapstone.so.5 for the current capstone version https://github.com/panda-re/panda/blob/dev/Dockerfile#L37-L41 |
|
Actually, I released I can test this on my own fork. Just be careful, you can only create tags say 5.0.4. If you have letters, think the debian part will break, just as a heads up. Works on 1.5.8 but not say v1.5.5 https://github.com/AndrewQuijano/capstone/releases/tag/1.5.8 |
|
Thanks for all the work!
This is very good to know. Can you may lint the version string given to the script?
We just need a way to have this checked, handled and documented in code. Because otherwise it will absolutely be forgotten at some point. |
Yes, we can release a new v5 version. Was on the list anyways. Just waiting for the TriCore bug to be resolved. |
There was a problem hiding this comment.
Thanks a lot for the corrections. I spent way more time now on checking everything.
I ran it manually on a Debian VM and some stuff broke. Package building worked fine (as root: ./setup.sh 6.0.0).
But there are a few problems:
The capstone.pc in the package is:
# After unpacking capstone.deb
> cat capstone/data/usr/local/lib/pkgconfig/capstone.pc
prefix=/usr/local
exec_prefix=${prefix}
libdir=${prefix}/lib
includedir=${prefix}/include
Name: capstone
Description: Capstone disassembly engine
Version: 6.0.
URL: https://www.capstone-engine.org/
archive=${libdir}/libcapstone.a
Libs: -L${libdir} -lcapstone
Libs.private: -L${libdir} -l:libcapstone.a
Cflags: -I${includedir} -I${includedir}/capstone
archs=- The version number is not correct, it should end with a
0. But ends with a.. archsis empty. I think it should bearch(nos) and be set tox86_64? But couldn't find docs about it. Just remove it, if you can't find an answer.libcapstone.ais not in the package. I think it is best, if you just fixCMakeLists.txtso it always compiles the static lib. Here the fix:diff --git a/CMakeLists.txt b/CMakeLists.txt index 345740c9..b8b01be6 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -761,16 +761,23 @@ set(ALL_HEADERS set_property(GLOBAL PROPERTY VERSION ${PROJECT_VERSION}) ## targets -add_library(capstone ${ALL_SOURCES} ${ALL_HEADERS}) +add_library(capstone OBJECT ${ALL_SOURCES} ${ALL_HEADERS}) add_library(capstone::capstone ALIAS capstone) +add_library(capstone_static STATIC $<TARGET_OBJECTS:capstone>) +# Use normal capstone name. Otherwise we get libcapstone_static.so +set_target_properties(capstone_static PROPERTIES OUTPUT_NAME "capstone") target_include_directories(capstone PUBLIC $<BUILD_INTERFACE:${PROJECT_SOURCE_DIR}/include> ) set_property(TARGET capstone PROPERTY C_STANDARD 99) if(BUILD_SHARED_LIBS) - target_compile_definitions(capstone PUBLIC CAPSTONE_SHARED) - set_target_properties(capstone PROPERTIES + set_property(TARGET capstone PROPERTY POSITION_INDEPENDENT_CODE 1) + add_library(capstone_shared SHARED $<TARGET_OBJECTS:capstone>) + # Use normal capstone name. Otherwise we get libcapstone_shared.so + set_target_properties(capstone_shared PROPERTIES OUTPUT_NAME "capstone") + target_compile_definitions(capstone_shared PUBLIC CAPSTONE_SHARED) + set_target_properties(capstone_shared PROPERTIES VERSION ${PROJECT_VERSION} SOVERSION ${PROJECT_VERSION_MAJOR} )
cstooldoesn't work after installing the package, becauseldconfigis not run (and/usr/local/libis not in the default lib search path). This seems to be the way to runldconfigafter install.- Please add a test script which installs the package and checks everything is correct (version number, libs are there, libs report the same version, cstool works etc). It would be good to have this in a separated job.
Sorry for being super nit-picky with this one. We had quite some problems with the Python release packages. And it is hard to fix, because we need to do a fix-release after changing it. Very stressful experience. This is why I ask you for so much stuff.
|
Hello, |
|
Oups, forgot to update the install targets: diff --git a/CMakeLists.txt b/CMakeLists.txt
index 345740c9..c8c10880 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -878,7 +885,13 @@ if(CAPSTONE_INSTALL)
DESTINATION ${CAPSTONE_CMAKE_CONFIG_INSTALL_DIR}
)
- install(TARGETS capstone
+ if(BUILD_SHARED_LIBS)
+ set(LIB_INSTALL_TARGETS capstone_static capstone_shared)
+ else()
+ set(LIB_INSTALL_TARGETS capstone_static)
+ endif()
+
+ install(TARGETS ${LIB_INSTALL_TARGETS}
EXPORT capstone-targets
RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}
LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR} |
|
OK, I just need to fix capstone.pc version and removing arch and I am done, but in the meantime please test what I got so far |
|
The workflow fails in my test release: |
|
Numbers seem fine. I think it is because it passes |
|
yeah just patched it |
|
Oh I forgot to pull out debian package because I was debugging the library issue. This should work end to end now |
|
I need to go. Will take a look at it tomorrow again. Thanks for fixing it so quickly. |
|
No worries! Thanks for your fast responses! Really look forward to updating PANDA with the package. Also QQ where did you find buster-slim is EOL? Figure I might as well update PANDA's Debian package thing too, but figure I want to to document it in the PR |
Debian Buster is EOL since two years. Hence we don't have to build on it. This is also likely the reason you could not install cmake via apt. |
|
Clear up usage of static libraries.
Install cmake via apt. |
|
@Rot127 just accepted your PR |
|
Thanks for your help with this! Just curious, when would you expect v 5.0.4 of capstone Debian Package be ready by? Figure I can start working on the PR for PANDA to use the debian package. I'm wondering if there might come a day you might have an apt repository? https://earthly.dev/blog/creating-and-hosting-your-own-deb-packages-and-apt-repo/ |
|
Nice! Lgtm. I fix the failing test tomorrow. |
It is not planned. We need maintainers for this. And we are already too busy with all the other stuff to do it on our own.
#2523 is the PR which has to be done. I review it tomorrow as well. Then we need to merge it into v5. |
|
btw I noticed it still says request changes blocking the merge:
|
Rot127
left a comment
There was a problem hiding this comment.
Please just address the last two small things.
Do you still want me to remove the 'archs' from the pkconfig?
Actually yeah. It is not used outside of the Make build. It was added to have a list of supported architectures. But this information doesn't belong there. And I couldn't find documentation about this archs field. So it is likely not standard anyways.
Safe to say the capstone debian package worked without needing ldconfig?
At least in my fresh Debian VM it didn't work otherwise. Feel free to test it again.
|
@Rot127 Just made your suggested changes, let me know about if the ldconfig works/the package installation works as intended. Look forward to merging the PR! |
version. Also updating release CI to not use end-of-life workflows
- Python bindings only use the dynamic lib. But built and copied the static ones sometimes nonetheless. - Add toggles to build only static, static/dyn or only dynamic.
|
@Rot127, let me know when this PR is to be merged. I made one last change, and provided more context in #2398. I changed the package name from 'capstone.deb' to 'libcapstone-dev.deb' so it matches the name of the apt-package. Just to confirm, this debian package essentially would be both libcapstone5.deb and libcapstone-dev.deb package from apt? I suspect the answer is yes given that both library file is added, and the /usr/local/include is populated with headers. And please confirm the package works as expected. Essentially, PANDA would have required both the capstone library, and does rely on capstone for development. |
There was a problem hiding this comment.
I am sorry, I thought I already approved this one. But I can't merge anyways. The review of @kabeor is required. I already reminded him a few days ago.
libcapstone5.deb and libcapstone-dev.deb package from apt?
Yeah, you are right. We should have a libcapstone6 (not libcapstone5) and libcapstone-dev package. But let's keep the PR like this for now please. Because it is already large enough.
Also, as long as we have not submitted it to the Debian/Ubuntu repos, we don't need the distinction yet.
If you want to take care of it, feel free to open a new PR of course.
And please confirm the package works as expected.
Works for me! Tested it with a Whonix VM. See https://github.com/Rot127/capstone/releases/tag/v99.99.103
* Update changelog for V6.0.0-Alpha1 (#2493) * update version to v6-alpha1 * update bindings const values * Update changelog for V6.0.0-Alpha1 * Remove irrelevant changes. (#2495) * Fixing UB santizer, `LITBASE` and assert errors. (#2499) * Update labeler with Xtensa and v6 files. (#2500) * Add hard asserts to all SStream functions and memset MCInst. (#2501) * Only trigger on released action. (#2497) * Fix cstest build with Ninja (#2506) * Tricore EA calculation (#2504) * Update libcyaml dependency in cstest to 1.4.2 (#2508) * AArch64: Replace vararg add_cs_detail by multiple concrete functions Fixes UB caused by various mismatches on how these arguments are passed and read. This became visible when running on PowerPC hosts with e.g. `cstool -d aarch64 204862f8`. Apart from the UB fix, this is meant to be a pure refactor. Partially addresses #2458 * xtensa: Fix Branch Target (#2516) * xtensa: Fix Branch Target * auto-sync: fix byte pattern * xtensa: add branch insn tests * Revert "auto-sync: fix byte pattern" This reverts commit cf8e870. * Fix #2509. (#2510) Compatibility headers should always include the header in the same dir. * Fix stringop-truncation warning some compilers raise. (#2522) * Add CC and VAS compatibility macros (#2525) * Fix endianess issue during assignment. (#2528) * This time actually fix big endian issue. (#2530) * tricore: fixes #2474 (#2523) * tricore: fix auto-sync tricore * tricore: fixes TriCoreGenCSMappingInsnName.inc * tricore: fixes * tricore: try fix ld.a SC * tricore: fixes all * Add TriCore to .github/workflows/auto-sync.yaml * Add TriCore details tests(a15, d15, a10|sp) * Change CI to create Debian Package to Release (#2521) * Updating CI to create Debian package and version is assigned by tag version. Also updating release CI to not use end-of-life workflows * Clear up usage of static libraries. - Python bindings only use the dynamic lib. But built and copied the static ones sometimes nonetheless. - Add toggles to build only static, static/dyn or only dynamic. --------- Co-authored-by: Rot127 <unisono@quyllur.org> * Rename build arguments: (#2534) - BUILD_SHARED_LIBS -> CAPSTONE_BUILD_SHARED_LIBS - BUILD_STATIC_LIBS -> CAPSTONE_BUILD_STATIC_LIBS - BUILD_STATIC_LIBS -> CAPSTONE_BUILD_STATIC_MSVC_RUNTIME * xtensa: update to espressif/llvm-project (#2533) * fix coverity (#2546) - cid 514642 - cid 514643 - cid 514644 - cid 514645 * Move debian package generation to a dispatch only workflow (#2543) * Move deb package gen files int package/deb * Fix basename check * Make debian package generation dispatch only * Python package building rework (#2538) * - Refactored setup.py to remove hacks regarding packaging of wheels for different platforms, improve and cleanup the code - Updated README.txt - Removed old Makefile and build_wheel.sh scripts - Created a new workflow that takes care of building and testing python packages for different platforms/architectures/python versions * Added SPDX headers to the setup.py * - cstest_py: Fixed positional argument since it doesn't accept a `required` flag. It turns to have a mandatory tests folder path - integration_tests.py: Use pathlib to determine the required path - GitHub action: Simplified the tests execution command * GitHub Actions: Run python 3.8 (lowest) and 3.13 (current highest) for native runners only during testings and the rest during tag release * GitHub Action: - Fixed the cibw_build matrix element - Added a step to prepare artifact name * GitHub Action: Added run_tests.py script to run all tests during CI workflow * - Added SPDX headers to the run_tests.py script and to the build-wheels-publish.yml workflow file - Minor fixes to the workflow as pointed out in the PR review - Updated MANIFEST.in to reflect the actual libraries built during python wheel creation process - Use subprocess.run in place of os.system in run_tests.py script * GitHub Action: - Run qemu step only if non-native Linux runner - Added arch:universal2 matrix element for macos-latest runner * Python bindings: Refreshed the list of files needed to be copied for sdist archive * GitHub Action: Commented out arch:x86 matrix elements * GitHub Action: Run qemu step only if non-native Linux runner * GitHub Action: Minor fixes * Python bindings: Added missing .in pattern when collecting src files for sdist archive * Auto-Sync reproducability + ARM update (#2532) * fix xtensa DecodeMR23RegisterClass and add tests for MAC16 instru… (#2551) * fix xtensa `DecodeMR23RegisterClass` and add tests for `MAC16` instructions * revert * Prepare for update (#2552) * Bindings(chore): Fix DeprecationWarning * Version(upgrade): update bindings const * Fix(chore): Fix ARMCC_Invalid is not defined * Update Changelog Version to 6.0.0-Alpha2 (#2553) * Bindings(chore): Fix DeprecationWarning * Version(upgrade): update bindings const * Fix(chore): Fix ARMCC_Invalid is not defined * Changelog: Update to version 6.0.0-Alpha2 --------- Co-authored-by: Rot127 <45763064+Rot127@users.noreply.github.com> Co-authored-by: Florian Märkl <info@florianmaerkl.de> Co-authored-by: billow <billow.fun@gmail.com> Co-authored-by: Andrew <afq2101@columbia.edu> Co-authored-by: Rot127 <unisono@quyllur.org> Co-authored-by: @Antelox <anteloxrce@gmail.com>
* Update changelog for V6.0.0-Alpha1 (capstone-engine#2493) (capstone-engine#2494) * update version to v6-alpha1 * update bindings const values * Update changelog for V6.0.0-Alpha1 * Remove irrelevant changes. (capstone-engine#2496) * Fixing UB santizer, `LITBASE` and assert errors. (capstone-engine#2499) * Update labeler with Xtensa and v6 files. (capstone-engine#2500) * Add hard asserts to all SStream functions and memset MCInst. (capstone-engine#2501) * Only trigger on released action. (capstone-engine#2497) * Fix cstest build with Ninja (capstone-engine#2506) * Tricore EA calculation (capstone-engine#2504) * Update libcyaml dependency in cstest to 1.4.2 (capstone-engine#2508) * AArch64: Replace vararg add_cs_detail by multiple concrete functions Fixes UB caused by various mismatches on how these arguments are passed and read. This became visible when running on PowerPC hosts with e.g. `cstool -d aarch64 204862f8`. Apart from the UB fix, this is meant to be a pure refactor. Partially addresses capstone-engine#2458 * xtensa: Fix Branch Target (capstone-engine#2516) * xtensa: Fix Branch Target * auto-sync: fix byte pattern * xtensa: add branch insn tests * Revert "auto-sync: fix byte pattern" This reverts commit cf8e870. * Fix capstone-engine#2509. (capstone-engine#2510) Compatibility headers should always include the header in the same dir. * Fix stringop-truncation warning some compilers raise. (capstone-engine#2522) * Add CC and VAS compatibility macros (capstone-engine#2525) * Fix endianess issue during assignment. (capstone-engine#2528) * This time actually fix big endian issue. (capstone-engine#2530) * tricore: fixes capstone-engine#2474 (capstone-engine#2523) * tricore: fix auto-sync tricore * tricore: fixes TriCoreGenCSMappingInsnName.inc * tricore: fixes * tricore: try fix ld.a SC * tricore: fixes all * Add TriCore to .github/workflows/auto-sync.yaml * Add TriCore details tests(a15, d15, a10|sp) * Change CI to create Debian Package to Release (capstone-engine#2521) * Updating CI to create Debian package and version is assigned by tag version. Also updating release CI to not use end-of-life workflows * Clear up usage of static libraries. - Python bindings only use the dynamic lib. But built and copied the static ones sometimes nonetheless. - Add toggles to build only static, static/dyn or only dynamic. --------- Co-authored-by: Rot127 <unisono@quyllur.org> * Rename build arguments: (capstone-engine#2534) - BUILD_SHARED_LIBS -> CAPSTONE_BUILD_SHARED_LIBS - BUILD_STATIC_LIBS -> CAPSTONE_BUILD_STATIC_LIBS - BUILD_STATIC_LIBS -> CAPSTONE_BUILD_STATIC_MSVC_RUNTIME * xtensa: update to espressif/llvm-project (capstone-engine#2533) * fix coverity (capstone-engine#2546) - cid 514642 - cid 514643 - cid 514644 - cid 514645 * Move debian package generation to a dispatch only workflow (capstone-engine#2543) * Move deb package gen files int package/deb * Fix basename check * Make debian package generation dispatch only * Python package building rework (capstone-engine#2538) * - Refactored setup.py to remove hacks regarding packaging of wheels for different platforms, improve and cleanup the code - Updated README.txt - Removed old Makefile and build_wheel.sh scripts - Created a new workflow that takes care of building and testing python packages for different platforms/architectures/python versions * Added SPDX headers to the setup.py * - cstest_py: Fixed positional argument since it doesn't accept a `required` flag. It turns to have a mandatory tests folder path - integration_tests.py: Use pathlib to determine the required path - GitHub action: Simplified the tests execution command * GitHub Actions: Run python 3.8 (lowest) and 3.13 (current highest) for native runners only during testings and the rest during tag release * GitHub Action: - Fixed the cibw_build matrix element - Added a step to prepare artifact name * GitHub Action: Added run_tests.py script to run all tests during CI workflow * - Added SPDX headers to the run_tests.py script and to the build-wheels-publish.yml workflow file - Minor fixes to the workflow as pointed out in the PR review - Updated MANIFEST.in to reflect the actual libraries built during python wheel creation process - Use subprocess.run in place of os.system in run_tests.py script * GitHub Action: - Run qemu step only if non-native Linux runner - Added arch:universal2 matrix element for macos-latest runner * Python bindings: Refreshed the list of files needed to be copied for sdist archive * GitHub Action: Commented out arch:x86 matrix elements * GitHub Action: Run qemu step only if non-native Linux runner * GitHub Action: Minor fixes * Python bindings: Added missing .in pattern when collecting src files for sdist archive * Auto-Sync reproducability + ARM update (capstone-engine#2532) * fix xtensa DecodeMR23RegisterClass and add tests for MAC16 instru… (capstone-engine#2551) * fix xtensa `DecodeMR23RegisterClass` and add tests for `MAC16` instructions * revert * Prepare for update (capstone-engine#2552) * Bindings(chore): Fix DeprecationWarning * Version(upgrade): update bindings const * Fix(chore): Fix ARMCC_Invalid is not defined * Update Changelog Version to 6.0.0-Alpha2 (capstone-engine#2553) * Bindings(chore): Fix DeprecationWarning * Version(upgrade): update bindings const * Fix(chore): Fix ARMCC_Invalid is not defined * Changelog: Update to version 6.0.0-Alpha2 --------- Co-authored-by: Wu ChenXu <kabeor00@gmail.com> Co-authored-by: Florian Märkl <info@florianmaerkl.de> Co-authored-by: billow <billow.fun@gmail.com> Co-authored-by: Andrew <afq2101@columbia.edu> Co-authored-by: @Antelox <anteloxrce@gmail.com>


Your checklist for this pull request
This is updating the CI, so every release, this would be tested.
Detailed description
A new feature is made such that every time a new release is created, a Debian package is created and uploaded to releases.
Approach:
Create a Debian folder, this contains everything needed to create a Debian package in the container. The container compiles and installs capstone, and creates the Debian package. The CI is updated to run the script which creates the container for upload release.
Also, I did slight tweaks as your release CI needed to be updated too. I updated the release process to not use end of life uploading.
https://github.com/actions/upload-release-asset
Based on my hack in the Dockerfile, it seems to identify which files I need to copy in the Debian package

Note, I am copying (essentially should I copy /usr/lib/libcapstone.so too into the Debian Package?)
libcapstone.a
libcapstone.so
libcapstone.so.6
Proof the Debian package is created

...
Test plan
See thread below, a script is created to check that certain packages/libraries are in the Debian Package before adding to release.
...
Closing issues
closes #2398
...