Skip to content

Change CI to create Debian Package to Release#2521

Merged
kabeor merged 2 commits intocapstone-engine:nextfrom
AndrewQuijano:debian
Nov 4, 2024
Merged

Change CI to create Debian Package to Release#2521
kabeor merged 2 commits intocapstone-engine:nextfrom
AndrewQuijano:debian

Conversation

@AndrewQuijano
Copy link
Copy Markdown
Contributor

@AndrewQuijano AndrewQuijano commented Oct 19, 2024

Your checklist for this pull request

  • I've documented or updated the documentation of every API function and struct this PR changes.
  • I've added tests that prove my fix is effective or that my feature works (if possible)

This is updating the CI, so every release, this would be tested.

Detailed description

A new feature is made such that every time a new release is created, a Debian package is created and uploaded to releases.

Approach:
Create a Debian folder, this contains everything needed to create a Debian package in the container. The container compiles and installs capstone, and creates the Debian package. The CI is updated to run the script which creates the container for upload release.

Also, I did slight tweaks as your release CI needed to be updated too. I updated the release process to not use end of life uploading.
https://github.com/actions/upload-release-asset

Based on my hack in the Dockerfile, it seems to identify which files I need to copy in the Debian package
image

Note, I am copying (essentially should I copy /usr/lib/libcapstone.so too into the Debian Package?)
libcapstone.a
libcapstone.so
libcapstone.so.6

Proof the Debian package is created
image

...

Test plan

See thread below, a script is created to check that certain packages/libraries are in the Debian Package before adding to release.

...

Closing issues

closes #2398
...

@github-actions github-actions Bot added the Github-files Github related files label Oct 19, 2024
@AndrewQuijano AndrewQuijano changed the title [DO NOT MERGE] Some initial work on creating Debian Package Change CI to create Debian Package to Release Oct 20, 2024
Copy link
Copy Markdown
Collaborator

@Rot127 Rot127 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! Please address the comments. We can test it by making a release in our own forks. And download and install the artifacts.

Comment thread .github/workflows/build_release.yml
Comment thread debian/Dockerfile Outdated
Comment thread debian/control Outdated
Comment thread debian/setup.sh Outdated
@Rot127
Copy link
Copy Markdown
Collaborator

Rot127 commented Oct 21, 2024

Note, I am copying (essentially should I copy /usr/lib/libcapstone.so too into the Debian Package?)

I think it should contain the static and shared lib file. You might need to build Capstone twice with cmake. Once for the static, once for creating the shared one.

@AndrewQuijano
Copy link
Copy Markdown
Contributor Author

AndrewQuijano commented Oct 22, 2024

Current diff looks like this, I suspect I can ignore the /sys stuff and I suspect I can ignore the stuff under /usr/local/lib/cmake/

7526a7527,7553
> /usr/local/include/capstone/loongarch.h
> /usr/local/include/capstone/riscv.h
> /usr/local/include/capstone/x86.h
> /usr/local/include/capstone/xcore.h
> /usr/local/include/capstone/tricore.h
> /usr/local/include/capstone/platform.h
> /usr/local/include/capstone/mips.h
> /usr/local/include/capstone/hppa.h
> /usr/local/include/capstone/bpf.h
> /usr/local/include/capstone/sh.h
> /usr/local/include/capstone/wasm.h
> /usr/local/include/capstone/systemz.h
> /usr/local/include/capstone/ppc.h
> /usr/local/include/capstone/aarch64.h
> /usr/local/include/capstone/tms320c64x.h
> /usr/local/include/capstone/arm.h
> /usr/local/include/capstone/mos65xx.h
> /usr/local/include/capstone/cs_operand.h
> /usr/local/include/capstone/m68k.h
> /usr/local/include/capstone/systemz_compatibility.h
> /usr/local/include/capstone/alpha.h
> /usr/local/include/capstone/evm.h
> /usr/local/include/capstone/m680x.h
> /usr/local/include/capstone/arm64.h
> /usr/local/include/capstone/xtensa.h
> /usr/local/include/capstone/capstone.h
> /usr/local/include/capstone/sparc.h
7549a7577,7582
> /usr/local/lib/pkgconfig/capstone.pc
> /usr/local/lib/cmake/capstone/capstone-targets.cmake
> /usr/local/lib/cmake/capstone/capstone-targets-release.cmake
> /usr/local/lib/cmake/capstone/capstone-config-version.cmake
> /usr/local/lib/cmake/capstone/capstone-config.cmake
> /usr/local/lib/libcapstone.so.6.0
10138a10172
> /usr/local/bin/cstool
21455,21523d21488
< /sys/devices/virtual/net/lo/uevent
< /sys/devices/virtual/net/lo/carrier_changes
< /sys/devices/virtual/net/lo/testing
< /sys/devices/virtual/net/lo/carrier
< /sys/devices/virtual/net/lo/dev_id
< /sys/devices/virtual/net/lo/carrier_down_count
< /sys/devices/virtual/net/lo/proto_down
< /sys/devices/virtual/net/lo/address
< /sys/devices/virtual/net/lo/operstate
< /sys/devices/virtual/net/lo/phys_switch_id
< /sys/devices/virtual/net/lo/link_mode
< /sys/devices/virtual/net/lo/dormant
< /sys/devices/virtual/net/lo/statistics/tx_errors
< /sys/devices/virtual/net/lo/statistics/rx_length_errors
< /sys/devices/virtual/net/lo/statistics/rx_packets
< /sys/devices/virtual/net/lo/statistics/tx_carrier_errors
< /sys/devices/virtual/net/lo/statistics/tx_dropped
< /sys/devices/virtual/net/lo/statistics/rx_missed_errors
< /sys/devices/virtual/net/lo/statistics/rx_over_errors
< /sys/devices/virtual/net/lo/statistics/tx_aborted_errors
< /sys/devices/virtual/net/lo/statistics/rx_crc_errors
< /sys/devices/virtual/net/lo/statistics/rx_frame_errors
< /sys/devices/virtual/net/lo/statistics/rx_nohandler
< /sys/devices/virtual/net/lo/statistics/tx_fifo_errors
< /sys/devices/virtual/net/lo/statistics/multicast
< /sys/devices/virtual/net/lo/statistics/tx_packets
< /sys/devices/virtual/net/lo/statistics/tx_window_errors
< /sys/devices/virtual/net/lo/statistics/rx_bytes
< /sys/devices/virtual/net/lo/statistics/collisions
< /sys/devices/virtual/net/lo/statistics/rx_dropped
< /sys/devices/virtual/net/lo/statistics/tx_bytes
< /sys/devices/virtual/net/lo/statistics/tx_heartbeat_errors
< /sys/devices/virtual/net/lo/statistics/rx_fifo_errors
< /sys/devices/virtual/net/lo/statistics/rx_errors
< /sys/devices/virtual/net/lo/statistics/tx_compressed
< /sys/devices/virtual/net/lo/statistics/rx_compressed
< /sys/devices/virtual/net/lo/mtu
< /sys/devices/virtual/net/lo/gro_flush_timeout
< /sys/devices/virtual/net/lo/carrier_up_count
< /sys/devices/virtual/net/lo/ifalias
< /sys/devices/virtual/net/lo/speed
< /sys/devices/virtual/net/lo/netdev_group
< /sys/devices/virtual/net/lo/napi_defer_hard_irqs
< /sys/devices/virtual/net/lo/phys_port_name
< /sys/devices/virtual/net/lo/ifindex
< /sys/devices/virtual/net/lo/broadcast
< /sys/devices/virtual/net/lo/type
< /sys/devices/virtual/net/lo/dev_port
< /sys/devices/virtual/net/lo/phys_port_id
< /sys/devices/virtual/net/lo/queues/tx-0/tx_maxrate
< /sys/devices/virtual/net/lo/queues/tx-0/xps_cpus
< /sys/devices/virtual/net/lo/queues/tx-0/byte_queue_limits/limit_min
< /sys/devices/virtual/net/lo/queues/tx-0/byte_queue_limits/limit
< /sys/devices/virtual/net/lo/queues/tx-0/byte_queue_limits/limit_max
< /sys/devices/virtual/net/lo/queues/tx-0/byte_queue_limits/hold_time
< /sys/devices/virtual/net/lo/queues/tx-0/byte_queue_limits/inflight
< /sys/devices/virtual/net/lo/queues/tx-0/tx_timeout
< /sys/devices/virtual/net/lo/queues/tx-0/xps_rxqs
< /sys/devices/virtual/net/lo/queues/tx-0/traffic_class
< /sys/devices/virtual/net/lo/queues/rx-0/rps_flow_cnt
< /sys/devices/virtual/net/lo/queues/rx-0/rps_cpus
< /sys/devices/virtual/net/lo/name_assign_type
< /sys/devices/virtual/net/lo/duplex
< /sys/devices/virtual/net/lo/addr_assign_type
< /sys/devices/virtual/net/lo/addr_len
< /sys/devices/virtual/net/lo/threaded
< /sys/devices/virtual/net/lo/tx_queue_len
< /sys/devices/virtual/net/lo/iflink
< /sys/devices/virtual/net/lo/flags
21592a21558,21626
> /sys/devices/virtual/net/lo/uevent
> /sys/devices/virtual/net/lo/carrier_changes
> /sys/devices/virtual/net/lo/testing
> /sys/devices/virtual/net/lo/carrier
> /sys/devices/virtual/net/lo/dev_id
> /sys/devices/virtual/net/lo/carrier_down_count
> /sys/devices/virtual/net/lo/proto_down
> /sys/devices/virtual/net/lo/address
> /sys/devices/virtual/net/lo/operstate
> /sys/devices/virtual/net/lo/phys_switch_id
> /sys/devices/virtual/net/lo/link_mode
> /sys/devices/virtual/net/lo/dormant
> /sys/devices/virtual/net/lo/statistics/tx_errors
> /sys/devices/virtual/net/lo/statistics/rx_length_errors
> /sys/devices/virtual/net/lo/statistics/rx_packets
> /sys/devices/virtual/net/lo/statistics/tx_carrier_errors
> /sys/devices/virtual/net/lo/statistics/tx_dropped
> /sys/devices/virtual/net/lo/statistics/rx_missed_errors
> /sys/devices/virtual/net/lo/statistics/rx_over_errors
> /sys/devices/virtual/net/lo/statistics/tx_aborted_errors
> /sys/devices/virtual/net/lo/statistics/rx_crc_errors
> /sys/devices/virtual/net/lo/statistics/rx_frame_errors
> /sys/devices/virtual/net/lo/statistics/rx_nohandler
> /sys/devices/virtual/net/lo/statistics/tx_fifo_errors
> /sys/devices/virtual/net/lo/statistics/multicast
> /sys/devices/virtual/net/lo/statistics/tx_packets
> /sys/devices/virtual/net/lo/statistics/tx_window_errors
> /sys/devices/virtual/net/lo/statistics/rx_bytes
> /sys/devices/virtual/net/lo/statistics/collisions
> /sys/devices/virtual/net/lo/statistics/rx_dropped
> /sys/devices/virtual/net/lo/statistics/tx_bytes
> /sys/devices/virtual/net/lo/statistics/tx_heartbeat_errors
> /sys/devices/virtual/net/lo/statistics/rx_fifo_errors
> /sys/devices/virtual/net/lo/statistics/rx_errors
> /sys/devices/virtual/net/lo/statistics/tx_compressed
> /sys/devices/virtual/net/lo/statistics/rx_compressed
> /sys/devices/virtual/net/lo/mtu
> /sys/devices/virtual/net/lo/gro_flush_timeout
> /sys/devices/virtual/net/lo/carrier_up_count
> /sys/devices/virtual/net/lo/ifalias
> /sys/devices/virtual/net/lo/speed
> /sys/devices/virtual/net/lo/netdev_group
> /sys/devices/virtual/net/lo/napi_defer_hard_irqs
> /sys/devices/virtual/net/lo/phys_port_name
> /sys/devices/virtual/net/lo/ifindex
> /sys/devices/virtual/net/lo/broadcast
> /sys/devices/virtual/net/lo/type
> /sys/devices/virtual/net/lo/dev_port
> /sys/devices/virtual/net/lo/phys_port_id
> /sys/devices/virtual/net/lo/queues/tx-0/tx_maxrate
> /sys/devices/virtual/net/lo/queues/tx-0/xps_cpus
> /sys/devices/virtual/net/lo/queues/tx-0/byte_queue_limits/limit_min
> /sys/devices/virtual/net/lo/queues/tx-0/byte_queue_limits/limit
> /sys/devices/virtual/net/lo/queues/tx-0/byte_queue_limits/limit_max
> /sys/devices/virtual/net/lo/queues/tx-0/byte_queue_limits/hold_time
> /sys/devices/virtual/net/lo/queues/tx-0/byte_queue_limits/inflight
> /sys/devices/virtual/net/lo/queues/tx-0/tx_timeout
> /sys/devices/virtual/net/lo/queues/tx-0/xps_rxqs
> /sys/devices/virtual/net/lo/queues/tx-0/traffic_class
> /sys/devices/virtual/net/lo/queues/rx-0/rps_flow_cnt
> /sys/devices/virtual/net/lo/queues/rx-0/rps_cpus
> /sys/devices/virtual/net/lo/name_assign_type
> /sys/devices/virtual/net/lo/duplex
> /sys/devices/virtual/net/lo/addr_assign_type
> /sys/devices/virtual/net/lo/addr_len
> /sys/devices/virtual/net/lo/threaded
> /sys/devices/virtual/net/lo/tx_queue_len
> /sys/devices/virtual/net/lo/iflink
> /sys/devices/virtual/net/lo/flags
29901c29935
< /before-install.txt
---
> /after-install.txt
30087a30122
> /capstone/build/install_manifest.txt
32467a32503
> /before-install.txt

@AndrewQuijano
Copy link
Copy Markdown
Contributor Author

Looks like it is working, going to push changes shortly

image

@AndrewQuijano
Copy link
Copy Markdown
Contributor Author

AndrewQuijano commented Oct 22, 2024

Oh and @Rot127, assuming all this works as intended, could you make a capstone 5.0.3 Debian package or a 5.0.4 Debian package? The tool I use PANDA, for now, seems to only support version 5. The goal is to replace this with a one line to download the Debian package and install it.

Essentially, I do need a libcapstone.so.5 for the current capstone version
https://github.com/panda-re/panda/blob/dev/panda/scripts/install_ubuntu.sh#L109-L118

https://github.com/panda-re/panda/blob/dev/Dockerfile#L37-L41

@AndrewQuijano
Copy link
Copy Markdown
Contributor Author

AndrewQuijano commented Oct 22, 2024

Actually, I released I can test this on my own fork. Just be careful, you can only create tags say 5.0.4. If you have letters, think the debian part will break, just as a heads up. Works on 1.5.8 but not say v1.5.5

https://github.com/AndrewQuijano/capstone/releases/tag/1.5.8
https://github.com/AndrewQuijano/capstone/releases/tag/v1.5.5

@Rot127
Copy link
Copy Markdown
Collaborator

Rot127 commented Oct 22, 2024

Thanks for all the work!

Just be careful, you can only create tags say 5.0.4. If you have letters, think the debian part will break, just as a heads up. Works on 1.5.8 but not say v1.5.5

This is very good to know. Can you may lint the version string given to the script?
So for example:

  • Remove letters from debian
  • Or don't build for debian if other letters than a v prefix is given (e.g. in the case of 6.0.0a for Alpha v6)?

We just need a way to have this checked, handled and documented in code. Because otherwise it will absolutely be forgotten at some point.

@Rot127
Copy link
Copy Markdown
Collaborator

Rot127 commented Oct 22, 2024

could you make a capstone 5.0.3 Debian package or a 5.0.4 Debian

Yes, we can release a new v5 version. Was on the list anyways. Just waiting for the TriCore bug to be resolved.

Copy link
Copy Markdown
Collaborator

@Rot127 Rot127 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for the corrections. I spent way more time now on checking everything.

I ran it manually on a Debian VM and some stuff broke. Package building worked fine (as root: ./setup.sh 6.0.0).

But there are a few problems:

The capstone.pc in the package is:

# After unpacking capstone.deb
> cat capstone/data/usr/local/lib/pkgconfig/capstone.pc

prefix=/usr/local
exec_prefix=${prefix}
libdir=${prefix}/lib
includedir=${prefix}/include

Name: capstone
Description: Capstone disassembly engine
Version: 6.0.
URL: https://www.capstone-engine.org/
archive=${libdir}/libcapstone.a
Libs: -L${libdir} -lcapstone
Libs.private: -L${libdir} -l:libcapstone.a
Cflags: -I${includedir} -I${includedir}/capstone
archs=
  • The version number is not correct, it should end with a 0. But ends with a ..
  • archs is empty. I think it should be arch (no s) and be set to x86_64? But couldn't find docs about it. Just remove it, if you can't find an answer.
  • libcapstone.a is not in the package. I think it is best, if you just fix CMakeLists.txt so it always compiles the static lib. Here the fix:
    diff --git a/CMakeLists.txt b/CMakeLists.txt
    
    index 345740c9..b8b01be6 100644
    --- a/CMakeLists.txt
    +++ b/CMakeLists.txt
    @@ -761,16 +761,23 @@ set(ALL_HEADERS
     set_property(GLOBAL PROPERTY VERSION ${PROJECT_VERSION})
     
     ## targets
    -add_library(capstone ${ALL_SOURCES} ${ALL_HEADERS})
    +add_library(capstone OBJECT ${ALL_SOURCES} ${ALL_HEADERS})
     add_library(capstone::capstone ALIAS capstone)
    +add_library(capstone_static STATIC $<TARGET_OBJECTS:capstone>)
    +# Use normal capstone name. Otherwise we get libcapstone_static.so
    +set_target_properties(capstone_static PROPERTIES OUTPUT_NAME "capstone")
     target_include_directories(capstone PUBLIC
         $<BUILD_INTERFACE:${PROJECT_SOURCE_DIR}/include>
     )
     set_property(TARGET capstone PROPERTY C_STANDARD 99)
     
     if(BUILD_SHARED_LIBS)
    -    target_compile_definitions(capstone PUBLIC CAPSTONE_SHARED)
    -    set_target_properties(capstone PROPERTIES
    +    set_property(TARGET capstone PROPERTY POSITION_INDEPENDENT_CODE 1)
    +    add_library(capstone_shared SHARED $<TARGET_OBJECTS:capstone>)
    +    # Use normal capstone name. Otherwise we get libcapstone_shared.so
    +    set_target_properties(capstone_shared PROPERTIES OUTPUT_NAME "capstone")
    +    target_compile_definitions(capstone_shared PUBLIC CAPSTONE_SHARED)
    +    set_target_properties(capstone_shared PROPERTIES
             VERSION ${PROJECT_VERSION}
             SOVERSION ${PROJECT_VERSION_MAJOR}
         )
  • cstool doesn't work after installing the package, because ldconfig is not run (and /usr/local/lib is not in the default lib search path). This seems to be the way to run ldconfig after install.
  • Please add a test script which installs the package and checks everything is correct (version number, libs are there, libs report the same version, cstool works etc). It would be good to have this in a separated job.

Sorry for being super nit-picky with this one. We had quite some problems with the Python release packages. And it is hard to fix, because we need to do a fix-release after changing it. Very stressful experience. This is why I ask you for so much stuff.

Comment thread debian/Dockerfile Outdated
Comment thread debian/Dockerfile
Comment thread debian/setup.sh
Comment thread debian/control Outdated
Comment thread debian/Dockerfile Outdated
Comment thread debian/Dockerfile Outdated
Comment thread .github/workflows/build_release.yml
Comment thread .github/workflows/build_release.yml
@AndrewQuijano
Copy link
Copy Markdown
Contributor Author

Hello,
I just got your changes in, seems like libcapstone.a is no longer appearing, not sure how to fix this

> /usr/local/include/capstone/loongarch.h
> /usr/local/include/capstone/riscv.h
> /usr/local/include/capstone/x86.h
> /usr/local/include/capstone/xcore.h
> /usr/local/include/capstone/tricore.h
> /usr/local/include/capstone/platform.h
> /usr/local/include/capstone/mips.h
> /usr/local/include/capstone/hppa.h
> /usr/local/include/capstone/bpf.h
> /usr/local/include/capstone/sh.h
> /usr/local/include/capstone/wasm.h
> /usr/local/include/capstone/systemz.h
> /usr/local/include/capstone/ppc.h
> /usr/local/include/capstone/aarch64.h
> /usr/local/include/capstone/tms320c64x.h
> /usr/local/include/capstone/arm.h
> /usr/local/include/capstone/mos65xx.h
> /usr/local/include/capstone/cs_operand.h
> /usr/local/include/capstone/m68k.h
> /usr/local/include/capstone/systemz_compatibility.h
> /usr/local/include/capstone/alpha.h
> /usr/local/include/capstone/evm.h
> /usr/local/include/capstone/m680x.h
> /usr/local/include/capstone/arm64.h
> /usr/local/include/capstone/xtensa.h
> /usr/local/include/capstone/capstone.h
> /usr/local/include/capstone/sparc.h
7549a7577,7581
> /usr/local/lib/pkgconfig/capstone.pc
> /usr/local/lib/cmake/capstone/capstone-targets.cmake
> /usr/local/lib/cmake/capstone/capstone-targets-release.cmake
> /usr/local/lib/cmake/capstone/capstone-config-version.cmake
> /usr/local/lib/cmake/capstone/capstone-config.cmake
10138a10171
> /usr/local/bin/cstool

@Rot127
Copy link
Copy Markdown
Collaborator

Rot127 commented Oct 22, 2024

Oups, forgot to update the install targets:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 345740c9..c8c10880 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -878,7 +885,13 @@ if(CAPSTONE_INSTALL)
         DESTINATION ${CAPSTONE_CMAKE_CONFIG_INSTALL_DIR}
     )
 
-    install(TARGETS capstone
+    if(BUILD_SHARED_LIBS)
+        set(LIB_INSTALL_TARGETS capstone_static capstone_shared)
+    else()
+        set(LIB_INSTALL_TARGETS capstone_static)
+    endif()
+
+    install(TARGETS ${LIB_INSTALL_TARGETS}
             EXPORT capstone-targets
             RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}
             LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}

@AndrewQuijano
Copy link
Copy Markdown
Contributor Author

OK, I just need to fix capstone.pc version and removing arch and I am done, but in the meantime please test what I got so far

Comment thread debian/check_capstone.sh
Comment thread debian/check_capstone.sh Outdated
@AndrewQuijano
Copy link
Copy Markdown
Contributor Author

Hm, this is tricky

image

@Rot127
Copy link
Copy Markdown
Collaborator

Rot127 commented Oct 22, 2024

@Rot127
Copy link
Copy Markdown
Collaborator

Rot127 commented Oct 22, 2024

Numbers seem fine. I think it is because it passes ./debian/capstone.deb to the check script. But the working dir is already ./debian/

@AndrewQuijano
Copy link
Copy Markdown
Contributor Author

yeah just patched it

@AndrewQuijano
Copy link
Copy Markdown
Contributor Author

Oh I forgot to pull out debian package because I was debugging the library issue. This should work end to end now

Comment thread debian/Dockerfile Outdated
@Rot127
Copy link
Copy Markdown
Collaborator

Rot127 commented Oct 22, 2024

I need to go. Will take a look at it tomorrow again. Thanks for fixing it so quickly.

@AndrewQuijano
Copy link
Copy Markdown
Contributor Author

No worries! Thanks for your fast responses! Really look forward to updating PANDA with the package. Also QQ where did you find buster-slim is EOL? Figure I might as well update PANDA's Debian package thing too, but figure I want to to document it in the PR

@AndrewQuijano
Copy link
Copy Markdown
Contributor Author

@Rot127
Copy link
Copy Markdown
Collaborator

Rot127 commented Oct 23, 2024

where did you find buster-slim is EOL?

Debian Buster is EOL since two years. Hence we don't have to build on it. This is also likely the reason you could not install cmake via apt.

@Rot127
Copy link
Copy Markdown
Collaborator

Rot127 commented Oct 23, 2024

@github-actions github-actions Bot added the Python Bindings label Oct 23, 2024
@AndrewQuijano
Copy link
Copy Markdown
Contributor Author

Clear up usage of static libraries.

  • Python bindings only use the dynamic lib. But built and copied the static ones sometimes nonetheless.
  • Add toggles to build only static, static/dyn or only dynamic.

Install cmake via apt.
cmake >3.12 was not present in Debian Buster.

@AndrewQuijano
Copy link
Copy Markdown
Contributor Author

@Rot127 just accepted your PR

@AndrewQuijano
Copy link
Copy Markdown
Contributor Author

Thanks for your help with this! Just curious, when would you expect v 5.0.4 of capstone Debian Package be ready by? Figure I can start working on the PR for PANDA to use the debian package. I'm wondering if there might come a day you might have an apt repository?

https://earthly.dev/blog/creating-and-hosting-your-own-deb-packages-and-apt-repo/

@Rot127
Copy link
Copy Markdown
Collaborator

Rot127 commented Oct 23, 2024

Nice! Lgtm. I fix the failing test tomorrow.
@kabeor Please review this one.

@Rot127
Copy link
Copy Markdown
Collaborator

Rot127 commented Oct 23, 2024

I'm wondering if there might come a day you might have an apt repository?

It is not planned. We need maintainers for this. And we are already too busy with all the other stuff to do it on our own.

when would you expect v 5.0.4 of capstone Debian Package be ready by?

#2523 is the PR which has to be done. I review it tomorrow as well. Then we need to merge it into v5.
Depending how much we have to change I guess 1-6 weeks. billow and me are pretty busy these days. So hard to say.

@AndrewQuijano
Copy link
Copy Markdown
Contributor Author

btw I noticed it still says request changes blocking the merge:
#2521 (review)

  1. Do you still want me to remove the 'archs' from the pkconfig?
  2. Safe to say the capstone debian package worked without needing ldconfig?

Copy link
Copy Markdown
Collaborator

@Rot127 Rot127 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please just address the last two small things.

Do you still want me to remove the 'archs' from the pkconfig?

Actually yeah. It is not used outside of the Make build. It was added to have a list of supported architectures. But this information doesn't belong there. And I couldn't find documentation about this archs field. So it is likely not standard anyways.

Safe to say the capstone debian package worked without needing ldconfig?

At least in my fresh Debian VM it didn't work otherwise. Feel free to test it again.

Comment thread debian/check_capstone.sh Outdated
Comment thread debian/postinst Outdated
@AndrewQuijano
Copy link
Copy Markdown
Contributor Author

@Rot127 Just made your suggested changes, let me know about if the ldconfig works/the package installation works as intended. Look forward to merging the PR!

AndrewQuijano and others added 2 commits November 2, 2024 13:05
version. Also updating release CI to not use end-of-life workflows
- Python bindings only use the dynamic lib. But built and copied the static ones sometimes nonetheless.
- Add toggles to build only static, static/dyn or only dynamic.
@AndrewQuijano
Copy link
Copy Markdown
Contributor Author

AndrewQuijano commented Nov 2, 2024

@Rot127, let me know when this PR is to be merged.

I made one last change, and provided more context in #2398. I changed the package name from 'capstone.deb' to 'libcapstone-dev.deb' so it matches the name of the apt-package. Just to confirm, this debian package essentially would be both libcapstone5.deb and libcapstone-dev.deb package from apt? I suspect the answer is yes given that both library file is added, and the /usr/local/include is populated with headers.

And please confirm the package works as expected. Essentially, PANDA would have required both the capstone library, and does rely on capstone for development.
https://www.capstone-engine.org/documentation.html

Copy link
Copy Markdown
Collaborator

@Rot127 Rot127 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am sorry, I thought I already approved this one. But I can't merge anyways. The review of @kabeor is required. I already reminded him a few days ago.

libcapstone5.deb and libcapstone-dev.deb package from apt?

Yeah, you are right. We should have a libcapstone6 (not libcapstone5) and libcapstone-dev package. But let's keep the PR like this for now please. Because it is already large enough.
Also, as long as we have not submitted it to the Debian/Ubuntu repos, we don't need the distinction yet.

If you want to take care of it, feel free to open a new PR of course.

And please confirm the package works as expected.

Works for me! Tested it with a Whonix VM. See https://github.com/Rot127/capstone/releases/tag/v99.99.103

Comment thread CMakeLists.txt
Comment thread CMakeLists.txt
Copy link
Copy Markdown
Member

@kabeor kabeor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything else are LGTM.

Copy link
Copy Markdown
Member

@kabeor kabeor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, this is amazing.

@kabeor kabeor merged commit d7be5f9 into capstone-engine:next Nov 4, 2024
@AndrewQuijano AndrewQuijano deleted the debian branch November 10, 2024 01:21
@AndrewQuijano AndrewQuijano mentioned this pull request Nov 10, 2024
2 tasks
@Rot127 Rot127 added the Ubuntu/Debian Linux - Distribution label Nov 10, 2024
kabeor added a commit that referenced this pull request Nov 24, 2024
* Update changelog for V6.0.0-Alpha1 (#2493)

* update version to v6-alpha1

* update bindings const values

* Update changelog for V6.0.0-Alpha1

* Remove irrelevant changes. (#2495)

* Fixing UB santizer, `LITBASE` and assert errors. (#2499)

* Update labeler with Xtensa and v6 files. (#2500)

* Add hard asserts to all SStream functions and memset MCInst. (#2501)

* Only trigger on released action. (#2497)

* Fix cstest build with Ninja (#2506)

* Tricore EA calculation (#2504)

* Update libcyaml dependency in cstest to 1.4.2 (#2508)

* AArch64: Replace vararg add_cs_detail by multiple concrete functions

Fixes UB caused by various mismatches on how these arguments are passed
and read. This became visible when running on PowerPC hosts with e.g.
`cstool -d aarch64 204862f8`.
Apart from the UB fix, this is meant to be a pure refactor.

Partially addresses #2458

* xtensa: Fix Branch Target (#2516)

* xtensa: Fix Branch Target

* auto-sync: fix byte pattern

* xtensa: add branch insn tests

* Revert "auto-sync: fix byte pattern"

This reverts commit cf8e870.

* Fix #2509. (#2510)

Compatibility headers should always include the header in the same dir.

* Fix stringop-truncation warning some compilers raise. (#2522)

* Add CC and VAS compatibility macros (#2525)

* Fix endianess issue during assignment. (#2528)

* This time actually fix big endian issue. (#2530)

* tricore: fixes #2474 (#2523)

* tricore: fix auto-sync tricore

* tricore: fixes TriCoreGenCSMappingInsnName.inc

* tricore: fixes

* tricore: try fix ld.a SC

* tricore: fixes all

* Add TriCore to .github/workflows/auto-sync.yaml

* Add TriCore details tests(a15, d15, a10|sp)

* Change CI to create Debian Package to Release (#2521)

* Updating CI to create Debian package and version is assigned by tag
version. Also updating release CI to not use end-of-life workflows

* Clear up usage of static libraries.

- Python bindings only use the dynamic lib. But built and copied the static ones sometimes nonetheless.
- Add toggles to build only static, static/dyn or only dynamic.

---------

Co-authored-by: Rot127 <unisono@quyllur.org>

* Rename build arguments: (#2534)

- BUILD_SHARED_LIBS -> CAPSTONE_BUILD_SHARED_LIBS
- BUILD_STATIC_LIBS -> CAPSTONE_BUILD_STATIC_LIBS
- BUILD_STATIC_LIBS -> CAPSTONE_BUILD_STATIC_MSVC_RUNTIME

* xtensa: update to espressif/llvm-project (#2533)

* fix coverity (#2546)

- cid 514642

- cid 514643

- cid 514644

- cid 514645

* Move debian package generation to a dispatch only workflow (#2543)

* Move deb package gen files int package/deb

* Fix basename check

* Make debian package generation dispatch only

* Python package building rework (#2538)

* - Refactored setup.py to remove hacks regarding packaging of wheels for different platforms, improve and cleanup the code
- Updated README.txt
- Removed old Makefile and build_wheel.sh scripts
- Created a new workflow that takes care of building and testing python packages for different platforms/architectures/python versions

* Added SPDX headers to the setup.py

* - cstest_py: Fixed positional argument since it doesn't accept a `required` flag. It turns to have a mandatory tests folder path
- integration_tests.py: Use pathlib to determine the required path
- GitHub action: Simplified the tests execution command

* GitHub Actions: Run python 3.8 (lowest) and 3.13 (current highest) for native runners only during testings and the rest during tag release

* GitHub Action:
- Fixed the cibw_build matrix element
- Added a step to prepare artifact name

* GitHub Action: Added run_tests.py script to run all tests during CI workflow

* - Added SPDX headers to the run_tests.py script and to the build-wheels-publish.yml workflow file
- Minor fixes to the workflow as pointed out in the PR review
- Updated MANIFEST.in to reflect the actual libraries built during python wheel creation process
- Use subprocess.run in place of os.system in run_tests.py script

* GitHub Action:
- Run qemu step only if non-native Linux runner
- Added arch:universal2 matrix element for macos-latest runner

* Python bindings: Refreshed the list of files needed to be copied for sdist archive

* GitHub Action: Commented out arch:x86 matrix elements

* GitHub Action: Run qemu step only if non-native Linux runner

* GitHub Action: Minor fixes

* Python bindings: Added missing .in pattern when collecting src files for sdist archive

* Auto-Sync reproducability + ARM update (#2532)

* fix xtensa DecodeMR23RegisterClass and add tests for MAC16 instru… (#2551)

* fix xtensa `DecodeMR23RegisterClass` and add tests for `MAC16` instructions

* revert

* Prepare for update (#2552)

* Bindings(chore): Fix DeprecationWarning

* Version(upgrade): update bindings const

* Fix(chore): Fix ARMCC_Invalid is not defined

* Update Changelog Version to 6.0.0-Alpha2 (#2553)

* Bindings(chore): Fix DeprecationWarning

* Version(upgrade): update bindings const

* Fix(chore): Fix ARMCC_Invalid is not defined

* Changelog: Update to version 6.0.0-Alpha2

---------

Co-authored-by: Rot127 <45763064+Rot127@users.noreply.github.com>
Co-authored-by: Florian Märkl <info@florianmaerkl.de>
Co-authored-by: billow <billow.fun@gmail.com>
Co-authored-by: Andrew <afq2101@columbia.edu>
Co-authored-by: Rot127 <unisono@quyllur.org>
Co-authored-by: @Antelox <anteloxrce@gmail.com>
Rot127 added a commit to Rot127/capstone that referenced this pull request Mar 31, 2025
* Update changelog for V6.0.0-Alpha1 (capstone-engine#2493) (capstone-engine#2494)

* update version to v6-alpha1

* update bindings const values

* Update changelog for V6.0.0-Alpha1

* Remove irrelevant changes. (capstone-engine#2496)

* Fixing UB santizer, `LITBASE` and assert errors. (capstone-engine#2499)

* Update labeler with Xtensa and v6 files. (capstone-engine#2500)

* Add hard asserts to all SStream functions and memset MCInst. (capstone-engine#2501)

* Only trigger on released action. (capstone-engine#2497)

* Fix cstest build with Ninja (capstone-engine#2506)

* Tricore EA calculation (capstone-engine#2504)

* Update libcyaml dependency in cstest to 1.4.2 (capstone-engine#2508)

* AArch64: Replace vararg add_cs_detail by multiple concrete functions

Fixes UB caused by various mismatches on how these arguments are passed
and read. This became visible when running on PowerPC hosts with e.g.
`cstool -d aarch64 204862f8`.
Apart from the UB fix, this is meant to be a pure refactor.

Partially addresses capstone-engine#2458

* xtensa: Fix Branch Target (capstone-engine#2516)

* xtensa: Fix Branch Target

* auto-sync: fix byte pattern

* xtensa: add branch insn tests

* Revert "auto-sync: fix byte pattern"

This reverts commit cf8e870.

* Fix capstone-engine#2509. (capstone-engine#2510)

Compatibility headers should always include the header in the same dir.

* Fix stringop-truncation warning some compilers raise. (capstone-engine#2522)

* Add CC and VAS compatibility macros (capstone-engine#2525)

* Fix endianess issue during assignment. (capstone-engine#2528)

* This time actually fix big endian issue. (capstone-engine#2530)

* tricore: fixes capstone-engine#2474 (capstone-engine#2523)

* tricore: fix auto-sync tricore

* tricore: fixes TriCoreGenCSMappingInsnName.inc

* tricore: fixes

* tricore: try fix ld.a SC

* tricore: fixes all

* Add TriCore to .github/workflows/auto-sync.yaml

* Add TriCore details tests(a15, d15, a10|sp)

* Change CI to create Debian Package to Release (capstone-engine#2521)

* Updating CI to create Debian package and version is assigned by tag
version. Also updating release CI to not use end-of-life workflows

* Clear up usage of static libraries.

- Python bindings only use the dynamic lib. But built and copied the static ones sometimes nonetheless.
- Add toggles to build only static, static/dyn or only dynamic.

---------

Co-authored-by: Rot127 <unisono@quyllur.org>

* Rename build arguments: (capstone-engine#2534)

- BUILD_SHARED_LIBS -> CAPSTONE_BUILD_SHARED_LIBS
- BUILD_STATIC_LIBS -> CAPSTONE_BUILD_STATIC_LIBS
- BUILD_STATIC_LIBS -> CAPSTONE_BUILD_STATIC_MSVC_RUNTIME

* xtensa: update to espressif/llvm-project (capstone-engine#2533)

* fix coverity (capstone-engine#2546)

- cid 514642

- cid 514643

- cid 514644

- cid 514645

* Move debian package generation to a dispatch only workflow (capstone-engine#2543)

* Move deb package gen files int package/deb

* Fix basename check

* Make debian package generation dispatch only

* Python package building rework (capstone-engine#2538)

* - Refactored setup.py to remove hacks regarding packaging of wheels for different platforms, improve and cleanup the code
- Updated README.txt
- Removed old Makefile and build_wheel.sh scripts
- Created a new workflow that takes care of building and testing python packages for different platforms/architectures/python versions

* Added SPDX headers to the setup.py

* - cstest_py: Fixed positional argument since it doesn't accept a `required` flag. It turns to have a mandatory tests folder path
- integration_tests.py: Use pathlib to determine the required path
- GitHub action: Simplified the tests execution command

* GitHub Actions: Run python 3.8 (lowest) and 3.13 (current highest) for native runners only during testings and the rest during tag release

* GitHub Action:
- Fixed the cibw_build matrix element
- Added a step to prepare artifact name

* GitHub Action: Added run_tests.py script to run all tests during CI workflow

* - Added SPDX headers to the run_tests.py script and to the build-wheels-publish.yml workflow file
- Minor fixes to the workflow as pointed out in the PR review
- Updated MANIFEST.in to reflect the actual libraries built during python wheel creation process
- Use subprocess.run in place of os.system in run_tests.py script

* GitHub Action:
- Run qemu step only if non-native Linux runner
- Added arch:universal2 matrix element for macos-latest runner

* Python bindings: Refreshed the list of files needed to be copied for sdist archive

* GitHub Action: Commented out arch:x86 matrix elements

* GitHub Action: Run qemu step only if non-native Linux runner

* GitHub Action: Minor fixes

* Python bindings: Added missing .in pattern when collecting src files for sdist archive

* Auto-Sync reproducability + ARM update (capstone-engine#2532)

* fix xtensa DecodeMR23RegisterClass and add tests for MAC16 instru… (capstone-engine#2551)

* fix xtensa `DecodeMR23RegisterClass` and add tests for `MAC16` instructions

* revert

* Prepare for update (capstone-engine#2552)

* Bindings(chore): Fix DeprecationWarning

* Version(upgrade): update bindings const

* Fix(chore): Fix ARMCC_Invalid is not defined

* Update Changelog Version to 6.0.0-Alpha2 (capstone-engine#2553)

* Bindings(chore): Fix DeprecationWarning

* Version(upgrade): update bindings const

* Fix(chore): Fix ARMCC_Invalid is not defined

* Changelog: Update to version 6.0.0-Alpha2

---------

Co-authored-by: Wu ChenXu <kabeor00@gmail.com>
Co-authored-by: Florian Märkl <info@florianmaerkl.de>
Co-authored-by: billow <billow.fun@gmail.com>
Co-authored-by: Andrew <afq2101@columbia.edu>
Co-authored-by: @Antelox <anteloxrce@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Github-files Github related files Python Bindings Ubuntu/Debian Linux - Distribution

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Create a Debian Package on releases

3 participants