Raw binary blob support

[technikelly]

Checklist

Which kind of PR do you create?

• This PR only contains minor fixes.
• This PR contains major feature update.
• This PR introduces a new function/api for Qiling Framework.

Coding convention?

• The new code conforms to Qiling Framework naming convention.
• The imports are arranged properly.
• Essential comments are added.
• The reference of the new code is pointed out.

Extra tests?

• No extra tests are needed for this PR.
• I have added enough tests for this PR.
• Tests will be added after some discussion and review.

Changelog?

• This PR doesn't need to update Changelog.
• Changelog will be updated after some proper review.
• Changelog has been updated in my PR.

Target branch?

• The target branch is dev branch.

One last thing

• I have read the contribution guide

---

Overview

This PR introduces a small code change that allows users to specify BLOB_RAW in the profile, instructing the Qiling loader not to map any memory segments. This functionality is particularly useful for emulating small code snippets or bare-metal code without an operating system, offering greater customization and flexibility. Users can now define all memory mappings directly within their Qiling scripts, tailoring the environment precisely to their use case.

With this change, it becomes easier to port existing Unicorn scripts to Qiling and take advantage of additional features Qiling provides, such as debugging support and code coverage generation. Users can simply replace Unicorn API calls with equivalent Qiling API calls to benefit from these enhancements.

Notes

The added unit tests are currently disabled because the test binary has not been merged to rootfs. The added example code will also not work until the test binary has been merged into rootfs. Corresponding rootfs PR - qilingframework/rootfs#36

[elicn]

Hi @technikelly, welcome and thank you for your contribution.

Though the motivation behind this PR does resonate with me, I believe it introduces a special use case that could be covered by the existing Blob loader with minimal changes. While providing a convenient method for their users, special use cases tend to serve a very focused or limited purpose and introduce more complexity to the framework as it needs to support these cases on various components.

This is what I've picked up in my review:

• The new approach uses binary_name key in profile, which is inconsistent with Qiling usage model. Blob is assumed to be provided directly as bytes, that is actually more "Unicorn-like"
• Profile has a binary_size key which appears to be redundant, as file size is inferred when it is being read or simply by the length of code bytes
• The existing Blob loader could be modified as follows to provide an equivalent functionality:
  • Treat the heap_size key as optional and configure the heap only if it is available. This way only the bare minimum memory is allocated, avoiding other Qiling "noise"
  • Everything else remains intact, while the user script can monkeypatch details as needed (e.g.: patch module name in loader, unmap unnecessary memory areas, etc.)

Also, please note that we cannot accept proprietary code with copyrights.
Please consider this feedback and let me know if you have any concerns.

[BTW - you may want to consider udbserver as a simpler method to debug Unicorn scripts]

[technikelly]

Hi @elicn, thank you for getting back to me so quickly. I appreciate your thorough explanation of your reasoning and I fully agree that this could be achieved with a much smaller code change to qiling and monkeypatching the user script.

I'll go ahead and make the change you suggested - of heap_size being configurable within the blob loader.

Regarding the copyrights, the code is not proprietary. Cisco requires a copyright notice on open source contributions, but the code ultimately falls under the license of the overall project which in this case is GPLv2. With that additional information is it still an issue to have copyright notices included with contributions to qiling?

Thanks for the tip about udbserver - it definitely is simpler, but having code coverage in addition to debug is really nice.

[xwings]

Hi @elicn, thank you for getting back to me so quickly. I appreciate your thorough explanation of your reasoning and I fully agree that this could be achieved with a much smaller code change to qiling and monkeypatching the user script.

I'll go ahead and make the change you suggested - of heap_size being configurable within the blob loader.

Regarding the copyrights, the code is not proprietary. Cisco requires a copyright notice on open source contributions, but the code ultimately falls under the license of the overall project which in this case is GPLv2. With that additional information is it still an issue to have copyright notices included with contributions to qiling?

Thanks for the tip about udbserver - it definitely is simpler, but having code coverage in addition to debug is really nice.

Hi,

Thank you for PR and also welcome to Qiling Framework.

We do have the "tradition" to include author (PR) during the early days. Along the way we found out the list will get longer and longer in the specific file.

Later, we decided update changelog and put the names in sparate for easier maintainance. I hope you understand where we come from.

[elicn]

@technikelly, thanks for the fixes.
Please see my review comments inline.

I believe we would need to get rid of those repeating copyright notices, as @xwings suggested we can include your name and your employer's name in the contributors list.

[technikelly]

Thank you @elicn and @xwings for explaining about the issue with copyright notices. I was able to get an exception, so I've removed them. I've also implemented the changes requested by @elicn (thank you for the review!). As it stands now, the changes in this PR consist of:

• moving the heap creation to the blob OS module, and making that heap creation optional based on the existence profile values heap_address and heap_size
• Test code
• Example showcasing the modifications needed to enable code coverage

As stated in the comments, let me know if you would prefer not to include an example with this PR.

I also wanted to point out that I modified the two instances of uboot_bin.ql so that they contain heap_address so that the code change wouldn't break the uboot example or test

[elicn]

Everything looks good to me. Thanks @technikelly.
We'll merge the changes to rootfs first to allow the tests here to pass.

[technikelly]

Great, thank you @elicn! I just enabled the test and updated rootfs.

[elicn]

@xwings, pending your review.

[xwings]

Looks good to me.

[xwings]

Merged and thanks! Again, welcome to Qiling Framework