Automate project dependencies & General optimizations

[Jean-Lessa]

UPDATE (2024-11-09): this PR now includes several improvements (mostly performance/RAM-wise) made in a former separate optimize branch that was merged locally today. In short we have:

• #include lines were revised across the WHOLE project - that alone reduced RAM usage significantly during compilation (see image below - tested with cmake .. && cmake --build . -- -j8, the gap in the middle is the divisory between src and tests)
• utils.h was considerably debloated:
  • SafeUint_t and SafeInt_t aliases were migrated to their respective contract/variables/safeXint.h files (no need to include utils.h anymore just to use those aliases
  • uint_t/int_t aliases and some int conversion functions (uintXToBytes, intXToBytes, bytesToUintX, bytesToIntX)
  • were migrated to a new file called utils/intconv.h
  • EVMC-specific utility functions were migrated to a new file called utils/evmcconv.h
  • String manipulation functions (padLeft/Right + Bytes equivalents, toUpper/toLower, cArrayToBytes) were moved to a new file called utils/strconv.h
  • Functions that remain (for now) in utils.h (those should be migrated too but I didn't get to them yet): stringToBytes, bytesToString, uintToBytes (the templated one), maybe appendBytes and bytesRequired
• The respective tests for all of the above were also updated (and some added)

NOTE: this was made BEFORE #135 was opened, there might be conflicts since that branch deprecated bytes::View and also made a bunch of other changes across the project. Whoever manages to get merged first, keep in mind there'll be a lot of fixing to be done after the fact.

NOTE 2: right now, running benchmark tests with -DBUILD_BENCHMARK=ON yields an AddressSanitizer error. This needs fixing, though it's an isolated problem - the rest of the tests are working just fine, it's just the [benchmark] tag that's borked.

---

This PR introduces a script in scripts/deps.sh that effectively decouples the project itself from its dependencies. So now we have a slightly more elaborate flow when setting up the environment, but we won't have to recompile the dependencies anymore for each clean build - just once in a fresh system, then off we go.

Keep in mind the assumed "minimum supported distro version" is now Debian 13 (Trixie) or greater - if you're on an older version or a distro that doesn't use APT, you'll have to do some things manually. I've updated the README with more details. As usual, docs will be updated accordingly after merge.

"Gimme a TL;DR"

If you're using Docker: simply rebuild your container. Dockerfiles were updated to use the script directly, dependencies will be handled automatically

If you're doing it manually:

• First, run deps.sh --check to know if your system has all the required dependencies installed
• If there are any dependencies missing (the script will show them as "not found"), simply run deps.sh --install or install them yourself by hand to either /usr (for repo packages) or /usr/local (for compiling from source)
• Check once again if you want, then compile the project

"Ok, now lift the hood and show me the engine!"

Right, so our dependencies are now divided into a few logical categories:

• Toolchain binaries - executables required for compiling the project, taken directly from the system repos (some are required, some are optional - the script shows the division but installs all of them anyway)
• Internal libraries - libraries that can be taken directly from the system repos
• External libraries - libraries that do not exist in the repos and have to be manually fetched or compiled

The script is hardcoded to check into both /usr and /usr/local for those dependencies, giving preference to the latter if there's anything installed there (e.g. we need Boost 1.83 but you're on Debian 12 which only has Boost 1.74, you can manually compile a newer version and install it to /usr/local so it'll be detected first).

Any internal dependency you need that doesn't match the minimum in the repos will have to be installed this way (the script smartly skips this part if if doesn't find APT). External dependencies can still be installed through the script as those are fetched by git.

Likewise, the project's CMakeLists.txt now also checks the system for those dependencies, instead of pulling them via ExternalProject_Add like we've been doing up until now, so the script has become the "single source of dependency truth". Any modifications to the project's dependencies must be reflected in the script from now on.

Caveats to keep in mind

• The script does NOT check the minimum versions of the dependencies, only if they are installed - CMake does this partially, but I'm not sure if this can be automated in a simple way, so it's on the dev/user to check the versions beforehand
• The script does NOT deal with dependency cleaning/rebuilding (e.g. if we upgrade an external dependency's version in the script, the dev/user would have to manually delete the older version so it can be recompiled, assuming they're not using Docker) - if this becomes too much of a burden, let me know so I can think about putting a --clean flag or something of the sort
• I have no idea if this will break SonarQube's CI/CD as I've updated its Dockerfile as well (and we can't test it unless it is merged) - I hope it won't, but if it does please let me know This was already fixed

[fcecin]

Wonderful!!!

Would be cool to have a "rebuild-all.sh" in the project root, like:

echo "this script installs all dependencies and rebuilds BDK to build/"

echo "checking dependencies"

scripts/deps.sh --check

IF check fails:
echo "will install missing dependencies; enter your root password"
sudo scripts/deps.sh --install

echo "removing old BDK build if any"

rm -rf ./build

echo "rebuilding BDK with default options"

mkdir build
cd build
cmake ..
cmake --build . -- -j4

Works as an instant go-to for people who download software but don't read documentation.

[fcecin]

About a possible --clean flag, there is at least one sure-fire way to clean up, which is to reinstall the OS in your development partition. If everything works by e.g. starting from a fresh Ubuntu 24 machine and running scripts/deps.sh --install, then it's already golden. That also solves "minimum library version" problem: if starting from the fresh install of the latest Ubuntu LTS gets you the minimum versions, then that's the basics covered.