Questions regarding your paper's evalutation

[nbars]

Hello, congratulations on your paper -- nice work, interesting idea! I was interested in reproducing your results and was looking into your artifact for that. During this, I noticed a few things that I would like clarified.

A note before my questions: Since not all the code of your artifacts is part of your main GitHub repository and can only be found in the Docker image pulled as part of the reproduction setup, I created a GitHub repository that contains a copy of the data located in your Docker image adamstorek/fox:latest (sha256:11ac4f0ceb501d734af81aa4dc4d9ca7d4c87d55239ccf0218271d48bea8b78d) in /workspace/fuzzopt-eval. I'll use that to refer to specific parts of your code.

Coverage on standalone targets

You evaluated 15 standalone targets in your paper to compare against the state of the art. Table 4 displays the results of these experiments. I included the table below for convenience.

Reproduction steps I've performed

To check the setup used for evaluation, I followed the instructions you provided as part of your artifact. First, I pulled the provided Docker image via docker pull adamstorek/fox:latest (sha256:11ac4f0ceb501d734af81aa4dc4d9ca7d4c87d55239ccf0218271d48bea8b78d). Second, I spawned a Docker container using your image as described:

docker run --privileged --network='host' -d -it adamstorek/fox:latest
docker exec -it optfuzz_eval /bin/bash

Next, I switched to the /workspace/fuzzopt-eval/fuzzdeployment/targets folder and modified the set_all_targets.sh script, such that it builds libarchive (bsdtar) and ffmpeg, by setting TARGETS="ffmpeg libarchive". I chose these targets since you reported a coverage increase of up to 97.25% and 49.04% in your paper, respectively.
After executing set_all_targets.sh and once the build process terminated, I checked the resulting binaries.

For this matter, I executed the following command, which prints the size of the SanitizerCoverage section of each individual binary that has been built for the libarchive and ffmpeg target:

# ffmpeg binaries
as5827@19e7b435119d:/workspace/fuzzopt-eval/fuzzdeployment/targets$ find ffmpeg/binaries/  -type f -executable -print -exec bash -c 'readelf -S {} | grep guard' \;
ffmpeg/binaries/optfuzz_build/ffmpeg
 [28] __sancov_guards   PROGBITS         0000000008ddcd6c  08ddbd6c
ffmpeg/binaries/cmplog_build/ffmpeg
 [28] __sancov_guards   PROGBITS         0000000005d94494  05d93494
ffmpeg/binaries/aflpp_build/ffmpeg
 [28] __sancov_guards   PROGBITS         0000000005b29d54  05b28d54

# libarchive binaries
as5827@19e7b435119d:/workspace/fuzzopt-eval/fuzzdeployment/targets$ find libarchive/binaries/  -type f -executable -print -exec bash -c 'readelf -S {} | grep guard' \;
libarchive/binaries/optfuzz_build/bsdtar
 [28] __sancov_guards   PROGBITS         000000000072540c  0072440c
libarchive/binaries/cmplog_build/bsdtar
 [28] __sancov_guards   PROGBITS         00000000004636b4  004626b4
libarchive/binaries/aflpp_build/bsdtar
 [28] __sancov_guards   PROGBITS         00000000004243f4  004233f4

For each edge in the target, this map contains one entry that is 4 bytes in size. As we can see, the map sizes differ significantly. I believe that this is due to additional instrumentation added by your modified AFL++ pass (for example, here).

Computing the relative guard section size between your fuzzer (optfuzz), AFL++ (aflpp), and AFL++ cmplog (cmplog_build) yields the following results.

For ffmpeg, we are getting the following numbers:
optfuzz / aflpp = 0x0000000008ddcd6c / 0x0000000005b29d54 = 1.52
optfuzz / cmplog_build = 0x0000000008ddcd6c / 0x0000000005d94494 = 1.52

And for libarchive, we are getting the following ratios:
optfuzz / aflpp = 0x000000000072540c / 0x00000000004243f4 = 1.73
optfuzz / cmplog_build = 0x000000000072540c / 0x00000000004636b4 = 1.63

As we can see, again, the guard sections are quite different in size. Essentially, this means that the binaries compiled using your fuzzer have a considerable number of additional edges compared to the ones of AFL++. This, in turn, has ramifications for computing coverage over time (at least the way you do to generate the plots for your paper): Of course, if your fuzzer's binary has significantly more edges (added by your instrumentation), it is easy to cover more edges than the baseline fuzzer, for which the binary has fewer edges.

Looking at the script you are using for coverage computation (after the fuzzing runs), you are parsing AFL++'s fuzzing_stats or plot_data to calculate the coverage:

https://github.com/fuzz-evaluator/FOX-fuzzopt-eval-upstream/blob/23a3277c6604616157bb085dd26ba2f365780ff1/fuzzdeployment/process_results/parse_results.py#L305-L335

I don't think this comparison is fair: Since your binaries contain more than 50% additional edges for both targets (compared to the other fuzzer's binaries), this inflates your results artificially.

Now, all of the above is based on my superficial understanding so far (I'll definitely take a closer look in the coming days), so please let me know if there's anything I missed or understood wrong. I'd appreciate if you could outline why/how the issue I'm seeing is taken care of.

On another note, I'm curious if there is any place I can find your fuzzbench configuration so I can rerun your exact evaluation. Thanks in advance!

[adamstorek]

Hi @nbars,

Thank you so much for your interest in our work and question!

Please note that our the entire evaluation package is available not only on Docker Hub but also on Zenodo, as referenced in the artifact README. Hosting our evaluation pipeline on Zenodo was required for CCS artifact evaluation.

Regarding the consistency of edge counts, are you sure you are not comparing section offsets instead of section sizes? I followed your steps and when I run the following command to get sizes and offsets of the sections, optfuzz (FOX) and aflpp have the same __sancov_guard section size:

as5827@bug05:/workspace/fuzzopt-eval/fuzzdeployment/targets$ find libarchive/binaries/aflpp_build libarchive/binaries/optfuzz_build  -type f -executable -print -exec bash -c 'size -A -d -x {} | awk "NR==2 || /__sancov_guards/"' \;
libarchive/binaries/aflpp_build/bsdtar
section                  size       addr
__sancov_guards       0x25578   0x4243f4
libarchive/binaries/optfuzz_build/bsdtar
section                   size       addr
__sancov_guards        0x25578   0x72540c
as5827@bug05:/workspace/fuzzopt-eval/fuzzdeployment/targets$ find ffmpeg/binaries/aflpp_build ffmpeg/binaries/optfuzz_build  -type f -executable -print -exec bash -c 'size -A -d -x {} | awk "NR==2 || /__sancov_guards/"' \;
ffmpeg/binaries/aflpp_build/ffmpeg
section                   size        addr
__sancov_guards       0x2d4430   0x5b29d54
ffmpeg/binaries/optfuzz_build/ffmpeg
section                    size        addr
__sancov_guards        0x2d4430   0x8ddcd6c

In fact, the numbers that you have mentioned match the section addresses. Let me know if you can reproduce this on your side. I also ran the respective fuzzers briefly to generate the fuzzer_stats files. From these files, the total_edges field is consistent across aflpp, cmplog, and optfuzz (FOX). I’ve included screenshots below for reference:

Please let me know if you’re able to reproduce this behavior on your end. Additionally, in the part of our results collection script you referenced, where we extract the edges_found field, we also have an assertion to ensure that the total_edges field remains consistent across all fuzzers for each project.

For the FuzzBench configurations, please refer to our evaluation framework where we have provided all the necessary scripts to run the FuzzBench targets. We chose this configuration so that the same standard coverage metric edges_found as reported by AFL++ was used across both our standalone and FuzzBench targets.

I’d be happy to address any further questions you may have!

Best,

Adam

[vanhauser-thc]

Hi @adamstorek
there is this fuzzbench run: https://www.fuzzbench.com/reports/experimental/2024-03-11-fox-eval-update/index.html
How current is that evaluation?

[adamstorek]

Hi @vanhauser-thc,

thank you for your interest in our work! The FuzzBench run you mention is not a current version, it is an older version that requires gllvm. The current version (the version used for our paper) does not require gllvm. Hope this helps!

Best,

Adam

[nbars]

Thanks for the detailed answer! Indeed, I mixed things up here.