tools: add benchmark for loading sets

[pzmarzly]

Measure time it takes to load a chain that contains a large set. This will be a common operation (especially since sets can update).

Requires #532.

$ rm -rf build && cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=release -DNO_DOCS=1 -DNO_CHECKS=1
$ ninja -C build benchmark_bin bfcli
$ sudo build/output/sbin/benchmark_bin \
    --cli build/output/sbin/bfcli \
    --srcdir . \
    --outfile /tmp/load.json \
    --filter chain_load

...
Run on (4 X 2396.4 MHz CPU s)
...
chain_load__ip4_saddr__x_elem_set/8/manual_time           7.20 ms         7.08 ms          106 load chain, ip4.saddr, 8 elements set
chain_load__ip4_saddr__x_elem_set/128/manual_time         7.69 ms         7.61 ms          107 load chain, ip4.saddr, 128 elements set
chain_load__ip4_saddr__x_elem_set/32768/manual_time       55.3 ms         61.5 ms           12 load chain, ip4.saddr, 32768 elements set

[github-actions]

Claude review of PR #535 (64c0686)

Suggestions

• SetIterationTime before error check — tools/benchmarks/main.cpp:235 — state.SetIterationTime() is called before verifying bf_chain_set succeeded; reordering would make the intent clearer

Nits

• Keyword-only params for Stats.failure() — tools/benchmarks/bfbencher:65 — inserting commit_sha before from_cache changes positional contract; consider keyword-only (*)
• Document commits ordering assumption — tools/benchmarks/bfbencher:1079 — executor.commits[-1] assumes topo-sort oldest-first; a brief comment would help readers
• Comment on iteration pattern divergence — tools/benchmarks/main.cpp:216 — for (auto _: state) differs from KeepRunningBatch used elsewhere; brief rationale comment would help

Workflow run

[pzmarzly]

@claude review

[github-actions]

Claude: suggestion: state.SetIterationTime() is called before checking whether bf_chain_set succeeded. If the load fails, a meaningless duration is recorded before SkipWithError discards it. While Google Benchmark discards results from skipped iterations, reordering makes the intent clearer:

if (ret < 0) {
    state.SkipWithError("failed to load chain");
    break;
}

state.SetIterationTime(
    std::chrono::duration<double>(end - start).count());

[github-actions]

Claude: nit: for (auto _: state) is the first use of this iteration pattern in the file; every other benchmark uses while (state.KeepRunningBatch(...)). A one-line comment explaining why a different pattern is needed here (manual timing of a heavy kernel operation that cannot be batched) would help readers understand the divergence.

[pzmarzly]

manual timing of a heavy kernel operation that cannot be batched

They probably can be batched though. The change from while to for was done for me by local Claude Code and I didn't catch it, but now it is a good moment to ask which one is more correct. Quentin what do you think? I'm leaning towards regular KeepRunningBatch, but I don't understand GBenchmark well.

[qdeslandes]

for (auto _: state) is correct, so let's keep it. This lets Google Benchmark define the number of iterations to run the benchmark for.

We don't use it to benchmark BPF programs, because BPF_PROG_TEST_RUN also accept an iterations argument, so we use it instead and use KeepRunningBatch() to check if we need more iterations. That ensures benchmarks are run with a warm cache, which is more reflective of real-world use cases.

[qdeslandes]

This can be done outside of the benchmarking loop.

[qdeslandes]

Use

{
    benchmark::ScopedPauseTiming pause(state); // Pauses timing
    ret = bf_chain_flush(chainName.c_str());
    if (ret < 0) {
        state.SkipWithError("failed to flush chain");
        break;
    }
}

So you don't have to manually set the iteration time.

[qdeslandes]

chain_set__ip4_saddr__x_elem_set

[qdeslandes]

for (auto _: state) is correct, so let's keep it. This lets Google Benchmark define the number of iterations to run the benchmark for.

We don't use it to benchmark BPF programs, because BPF_PROG_TEST_RUN also accept an iterations argument, so we use it instead and use KeepRunningBatch() to check if we need more iterations. That ensures benchmarks are run with a warm cache, which is more reflective of real-world use cases.

[qdeslandes]

Let's use different values to have a clear overview of the perf here:

• 1 << 3 -> 8
• 1 << 16 -> 65k
• 1 << 22 -> ~4M