Thanks for this useful package! I am currently setting up both PkgBenchmark and BenchmarkCI in the same branch of my repo. As a result, the file benchmark/benchmarks.jl does not exist on the master branch. This causes the baseline run of benchmarks to fail on the pending PR.
I had to dig into the BenchmarkCI source code to figure out what was going wrong. It would be nice if a more informative error, or even just a warning, was raised when the baseline branch is not set up for benchmarks.