API break runner is too noisy

[IAlibay]

I love the idea, but I think it needs a bit of refinement.

Is there any way to do an API break on the diff against main? As far as I'm aware it's not and it's complaining on changes done in other commits.

[mikemhenry]

It can be improved, I think posting once and also updating if there are no breaking changes is good, so it should work like codecov IMHO.

Can you link to a spot where it is

complaining on changes done in other commits.

[IAlibay]

@mikemhenry here's an example: https://github.com/OpenFreeEnergy/openfe/actions/runs/12296910926/job/34316900779?pr=1047

From the contents it looks like it's not liking a test thing that Josh isn't changing (also is it worth ignoring tests in the diff?)

[mikemhenry]

Okay, I will get the bot working better and investigate the weirdness with that run.

Looking at the docs: https://mkdocstrings.github.io/griffe/guide/users/checking/#ci-github

I want to make sure I check out the source code in a way it has everything needed to compare. Since it only works on the diff, it is weird that it is complaining about things changing. I want to dig into it a bit since I am guessing something weird is happening with the way checkout the code and the diff is calculated.

I like keeping the tests checked since if we are modifying a test to match upstream behavior in a way that "breaks" the test API, that may warrant a change log entry so users know that they need to modify their code. It could be something like "Note: Newer versions of rdkit (2025.01.24) require this new argument in Chem.Mol to maintain old behavior". While we will get some false positives, especially since our test API isn't public, I rather catch the few cases where it is important and let the reviewer of the PR note when it can be ignored.

[mikemhenry]

Ah,

By default, Griffe will compare the current code to the latest tag

I will update it so it uses the last commit on main for the diff, that way we don't accumlate "breaks" as we go