Compare with some closed API-based models

[neubig]

These should be compared in our scaffold.

• GPT-5?
• Claude Opus?

[adityasoni9998]

The evaluations are done with GPT-5, our scaffold and with an additional reminder message after the agent exhausted its max. steps. Analysis of results:

• GPT-5 is considerably stronger than our best trained models. The performance difference follows the trend: function > class > file
• Our scaffold + GPT-5 has better performance than RepoNavigator + Claude Sonnet 4.5/3.7. This is a concrete evidence of the fact that a simpler scaffold is enough for stronger LLMs to do the task.

SWE-Bench Verified:

File-level F1 Score:  0.7818
Module-level F1 Score:  0.6117
Function-level F1 Score:  0.5479

SWE-Bench Pro:

File-level F1 Score:  0.6118
Module-level F1 Score:  0.4220
Function-level F1 Score:  0.3586

SWE-Bench Lite:

File-level F1 Score:  0.7777
Module-level F1 Score:  0.6786
Function-level F1 Score:  0.6112

[adityasoni9998]

The evaluations are done with Claude-Sonnet-4.5, our scaffold and with an additional reminder message after the agent exhausted its max. steps. Analysis of results:

• Claude-Sonnet-4.5 is considerably stronger than our best trained models and than GPT-5. The performance difference follows the trend: function > class > file
• Our scaffold + Claude-Sonnet-4.5 has significantly better performance than RepoNavigator + Claude Sonnet 4.5 (+18% points absolute improvement in function-level F1 score). This is a concrete evidence of the fact that a simpler scaffold is enough for stronger LLMs to do the task.

SWE-Bench Verified:

File-level F1 Score:  0.8201
Module-level F1 Score:  0.6719
Function-level F1 Score:  0.6178

SWE-Bench Pro:

File-level F1 Score:  0.6475
Module-level F1 Score:  0.4854
Function-level F1 Score:  0.4226

SWE-Bench Lite:

File-level F1 Score:  0.8187
Module-level F1 Score:  0.6962
Function-level F1 Score:  0.6111