Baselines for open source models

[kalyani7195]

Hello!
Thank you for open-sourcing the code base. I was wondering if it is possible to release the code for open source models such as llama and llava. I unfortunately do not have access to gemini/ claude/ gpt APIs. So it would be great if I could run the open sourced baselines.
Kalyani

[arjunmajum]

Quick update: we're adding code for LLaMA in PR #4.

[asuglia-alana]

Hello @arjunmajum,

On a related note, I believe that to aid reproducibility of your evaluation, it would be best to use an open-source model as an evaluator instead of using GPT-4 which could change and might be deprecated one day. Why not use something Llama-3 as an evaluator?

In general, I think that the best way to solve this issue is to release the predictions of the models you have used so that everybody can run the evaluation again with the latest and greatest model available. Would it be possible for you to release the predictions?

Thanks,
Alessandro

[asuglia-alana]

Hello @arjunmajum any updates?