Transparency of docker images and scoring programs

[Didayolo]

It is shown in #723 that the starting kit and public data buttons do not exist anymore in Codabench.

More generally, we would like to add a "transparency" feature. Organizers could have option to decide what they want to make public: the competition docker image, the scoring program, ingestion program, input data, etc.

Each piece made public would have a download button from the competition web pages.
The docker image could simply be shown using its DockerHub tag.

The sample submission (solution) should also be subject to this transparency, not only by being downloadable, but by being automatically run and evaluated.

The fact that CodaLab Competitions and Codabench are open-source is a crucial point in making these platforms great tools for the scientific community. This way, everybody can review the code and the benchmarks and competitions are more valuable.
However, the code and data used by the organizers is not necessarily open. Even if some organizers provide the code via Github with a link in the documentation, one can not ensure that it is really the code used to evaluate the submissions, making the final results less transparent. The goal of this feature is to have a real transparency, when enabled, between the organizers and the participants.

---

Similar ideas:

• Organizers contact email

It would be good to have a place, typically a field in the editor, for the organizer's email address. This is important because it is very common that participants contact the administrators of the platform instead of the organizers for competition specific questions because they can't find their contact.

Inside parenthesis next to "organized by":

A sentence in the forum:

• Link to the report

It would be interesting to have a way to show the link to a competition report. The link could be accessible from an icon benchmark listing, and also well visible in the competition home page.

• Reward (prize) of the competition

As stated in #954, we should be able to set a reward (e.g. 4000$) from the editor. I don't think we want it to be doable from the competition.yaml file because that should be done only one the "production" version of the competition, after the tests. It could then be shown in the interface, in the home page of the challenge and in the public challenges list.

In the list:

Below "participants" and "submissions" using the same styling:

TODOs:

Editor / YAML fields

• Display docker image name in competition
  Solved by: docker image name in competition page #944
• Add a YAML and editor field Organizers contact where the organizers put an email address that will be shown in the interface of the competition
  Solved by: Competition - reward, contact_email, report #1063
• Editor field Reward (free-form string?)
  Solved by: Competition - reward, contact_email, report #1063
• Editor field Challenge report
  Solved by: Competition - reward, contact_email, report #1063

Files tab

First PR #970:

• For non-approved participants: Not accessible
• For approved participants: Showing objects made available by organizers
• For organizers: Showing all object and a column showing what is available or not to participants

Next TODOs:

• In editor / YAML, two options: make programs available, make input data available (Options to make programs input data available #1136)
• Add documentation about the fields to add starting kit and public data in the file competition.yaml
• Solutions appear even when not registered to the competition. Should be treated in the same way as public data and starting kit (show only to registered participants and organizers, and always in available state)
• The No Files Available Yet text is always showing (no files available fix #1041)
• Fix the column Available that always displays "Yes" (see Download buttons #970) (Options to make programs input data available #1136)
• Green checks instead of yes (see screenshot below, like the resources interface). These ticks are just markers and should not be clickable. (Options to make programs input data available #1136)

See the discussion below for better understanding

[ihsaan-ullah]

@Didayolo we should discuss this to make a plan for these features

[Didayolo]

@ihsaan-ullah Yes, I think some details should be discussed to agree on what we want to do. Also, it is important to discuss with Benjamin who is working on adding the "Files" tab to download data and starting kit.

[ihsaan-ullah]

@Didayolo Benjamin is working on download download buttons but the decision is not finalized yet about where to inject starting kit and sample submission

[Didayolo]

Let's discuss this during the next meeting. I put a point in the agenda.

[Didayolo]

@ihsaan-ullah

Docker image could be shown here:

[Didayolo]

@bbearce

A good solution to how organizers can chose what is available for participants or not is to use the "files" tab for this:

This tab should be:

• For non-approved participants: Not accessible
• For approved participants: Showing objects made available by organizers
• For organizers: Showing all object and a column showing what is available or not to participants (and an interface to choose to make each object available/unavailable)

Warning: public and available are different concepts.

[Didayolo]

For non-approved, we could have the same thing as the submissions tab:

(check the behavior when revoked, etc.)

[ihsaan-ullah]

@Didayolo

Add a yaml / editor field Organizers contact where the organizers put an email address that will be shown in the interface of the competition

Is this required? Normally we put contact info in the html/markdown pages

Think about the possibility of having a Challenge report field or something

What is challenge report?

[Didayolo]

Is this required? Normally we put contact info in the html/markdown pages

We need to think about it. The problem is that many organizers forget to put their contact in the documentation. Also, as it is free form, the contact info can be hard to find when it is somewhere.

What is challenge report?

I was thinking about after the competition end: an analysis paper, or a blog post sharing the results, etc.