V1.5 compatibility - Unsupported features

[Didayolo]

This issue regroups several reported issue of compatibility and missing features with V1.5 (CodaLab Competitions).

My submissions

• Show the submission score
  Solved by: Score shown in submission panel for user and admin #877
• Submissions left: [Feature] Number of submissions left #1015
  Solved by: Submissions panel - show remaining submissions count #1094

Leaderboard

Set number of digits/decimals shown

• Leaderboard: limit/specify significant digits from leaderboard's level, For exmaple in V1, https://github.com/codalab/codalab-competitions/wiki/Organizer_Leaderboard, numeric_format is used
  Solved by: leaderboard score precision added #834

Computation columns

• v1 competition unpacker not creating computation columns #368

Linked to #406 and #851

I have a V1/V2 challenge with multiple tasks. The final rank of a participant is computed regarding its average score on all tasks.

It works on Codalab V1:

But the exact same bundle and submission on Codalab V2 gives:

Here is the YAML configuration of the leaderboard:

leaderboard:
  columns:
    auc_classification:
      label: Classification (AUC ROC)
      leaderboard: &id001
        label: RESULTS
        rank: 1
      rank: 2
      sort: desc
    acc_selection:
      label: Feature Selection (accuracy)
      leaderboard: *id001
      rank: 3
      sort: desc
    acc_causality:
      label: Causal Inference (accuracy)
      leaderboard: *id001
      rank: 4
      sort: desc
    ave_score:
      label: < Rank >
      leaderboard: *id001
      rank: 1
      sort: asc
      computed:
        operation: Avg
        fields: auc_classification, acc_selection, acc_causality
  leaderboards:
    RESULTS: *id001

Download buttons, HTML files

Related:

• Transparency of docker images and scoring programs #764
  Solved by:
  docker image name in competition page #944
  Download buttons #970

Public data and starting kit

• [V1 Bundle] No public data and starting kit in "Files" tab #277 (taken care by Transparency of docker images and scoring programs #764)
  Solved by: Download buttons #970

In V1, after uploading the bundle, it automatically adds "Starting Kit" and "Public data" to the Files tab:

With the exact same bundle in V2, it doesn't:

Example bundle.

HTML documentation not interpreted

This issue is not specifically related to v1 bundles, it also happen with v2 bundle. However we can consider it a v1 compatibility issue because all v1 bundles use HTML, while you can use Markdown in v2 bundles.

• HTML not interpreted #836
  Partially solved by html is now interpreted in compettion pages #878
  Fully solved by HTML - Markdown - Latex : Rendering Fixed #1090

[Didayolo]

I don't think it is solved. Most v1 bundles have the "rank" field and are working fine. Maybe we need a default value so all bundles work fine. I will move the information into a new issue (#932).

[ihsaan-ullah]

@Didayolo this point is not clear

Assets: a directory in the competition bundle containing files to which the doc files can link to.

[Didayolo]

Yes, it was an old point made by the previous devs so I did not wanted to remove it. I think the idea is to have a point (a folder) in the bundle that can be used to store files (images, etc.) for the documentation pages. Maybe we should remove it.

[ihsaan-ullah]

@Didayolo

Is this related to #406? Or is it something else

Computation columns

• v1 competition unpacker not creating computation columns #368

Linked to #406 and #851

I have a V1/V2 challenge with multiple tasks. The final rank of a participant is computed regarding its average score on all tasks.

[Didayolo]

@ihsaan-ullah

Yes, I think so. In CodaLab, you could define a column computing the average rank in the competition.yaml.
Solving #406 (which is not trivial) would solve this point too.

[ihsaan-ullah]

@ihsaan-ullah

Yes, I think so. In CodaLab, you could define a column computing the average rank in the competition.yaml. Solving #406 (which is not trivial) would solve this point too.

Can we shift this explanation to #406 so that this issue can be closed (one less issue = happiness) and to track related stuff in 406 only