Competition Auto-Run Submissions

[ihsaan-ullah]

@ mention of reviewers

@Didayolo

A brief description of the purpose of the changes contained in this PR.

Organizers can enable or disable auto_run_submissions feature from competition setting to allow participant's submissions to auto-run (the default way) or to require an organizer to run the submission manually (the new feature)

How to set auto_run_submissions

1. You can set it in the competition settings:

2. You can set it in the yaml file:

Note: default value is True so that the

How do participants know their submission are not auto-run?

Participants see a question mark with a message to tell them that organizers will run their submissions:

How can organizer run the submission?

Organizer can see a play button for submissions which should be executed manually. By clicking this button, the submission will run as a normal submission

Issues this PR resolves

• [New Feature] Toggle Auto-run submissions #1281

A checklist for hand testing

• Create a competition and disable auto_run_submissions either from competition setting or from competition yaml
• Submit a submission and check if you see a message that the submission will be run by organizer
• As an organizer, check that you see the play button in front of the submission and submission runs when you click it
• upload two competitions one with auto_run_submission set to true and other with false In the yaml file to see they are correctly shown in the competition setting
• Enable/Disable auto_run_submissions from the setting to check that uploaded submissions follow the setting

Checklist

• Code review by me
• Hand tested by me
• I'm proud of my work
• Code review by reviewer
• Hand tested by reviewer
• CircleCi tests are passing
• Ready to merge

[cjh1]

Quick question, what is the difference between "Submitting and Submitted". To me I would expect as submission to be in the "Submitted" state after I have uploaded my submission?

[ihsaan-ullah]

Quick question, what is the difference between "Submitting and Submitted". To me I would expect as submission to be in the "Submitted" state after I have uploaded my submission?

The first status isSubmitting when you upload a submission. Submitted is used when submission is forwarded for running

[Didayolo]

@ihsaan-ullah
Tested and approved!

Last thing, I think you need to add this new feature in the documentation.

Typically here:

• https://github.com/codalab/codabench/wiki/Yaml-Structure

Maybe also in a page describing the Editor. Thanks!