15 GB quota; cannot delete submissions

[gholste]

Why can I not delete my own competition submissions?

I have hit the 15 GB storage quota from making routine submissions to https://www.codabench.org/competitions/4779/. As suggested, I went to my "Resources" tab to clean up my storage only to find that I cannot delete any of my submissions (error "Cannot delete submission: submission belongs to an existing competition").

My account is effectively locked -- I cannot make any more submissions on CodaBench. Is this really the intended behavior?

Many of these submissions finished running but internally failed, meaning they were not given a score. Perhaps if these were marked "Failed", then they could be deleted? Even so, I don't understand the design choice to force submissions to stay up -- why do I not have control over this given the strict 15 GB limit?

[Didayolo]

Hi @gholste,

This is indeed a known problem that we intent to solve soon. Letting participants remove their own submissions come with downside for competition organizers.

The problem is discussed in the issue #713:

• Have an user interface to manage and delete submissions
  Solved by: Resource interface : Cleanup and Quota #918
• Actually, submissions can't be removed if there are part of a benchmark (which makes sense to avoid spamming > submissions when there is a daily limit).
• Solution: submissions are counted on the quota of the participants, but they can remove a submission (even if it's part of a benchmark) after a delay of e.g. 7 days.
• WARNING: disallow removing leaderboard submissions
• Other idea: organizer phase setting to allow or disallow deleting submissions

Thank you for reporting this problem again. This is an important feature to resolve, so I'll add the "P1" tag here.

[Didayolo]

Or simply:

• Allow removing
• Removing does not affect counter (competition limit)
• But it does affect the user storage quota
• Disallow removing leaderboard submissions (avoiding potentially problematic behaviors)

We can do this with a kind of "soft deletion", so we do not need to re-write the counters.

[gholste]

@Didayolo Thanks for the response. The point about skirting the submission limit to spam submissions makes sense. However, this policy has turned me from someone using your service to someone who literally cannot use CodaBench in one day. I can't imagine this is what you or the organizers want.

To clarify, is the solution of allowing users to remove submissions after 7 days a proposed solution or actually currently implemented?

Also, here is a suggestion: decouple the submission count from the submissions contributing to the quota. I.e., let users delete whatever they want; when they delete a submission, simply don't decrement the submission counter.

If you don't want to completely lose track of the mapping from submission count to the database of submissions, perhaps there is a way to delete all submission content except the ID so that deleted submissions still appear in the database (with nan/empty contents) but do not contribute meaningfully to storage.

[Didayolo]

To clarify, is the solution of allowing users to remove submissions after 7 days a proposed solution or actually currently implemented?

It was a proposed solution, but we are now thinking of a simpler solution (see the comment above).

Also, here is a suggestion: decouple the submission count from the submissions contributing to the quota. I.e., let users delete whatever they want; when they delete a submission, simply don't decrement the submission counter.

Exactly, we reached the same idea.

Also, if an organizer re-run or duplicate submissions, it should not affect the counters.

[Didayolo]

@gholste While we are still working on this matter, would you like that we manually increase your quota? If yes, please share your Codabench username.

[gholste]

Sure! My username is gholste. I appreciate the responsiveness, and thank you for working on this.

[ihsaan-ullah]

Soft deletion is a possible solution. To keep the submission but delete the data to freeup some space. Currently organizers can completely delete a submission.

• Do we want to change that and only soft delete submissions?
• Do we want to show soft deleted submissions in the admin view?

[Didayolo]

Do we want to change that and only soft delete submissions?

No, I don't think so, it is good that organizers have a powerful deletion, especially while debugging.

Do we want to show soft deleted submissions in the admin view?

Yes, I think so, this way organizers keep an eye on what happened in their competitions.