Resource interface : Cleanup and Quota

[ihsaan-ullah]

@ mention of reviewers

@Didayolo

A brief description of the purpose of the changes contained in this PR.

This PR solves the following points

• show remaining quota (done but hidden til functionality is implementad by Tristan)
• show unused tasks and delete option
• show unused datasets and programs and delete option
• show unused submissions and delete option
• show failed submissions and delete option
• implement delete api for unused tasks
• implement delete api for unused datasets and programs
• implement delete api for unused submissions
• implement delete api for unused failed submissions
• refresh resources after cleanup
• refresh quota and cleanup after adding new submission/dataset/task
• refresh quota and cleanup after deleting single submission/dataset/task
• refresh quota and cleanup after deleting multiple submission/dataset/task

Issues this PR resolves

#713 -> Interface -> Points: 12, 13, 14, 15, 16

UI for quota management and cleanup (A "progress bar" for the storage quota)
Delete unused tasks
Delete unused datasets and programs
Delete unused submissions
Delete failed submissions

Screenshots

with quota progress bar (hidden now)

Working UI based on new api for getting cleaning statistics

Checklist

• Code review by me
• Hand tested by me
• I'm proud of my work
• Code review by reviewer
• Hand tested by reviewer
• CircleCi tests are passing
• Ready to merge

[ihsaan-ullah]

To discuss

• the difference between Submission and Data with type submission
• failed Submission when deleted still remains as Data

[Didayolo]

This looks very nice. I need some time to check this.

Just a quick question: what values did you use for the quota bar? It is just a fake quota or do you really compute some storage stuff here?

[ihsaan-ullah]

Just a quick question: what values did you use for the quota bar? It is just a fake quota or do you really compute some storage stuff here?

Ahh, this is fake :D It just uses 25% right now

[Didayolo]

I rebased this pull request to avoid migration issues.

[Didayolo]

@ihsaan-ullah When I load the interface, I get the following error:

[ihsaan-ullah]

I need more details. Error logs from docker django

[Didayolo]

Here are the Django logs:

codabench-django-1  | 
codabench-django-1  | The above exception was the direct cause of the following exception:
codabench-django-1  | 
codabench-django-1  | Traceback (most recent call last):
codabench-django-1  |   File "/usr/local/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
codabench-django-1  |     response = get_response(request)
codabench-django-1  |   File "/usr/local/lib/python3.8/site-packages/django/core/handlers/base.py", line 115, in _get_response
codabench-django-1  |     response = self.process_exception_by_middleware(e, request)
codabench-django-1  |   File "/usr/local/lib/python3.8/site-packages/django/core/handlers/base.py", line 113, in _get_response
codabench-django-1  |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
codabench-django-1  |   File "/usr/local/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
codabench-django-1  |     return view_func(*args, **kwargs)
codabench-django-1  |   File "/usr/local/lib/python3.8/site-packages/django/views/generic/base.py", line 71, in view
codabench-django-1  |     return self.dispatch(request, *args, **kwargs)
codabench-django-1  |   File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 495, in dispatch
codabench-django-1  |     response = self.handle_exception(exc)
codabench-django-1  |   File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 455, in handle_exception
codabench-django-1  |     self.raise_uncaught_exception(exc)
codabench-django-1  |   File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 492, in dispatch
codabench-django-1  |     response = handler(request, *args, **kwargs)
codabench-django-1  |   File "/usr/local/lib/python3.8/site-packages/rest_framework/decorators.py", line 54, in handler
codabench-django-1  |     return func(*args, **kwargs)
codabench-django-1  |   File "/app/src/apps/api/views/quota.py", line 19, in user_quota_cleanup
codabench-django-1  |     unused_tasks = Task.objects.filter(
codabench-django-1  |   File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 392, in count
codabench-django-1  |     return self.query.get_count(using=self.db)
codabench-django-1  |   File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/query.py", line 504, in get_count
codabench-django-1  |     number = obj.get_aggregation(using, ['__count'])['__count']
codabench-django-1  |   File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/query.py", line 489, in get_aggregation
codabench-django-1  |     result = compiler.execute_sql(SINGLE)
codabench-django-1  |   File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1142, in execute_sql
codabench-django-1  |     cursor.execute(sql, params)
codabench-django-1  |   File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
codabench-django-1  |     return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
codabench-django-1  |   File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
codabench-django-1  |     return executor(sql, params, many, context)
codabench-django-1  |   File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
codabench-django-1  |     return self.cursor.execute(sql, params)
codabench-django-1  |   File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
codabench-django-1  |     raise dj_exc_value.with_traceback(traceback) from exc_value
codabench-django-1  |   File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
codabench-django-1  |     return self.cursor.execute(sql, params)
codabench-django-1  | django.db.utils.ProgrammingError: more than one row returned by a subquery used as an expression
codabench-django-1  | 

Also, I think it would be cleaner to have this new interface at the bottom of the page, and with some (small) space between the two interfaces.

[ihsaan-ullah]

Moved to bottom

[ihsaan-ullah]

@Didayolo with the latest change, it should be working.

[Didayolo]

I merge develop to have the last changes here.

[Didayolo]

This works fine, I merge it.

@ihsaan-ullah Maybe we need to add CircleCI tests for this? What do you think?

[ihsaan-ullah]

I am creating new issue for the tests