Skip to content

Conversation

@OhMaley
Copy link
Collaborator

@OhMaley OhMaley commented May 23, 2024

A brief description of the purpose of the changes contained in this PR.

This piece of code avoids crash when trying to access the size of a file that does not exist anymore in the storage. If an error is caught it then clears the file_size and data_file attribute in the DB.

Issues this PR resolves

Should solve Problem 1 of issue #1451

Simple test

A simple test would be to:

  • create a private competition
  • upload a submission
  • change in the database (manually or by using the django admin interface) the file_size attribute to -1 of one of the submission's dataset (table datasets_data)
  • remove in the storage (by using the minio interface) the corresponding file (its path is the value of the column data_file of the row in which you changed the file_size attribute)
  • Open up the shell_plus console inside the container of django
  • Call the save() method of the dataset object you modified. It will do some checks on the existence of the file in the storage and do the necessary updates. It should be something in those lines:
docker-compose exec django ./manage.py shell_plus
>>> d = Data.objects.get(id=<your_dataset_id>)
>>> d.save()
  • Check that their is no crash and that some changes occured in the DB

Use case test

  • Remove a file from the storage
  • Wait for the next create_storage_analytics_snapshot task to be ran (or trigger it manually)
  • Check that everything worked correctly

Checklist

  • Code review by me
  • Hand tested by me (on a local setup)
  • I'm proud of my work
  • Code review by reviewer

@OhMaley OhMaley requested review from ObadaS and acletournel May 30, 2024 14:13
@Didayolo Didayolo changed the base branch from develop to bugfix/storage/monitoring-tool/handle-inexistant-object May 31, 2024 14:38
@Didayolo
Copy link
Member

I merge this to a branch on codalab/codabench to trigger automatic testing.

@Didayolo Didayolo merged commit 696a024 into codalab:bugfix/storage/monitoring-tool/handle-inexistant-object May 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants