File Sizes cleanup

[ihsaan-ullah]

@ mention of reviewers

@Didayolo

A brief description of the purpose of the changes contained in this PR.

The way of dealing with file sizes was not uniform in the platform. At some places size was stored in Bytes and in other places in KiB. This is now fixes and we store bytes in the db and use KB/MB/GB instead of KiB/MiB/GiB. The size formatter used were different at different places. Now we have one formatter named pretty_bytes that is declared in both javascript and python.

Issues this PR resolves

• Quota related issues #1748 -> Find the use of GiB/MiB/KiB in storage analytics and other places and replace with GB/MB/KB
• Quota related issues #1748 -> Size of files is formatted at different places with different functions, use only one size formatter everywhere

Important Note

I have left some size unit conversion in the following files because there is a confusion in what is going on in these files. I cannot see any data in the analytics to match the data with code BUT once analytics start working then I will check these

• src/static/riot/analytics/_competitions_usage.tag
• src/static/riot/analytics/_usage_history.tag
• src/static/riot/analytics/_users_usage.tag
  All these files can be found here: https://github.com/codalab/codabench/tree/master/src/static/riot/analytics

Important Todos for deployment:

We have some critical changes here so before deployment we should run the following 3 blocks of code to get the last ids of Data, Submission and SubmissionDetail

# Get the maximum ID for Data
from datasets.models import Data
latest_id_data = Data.objects.latest('id').id
print("Data Last ID: ", latest_id_data)

# Get the maximum ID for Submission
from competitions.models import Submission
latest_id_submission = Submission.objects.latest('id').id
print("Submission Last ID: ", latest_id_submission)

# Get the maximum ID for Submission Detail
from competitions.models import SubmissionDetails
latest_id_submission_detail = SubmissionDetails.objects.latest('id').id
print("SubmissionDetail Last ID: ", latest_id_submission_detail)

After we have the latest ids, we should deploy and run the 3 blocks of code below to fix the sizes i.e. to convert all kib to bytes to make everything consistent. For new files uploaded after the deployment, the sizes will be saved in bytes automatically that is why we need to run the following code for older files only.

# Run the conversion only for records with id <= latest_id
from datasets.models import Data
for data in Data.objects.filter(id__lte=latest_id_data):
    if data.file_size:
        data.file_size = data.file_size * 1024  # Convert from KiB to bytes
        data.save()

# Run the conversion only for records with id <= latest_id
from competitions.models import Submission
for sub in Submission.objects.filter(id__lte=latest_id_submission):
    updated = False  # Track if any field is updated

    if sub.prediction_result_file_size:
        sub.prediction_result_file_size = sub.prediction_result_file_size * 1024  # Convert from KiB to bytes
        updated = True
    
    if sub.scoring_result_file_size:
        sub.scoring_result_file_size = sub.scoring_result_file_size * 1024  # Convert from KiB to bytes
        updated = True

    if sub.detailed_result_file_size:
        sub.detailed_result_file_size = sub.detailed_result_file_size * 1024  # Convert from KiB to bytes
        updated = True
    
    if updated:
        sub.save()

# Run the conversion only for records with id <= latest_id
from competitions.models import SubmissionDetails
for sub_det in SubmissionDetails.objects.filter(id__lte=latest_id_submission_detail):
    if sub_det.file_size:
        sub_det.file_size = sub_det.file_size * 1024  # Convert from KiB to bytes
        sub_det.save()

Checklist

• Code review by me
• Hand tested by me
• I'm proud of my work
• Code review by reviewer
• Hand tested by reviewer
• CircleCi tests are passing
• Ready to merge

[Didayolo]

@ihsaan-ullah

For the manual intervention, I replaced:

• <latest_data_id> by latest_id_data
• <latest_sub_id> by latest_id_submission
• <latest_sub_det_id> by latest_id_submission_detail

Is that correct?

---

The manual intervention raises this error:

    if sub.file_size:
AttributeError: 'Submission' object has no attribute 'file_size'

I had one more migration file after a makemigrations, but the problem persists even including this one:

# Generated by Django 2.2.28 on 2025-03-21 13:41

from django.db import migrations, models


class Migration(migrations.Migration):

    dependencies = [
        ('competitions', '0053_auto_20250218_1151'),
    ]

    operations = [
        migrations.AlterField(
            model_name='submissiondetails',
            name='file_size',
            field=models.DecimalField(blank=True, decimal_places=2, max_digits=15, null=True),
        ),
    ]

[ihsaan-ullah]

For the manual intervention, I replaced:

• <latest_data_id> by latest_id_data
• <latest_sub_id> by latest_id_submission
• <latest_sub_det_id> by latest_id_submission_detail

Is that correct?

Yes, I have updated the code in the main comment with the right var name

[ihsaan-ullah]

@Didayolo I have

• fixed the manual intervention code
• added the missing migration
• resolved merge conflicts.

For this PR deployment we have to stop users from using the platform. I think @ObadaS knows how to do it

[Didayolo]

@ihsaan-ullah

Great, thanks.

Not sure if it is related to this PR (or the other?), but the quota is written "B" instead of "GB":