Skip to content

Conversation

@bbearce
Copy link
Collaborator

@bbearce bbearce commented Sep 27, 2022

@ mention of reviewers

@Didayolo

A brief description of the purpose of the changes contained in this PR.

Attempting to make postgres db backups and upload to MINIO storage. Generally everything works, but the names of variables in files and the names of containers\services\folders changed as we renamed the repo to "codabench" from "competitions-v2".

Issues this PR resolves

None.

A checklist for hand testing:

First Set of Tests:

DB_NAME=...
DB_USERNAME=...
DB_PASSWORD=...
DUMP_NAME=test_dump.dump
docker exec codabench-db-1 bash -c "PGPASSWORD=$DB_PASSWORD pg_dump -Fc -U $DB_USERNAME $DB_NAME > /app/backups/$DUMP_NAME.dump"
  • Shell into django container docker-compose exec django bash
  • Execute command to upload dump:
root@9270c8888b98:/app# python manage.py upload_backup test_dump.dump
Uploading backup '/app/backups/test_dump.dump' to 'https://XXXXXXXXXXXXXXXXXX/codabench-private/backups/test_dump.dump?AWSAccessKeyId=XXXXXXXXXXXXXXXX&Signature=XXXXXXXXXXXXXXXXXXX&Expires=1664403807'
Success!
Removing local dump file '/app/backups/test_dump.dump'
  • Verify dump is in storage bucket:
    image

Second Set of Tests (Automated):

  • cd to codabench/bin and run pg_dump.py file:
$ bin/pg_dump.py
  • Check that an automatically generated dump is made and moved to storage and named as "%Y-%m-%d_%H-%M-%S.dump":
  • Following along with Creating and Restoring from Backup we can restore from backup.
container$ dropdb $DB_NAME -U $DB_USERNAME
container$ createdb $DB_NAME -U $DB_USERNAME
container$ pg_restore -U $DB_USERNAME -d $DB_NAME -1 /app/backups/<filename>.dump
  • Restart django and compute_worker containers as they seem to get confused.
docker-compose restart django
docker-compose restart compute_worker

Any relevant files for testing

I edited this file to go along with the checklist for testing and to keep docs updated.
https://github.com/codalab/codabench/wiki/Creating-and-Restoring-from-Backup

Misc. comments

My changes to https://github.com/codalab/codabench/wiki/Creating-and-Restoring-from-Backup are not permanent and there are versions we can go back to. I plan to live demo this to you all.

Checklist

  • Code review by me
  • Hand tested by me
  • I'm proud of my work
  • Code review by reviewer
  • Hand tested by reviewer
  • Ready to merge

@Didayolo Didayolo merged commit bd3ba41 into develop Sep 28, 2022
@bbearce bbearce deleted the database_backups branch October 2, 2022 21:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants