Skip to content

Paginate delete blockers to prevent OOM#7939

Open
foozleface wants to merge 1 commit intospecify:mainfrom
calacademy-research:cas/perf-delete-blockers-7515
Open

Paginate delete blockers to prevent OOM#7939
foozleface wants to merge 1 commit intospecify:mainfrom
calacademy-research:cas/perf-delete-blockers-7515

Conversation

@foozleface
Copy link
Copy Markdown
Collaborator

Fixes #7515
Contributed by @foozleface

The delete_blockers endpoint used Django's Collector.collect() which loads ALL related objects into memory before returning. For a Taxon like "Life" with 900K+ determinations, this causes out-of-memory kills. This PR replaces the Collector with targeted COUNT queries and limited ID lists per PROTECT relationship, keeping memory bounded regardless of how many blocking rows exist.

Implementation

  • Replace Collector.collect() in _collect_delete_blockers with iteration over obj._meta.related_objects, filtering for protect_with_blockers on_delete
  • For each PROTECT relationship, run a COUNT query and fetch at most id_limit (default 100) primary keys
  • Add total_count field to the response so the frontend knows the full count even when IDs are capped
  • Update frontend test fixtures to match the new isstrict: undefined serialization
  • Add comprehensive backend tests for the pagination behavior

Testing instructions

  • Navigate to a record with many dependent records (e.g., a high-level Taxon node) and attempt to delete it
  • Verify the delete blocker dialog shows the correct count and a reasonable number of linked record IDs
  • Confirm no OOM or timeout on records with very large numbers of blockers
  • Run the test suite: python manage.py test specifyweb.backend.delete_blockers.tests.test_pagination

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: 📋Back Log

Development

Successfully merging this pull request may close these issues.

[Large Databases]: Paginate the delete blockers

1 participant