Misc troubleshooting improvements

[galvana]

Ticket ENG-1860

Description Of Changes

Misc improvements to help troubleshoot and deal with stuck privacy requests.

Code Changes

• Added endpoint to bulk cancel requests PATCH /privacy-request/administrate/cancel
• Allowing pending (new) requests to be resubmitted
• Added a memory dump output to the logs when the memory watchdog hits the 90% memory limit
• Allowing dynamic Celery environment variables to be loaded if they match the FIDES__CELERY__ prefix

Pre-Merge Checklist

• Issue requirements met
• All CI pipelines succeeded
• CHANGELOG.md updated
  • Add a db-migration This indicates that a change includes a database migration label to the entry if your change includes a DB migration
  • Add a high-risk This issue suggests changes that have a high-probability of breaking existing code label to the entry if your change includes a high-risk change (i.e. potential for performance impact or unexpected regression) that should be flagged
  • Updates unreleased work already in Changelog, no new entry necessary
• UX feedback:
  • All UX related changes have been reviewed by a designer
  • No UX review needed
• Followup issues:
  • Followup issues created
  • No followup issues
• Database migrations:
  • Ensure that your downrev is up to date with the latest revision on main
  • Ensure that your downgrade() migration is correct and works
    • If a downgrade migration is not possible for this change, please call this out in the PR description!
  • No migrations
• Documentation:
  • Documentation complete, PR opened in fidesdocs
  • Documentation issue created in fidesdocs
  • If there are any new client scopes created as part of the pull request, remember to update public-facing documentation that references our scope registry
  • No documentation updates required

[vercel]

Deployment failed with the following error:

You must set up Two-Factor Authentication before accessing this team.

View Documentation: https://vercel.com/docs/two-factor-authentication

[greptile-apps]

Greptile Overview

Greptile Summary

This PR adds troubleshooting improvements for stuck privacy requests and memory management:

Key Changes:

• Added new /privacy-request/administrate/cancel endpoint to allow bulk cancellation of privacy requests in non-terminal states
• Relaxed resubmission restrictions to allow pending status requests to be resubmitted (previously only non-complete/non-pending could be resubmitted)
• Enhanced memory watchdog with heap dump capture functionality using objgraph to help diagnose memory leaks when thresholds are exceeded
• Added support for arbitrary Celery configuration via FIDES__CELERY__* environment variables beyond the explicitly defined fields

How it works:
The cancel endpoint follows the same pattern as approve/deny endpoints, using bulk operations with proper validation. The heap dump feature captures object type counts, GC stats, and uncollectable objects when memory limits are exceeded, logging them as a single comprehensive report. The Celery env var merging uses JSON parsing to handle type conversion for booleans, integers, and other JSON-compatible values.

Confidence Score: 4/5

• This PR is safe to merge with minor attention to the Celery config parsing logic
• The PR adds valuable troubleshooting features with comprehensive test coverage. One logical issue was found in celery_settings.py:56 where lowercasing values before JSON parsing could break case-sensitive string values (e.g., URLs with uppercase). The exception handling catches this, but it's not the intended behavior. All other changes are well-tested and follow existing patterns.
• Pay close attention to src/fides/config/celery_settings.py - the JSON parsing logic should not lowercase values before parsing

Important Files Changed

File Analysis

Filename	Score	Overview
src/fides/api/api/v1/endpoints/privacy_request_endpoints.py	5/5	Added cancel endpoint and relaxed resubmit restrictions for pending requests
src/fides/api/util/memory_watchdog.py	5/5	Added comprehensive heap dump functionality for memory leak diagnosis
src/fides/config/celery_settings.py	3/5	Added merge function for arbitrary Celery env vars, but has case-sensitivity issue with JSON parsing
src/fides/service/privacy_request/privacy_request_service.py	5/5	Added cancel_privacy_requests method and relaxed resubmit restrictions

_{12 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

logic: lowercasing value before JSON parsing breaks case-sensitive string values

For example: FIDES__CELERY__BROKER_URL=Redis://myhost would become redis://myhost after lowercasing, which might cause unexpected behavior. Consider only lowercasing for known boolean strings:

Suggested change

	celery_dict[stripped_key] = json.loads(value.lower())
	# Try parsing as-is first for case-sensitive values
	celery_dict[stripped_key] = json.loads(value)

[erosselli]

not sure how correct this is, @galvana can you double-check?

[galvana]

Yup, this is valid, I fixed it

[codecov]

Codecov Report

❌ Patch coverage is 81.48148% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.27%. Comparing base (eb3f579) to head (f0f794c).
⚠️ Report is 8 commits behind head on main.

Files with missing lines	Patch %	Lines
src/fides/api/util/memory_watchdog.py	66.66%	10 Missing and 2 partials ⚠️
...service/privacy_request/privacy_request_service.py	90.00%	2 Missing ⚠️
src/fides/config/celery_settings.py	94.11%	0 Missing and 1 partial ⚠️

❌ Your patch status has failed because the patch coverage (81.48%) is below the target coverage (100.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6916      +/-   ##
==========================================
- Coverage   87.29%   87.27%   -0.02%     
==========================================
  Files         523      523              
  Lines       34172    34251      +79     
  Branches     3932     3943      +11     
==========================================
+ Hits        29829    29892      +63     
- Misses       3487     3499      +12     
- Partials      856      860       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[erosselli]

Let's add a changelog entry. Approving with some questions/comments

[erosselli]

not sure how correct this is, @galvana can you double-check?