[codex] add celery worker and beat to local compose

[jruszo]

What changed

• Added celery and celerybeat services to the local compose stacks.
• Switched the local environment defaults to TASK_BACKEND=celery with Redis broker/result settings.
• Updated the app startup script to skip Django Q when Celery is the active backend.

Why

• The local development stack needs a running Celery coordinator and one worker for task-backed flows.
• Django Q should not start alongside Celery when the local environment is configured for Celery.

Validation

• docker-compose -f src/docker-compose/docker-compose.local-arm.yml config
• docker-compose -f src/docker-compose/docker-compose.yml config
• Brought the local ARM stack up in Colima and confirmed datamingle-app, datamingle-celery, and datamingle-celerybeat were running.

Summary by CodeRabbit

Release Notes

• New Features

  • Added asynchronous task queue processing capability with dedicated worker and scheduler services.

• Chores

  • Extended deployment configuration with flexible task backend support and environment-based initialization.

[coderabbitai]

📝 Walkthrough

Walkthrough

This pull request adds Celery as an alternative task backend to the system. Environment variables configure Celery's broker, result storage, and task limits; the startup script conditionally launches Django Q based on the selected backend; and Docker Compose services are added to run Celery worker and beat scheduler containers.

Changes

Task Backend Introduction with Celery Support

Layer / File(s)	Summary
Configuration src/docker-compose/.env, src/docker-compose/.env.local-arm	Environment variables added for Celery: TASK_BACKEND=celery, CELERY_BROKER_URL, CELERY_RESULT_BACKEND, CELERY_TASK_DEFAULT_QUEUE=datamingle-default, CELERY_TASK_SOFT_TIME_LIMIT, and CELERY_TASK_TIME_LIMIT.
Startup Logic src/docker/startup.sh	Django Q cluster startup is now conditional: only runs supervisord when TASK_BACKEND=django_q; skips cluster startup for other backends.
Container Orchestration src/docker-compose/docker-compose.yml, src/docker-compose/docker-compose.local-arm.yml	Two new services added per compose file: celery service runs celery -A archery worker with queue configuration, and celerybeat service runs celery -A archery beat with schedule persistence, both with venv activation and proper dependency conditions on redis, mysql, and datamingle.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~15 minutes

Poem

🐰 Hop hop, the queues now bloom,
Celery workers fill the room,
Beat scheduler keeps the time,
Tasks flow smooth, all in rhyme!
Backend switched with graceful care,
Config and compose work fair! 🌿✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately reflects the main change: adding Celery worker and beat services to the local Docker Compose configuration.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/celery-compose-local

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Review rate limit: 0/1 reviews remaining, refill in 60 minutes.}

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@src/docker-compose/docker-compose.yml`:
- Around line 75-82: The compose setup lets celery and celerybeat depend on
datamingle with condition service_started which is too weak because datamingle
runs migrations at startup; add a proper readiness mechanism by either (A)
adding a healthcheck to the datamingle service that verifies migrations
completed and the app is responsive (e.g., hitting a /health or checking DB
schema) and change celery/celerybeat to depend_on datamingle: condition:
service_healthy, or (B) create a dedicated one-shot migration service (e.g.,
datamingle-migrate) that runs migrations and exits, make celery and celerybeat
depend_on that service with condition: service_completed_successfully, and
remove the service_started dependency; update service names datamingle, celery,
celerybeat, and datamingle-migrate accordingly.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: a2d1538e-0d69-4fe0-8e8e-49c065268165

📥 Commits

Reviewing files that changed from the base of the PR and between f68302b and 96e8d21.

📒 Files selected for processing (5)

• src/docker-compose/.env
• src/docker-compose/.env.local-arm
• src/docker-compose/docker-compose.local-arm.yml
• src/docker-compose/docker-compose.yml
• src/docker/startup.sh

[coderabbitai]

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Inspect migration flag and dependency conditions in compose:"
rg -n -C2 'RUN_MIGRATIONS_ON_START|depends_on:|condition: service_(started|healthy)|healthcheck:' src/docker-compose/docker-compose.yml

Repository: jruszo/Datamingle

Length of output: 1521

---

service_started is too weak for Celery startup ordering — workers can start before migrations complete.

celery and celerybeat wait only for datamingle to start (Lines 80–81, 104–105), not to finish migrations. Since datamingle runs migrations at startup (Line 66) but has no healthcheck, workers can begin executing tasks against an incomplete database schema, causing early boot failures.

Add a healthcheck to datamingle that waits for migrations and app readiness, or introduce a dedicated one-shot migration service that both celery and celerybeat depend on with condition: service_completed_successfully.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@src/docker-compose/docker-compose.yml` around lines 75 - 82, The compose
setup lets celery and celerybeat depend on datamingle with condition
service_started which is too weak because datamingle runs migrations at startup;
add a proper readiness mechanism by either (A) adding a healthcheck to the
datamingle service that verifies migrations completed and the app is responsive
(e.g., hitting a /health or checking DB schema) and change celery/celerybeat to
depend_on datamingle: condition: service_healthy, or (B) create a dedicated
one-shot migration service (e.g., datamingle-migrate) that runs migrations and
exits, make celery and celerybeat depend_on that service with condition:
service_completed_successfully, and remove the service_started dependency;
update service names datamingle, celery, celerybeat, and datamingle-migrate
accordingly.