Skip to content

feat: run bg jobs in execution context#1998

Merged
stalniy merged 2 commits intomainfrom
feat/bg-jobs-exec-context
Oct 3, 2025
Merged

feat: run bg jobs in execution context#1998
stalniy merged 2 commits intomainfrom
feat/bg-jobs-exec-context

Conversation

@stalniy
Copy link
Contributor

@stalniy stalniy commented Oct 3, 2025

Why

Due to the nature of nodejs and DI, we need to run bg jobs in execution context to ensure that it's safe to use application services (which may depend on executionContext) inside background jobs.

For now, use empty ability (to ensure nothing bad happens) and fake user (to ensure feature flags service will produce stable values during gradual rollout). Just to minimize the changes because the final/proper decision in this regard should be done via a discussion proposal with enforcement on CI level. See #1997

Summary by CodeRabbit

  • New Features

    • Background jobs now run inside an execution context that provides a system user and permission checks per job.
    • Job retry behavior updated: retry backoff enabled, max retry delay added (up to 5 minutes), and retry limit reduced for controlled retries.
    • Notification job name standardized for clearer naming and observability.
    • Per-job logging now occurs within the execution context.
  • Tests

    • Tests updated to cover context-aware job execution, logging, and adjusted queue retry configuration.

@stalniy stalniy requested a review from a team as a code owner October 3, 2025 03:14
@coderabbitai
Copy link
Contributor

coderabbitai bot commented Oct 3, 2025

Walkthrough

JobQueueService now requires ExecutionContextService, runs each job inside runWithContext seeding CURRENT_USER and ABILITY (via createMongoAbility), and updates queue retry options (retryLimit 5, retryBackoff enabled, retryDelayMax 5min). Tests updated for constructor change. Notification job name changed to "NotificationJob".

Changes

Cohort / File(s) Summary
Job queue service
apps/api/src/core/services/job-queue/job-queue.service.ts
Added ExecutionContextService import and constructor param; wrap job processing in executionContextService.runWithContext seeding CURRENT_USER and ABILITY (via createMongoAbility); moved JOB_STARTED log into context; preserve try/catch and log JOB_DONE/JOB_FAILED; set queue options retryLimit: 5, retryBackoff: true, retryDelayMax: 5 * 60 * 1000.
Tests — job queue
apps/api/src/core/services/job-queue/job-queue.service.spec.ts
Added mocks/executionContextService with set/runWithContext; updated test setup to pass mocks.executionContextService into JobQueueService before pgBoss; adjusted queue creation expectations to include retryBackoff, retryDelayMax, and updated retryLimit.
Notification job identifier
apps/api/src/notifications/services/notification-handler/notification.handler.ts
Renamed static job name constant from "notifications" to "NotificationJob" (affects observable JOB_NAME value).

Sequence Diagram(s)

sequenceDiagram
  autonumber
  actor PGBoss as PG Boss
  participant JQS as JobQueueService
  participant ECS as ExecutionContextService
  participant Handler as JobHandler
  participant Logger as Logger

  PGBoss->>JQS: deliver(job)
  Note right of JQS: prepare context (CURRENT_USER, ABILITY)
  rect rgba(200,230,255,0.18)
    JQS->>ECS: runWithContext(context, fn)
    activate ECS
    ECS->>Handler: handle(job.data)
    Handler-->>ECS: result / throws
    deactivate ECS
  end
  alt success
    JQS->>Logger: log(JOB_DONE, job.id)
  else failure
    JQS->>Logger: log(JOB_FAILED, job.id, error)
  end
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Poem

I hop through queues with context bright,
I plant a user, craft ABILITY on sight.
I run each task inside a gentle nest,
Retry a few times until it’s at its best.
Thump! A Rabbit cheers for jobs that rest. 🐇✨

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check ✅ Passed The title succinctly and accurately describes the primary change of the pull request, which is enabling background jobs to run within an execution context, and it uses concise, clear phrasing without unnecessary noise.
✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch feat/bg-jobs-exec-context

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 8a15284 and 41e185e.

📒 Files selected for processing (2)
  • apps/api/src/core/services/job-queue/job-queue.service.spec.ts (2 hunks)
  • apps/api/src/core/services/job-queue/job-queue.service.ts (3 hunks)
🧰 Additional context used
📓 Path-based instructions (3)
**/*.{ts,tsx}

📄 CodeRabbit inference engine (.cursor/rules/general.mdc)

Never use type any or cast to type any. Always define the proper TypeScript types.

Files:

  • apps/api/src/core/services/job-queue/job-queue.service.ts
  • apps/api/src/core/services/job-queue/job-queue.service.spec.ts
**/*.{js,ts,tsx}

📄 CodeRabbit inference engine (.cursor/rules/general.mdc)

**/*.{js,ts,tsx}: Never use deprecated methods from libraries.
Don't add unnecessary comments to the code

Files:

  • apps/api/src/core/services/job-queue/job-queue.service.ts
  • apps/api/src/core/services/job-queue/job-queue.service.spec.ts
**/*.spec.{ts,tsx}

📄 CodeRabbit inference engine (.cursor/rules/no-jest-mock.mdc)

Don't use jest.mock() to mock dependencies in test files. Instead, use jest-mock-extended to create mocks and pass mocks as dependencies to the service under test.

**/*.spec.{ts,tsx}: Use setup function instead of beforeEach in test files
setup function must be at the bottom of the root describe block in test files
setup function creates an object under test and returns it
setup function should accept a single parameter with inline type definition
Don't use shared state in setup function
Don't specify return type of setup function

Files:

  • apps/api/src/core/services/job-queue/job-queue.service.spec.ts
🧬 Code graph analysis (1)
apps/api/src/core/services/job-queue/job-queue.service.ts (1)
packages/logging/src/services/logger/logger.service.ts (1)
  • error (107-109)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
  • GitHub Check: validate / validate-app
  • GitHub Check: test-build

@codecov
Copy link

codecov bot commented Oct 3, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 45.52%. Comparing base (06b4e45) to head (7573320).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1998      +/-   ##
==========================================
- Coverage   45.88%   45.52%   -0.36%     
==========================================
  Files        1012     1002      -10     
  Lines       28554    28211     -343     
  Branches     7480     7439      -41     
==========================================
- Hits        13101    12844     -257     
  Misses      14248    14248              
+ Partials     1205     1119      -86     
Flag Coverage Δ *Carryforward flag
api 82.27% <100.00%> (+0.01%) ⬆️
deploy-web 23.84% <ø> (ø) Carriedforward from 06b4e45
log-collector ?
notifications 88.16% <ø> (ø) Carriedforward from 06b4e45
provider-console 81.48% <ø> (ø) Carriedforward from 06b4e45
provider-proxy 84.98% <ø> (ø) Carriedforward from 06b4e45

*This pull request uses carry forward flags. Click here to find out more.

Files with missing lines Coverage Δ
...i/src/core/services/job-queue/job-queue.service.ts 98.24% <100.00%> (+0.20%) ⬆️
...vices/notification-handler/notification.handler.ts 100.00% <100.00%> (ø)

... and 41 files with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@stalniy stalniy force-pushed the feat/bg-jobs-exec-context branch from 41e185e to 1cee9bc Compare October 3, 2025 03:45
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
apps/api/src/core/services/job-queue/job-queue.service.spec.ts (1)

18-21: Update test expectations for the new retry configuration

The production code now registers queues with { retryLimit: 5, retryBackoff: true, retryDelayMax: 5 * 60 }, but the tests still expect retryLimit: 10 and don’t assert the new flags (Lines 18-21, 49-55). As written, the spec will fail. Please align the expectations with the implementation.

Also applies to: 48-55

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 41e185e and 1cee9bc.

📒 Files selected for processing (3)
  • apps/api/src/core/services/job-queue/job-queue.service.spec.ts (2 hunks)
  • apps/api/src/core/services/job-queue/job-queue.service.ts (4 hunks)
  • apps/api/src/notifications/services/notification-handler/notification.handler.ts (1 hunks)
🧰 Additional context used
📓 Path-based instructions (3)
**/*.spec.{ts,tsx}

📄 CodeRabbit inference engine (.cursor/rules/no-jest-mock.mdc)

Don't use jest.mock() to mock dependencies in test files. Instead, use jest-mock-extended to create mocks and pass mocks as dependencies to the service under test.

**/*.spec.{ts,tsx}: Use setup function instead of beforeEach in test files
setup function must be at the bottom of the root describe block in test files
setup function creates an object under test and returns it
setup function should accept a single parameter with inline type definition
Don't use shared state in setup function
Don't specify return type of setup function

Files:

  • apps/api/src/core/services/job-queue/job-queue.service.spec.ts
**/*.{ts,tsx}

📄 CodeRabbit inference engine (.cursor/rules/general.mdc)

Never use type any or cast to type any. Always define the proper TypeScript types.

Files:

  • apps/api/src/core/services/job-queue/job-queue.service.spec.ts
  • apps/api/src/notifications/services/notification-handler/notification.handler.ts
  • apps/api/src/core/services/job-queue/job-queue.service.ts
**/*.{js,ts,tsx}

📄 CodeRabbit inference engine (.cursor/rules/general.mdc)

**/*.{js,ts,tsx}: Never use deprecated methods from libraries.
Don't add unnecessary comments to the code

Files:

  • apps/api/src/core/services/job-queue/job-queue.service.spec.ts
  • apps/api/src/notifications/services/notification-handler/notification.handler.ts
  • apps/api/src/core/services/job-queue/job-queue.service.ts
🧬 Code graph analysis (2)
apps/api/src/notifications/services/notification-handler/notification.handler.ts (2)
apps/api/src/core/services/job-queue/job-queue.service.ts (1)
  • JOB_NAME (188-188)
apps/api/src/core/services/domain-events/domain-events.service.ts (1)
  • JOB_NAME (6-6)
apps/api/src/core/services/job-queue/job-queue.service.ts (1)
packages/logging/src/services/logger/logger.service.ts (1)
  • error (107-109)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
  • GitHub Check: validate / validate-app
  • GitHub Check: test-build

baktun14
baktun14 previously approved these changes Oct 3, 2025
@stalniy stalniy force-pushed the feat/bg-jobs-exec-context branch from 53cf669 to 7573320 Compare October 3, 2025 12:54
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

♻️ Duplicate comments (1)
apps/api/src/core/services/job-queue/job-queue.service.ts (1)

107-126: Await context seeding to ensure initialization completes before handler execution.

The executionContextService.set() calls on lines 107 and 126 are not awaited. Since set() returns a promise (confirmed by the test mock at line 263 in the spec file), the handler may start before the context is fully initialized, and any rejection will be silently ignored.

Apply this diff to await both context-seeding calls:

-            this.executionContextService.set("CURRENT_USER", {
+            await this.executionContextService.set("CURRENT_USER", {
               id: "bg-job-user",
               bio: "",
               email: "bg-job-user@akash.network",
               emailVerified: false,
               stripeCustomerId: "",
               subscribedToNewsletter: false,
               createdAt: new Date(),
               lastActiveAt: new Date(),
               lastIp: null,
               lastUserAgent: null,
               lastFingerprint: null,
               youtubeUsername: null,
               twitterUsername: null,
               githubUsername: null,
               userId: "system:bg-job-user",
               username: "___bg_job_user___",
               trial: false
             });
-            this.executionContextService.set("ABILITY", createMongoAbility<MongoAbility>());
+            await this.executionContextService.set("ABILITY", createMongoAbility<MongoAbility>());
🧹 Nitpick comments (1)
apps/api/src/core/services/job-queue/job-queue.service.spec.ts (1)

262-265: Replicate context isolation in the mock
The real ExecutionContextService.runWithContext uses AsyncLocalStorage.run(new Map(), …) to isolate each invocation. The current mock just calls cb() directly, so tests won’t catch cross‐run state leakage. Either use the real service in your specs or update the mock to simulate per‐run storage (e.g. invoke storage.run(new Map(), …)).

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 53cf669 and 7573320.

📒 Files selected for processing (3)
  • apps/api/src/core/services/job-queue/job-queue.service.spec.ts (5 hunks)
  • apps/api/src/core/services/job-queue/job-queue.service.ts (4 hunks)
  • apps/api/src/notifications/services/notification-handler/notification.handler.ts (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • apps/api/src/notifications/services/notification-handler/notification.handler.ts
🧰 Additional context used
📓 Path-based instructions (3)
**/*.spec.{ts,tsx}

📄 CodeRabbit inference engine (.cursor/rules/no-jest-mock.mdc)

Don't use jest.mock() to mock dependencies in test files. Instead, use jest-mock-extended to create mocks and pass mocks as dependencies to the service under test.

**/*.spec.{ts,tsx}: Use setup function instead of beforeEach in test files
setup function must be at the bottom of the root describe block in test files
setup function creates an object under test and returns it
setup function should accept a single parameter with inline type definition
Don't use shared state in setup function
Don't specify return type of setup function

Files:

  • apps/api/src/core/services/job-queue/job-queue.service.spec.ts
**/*.{ts,tsx}

📄 CodeRabbit inference engine (.cursor/rules/general.mdc)

Never use type any or cast to type any. Always define the proper TypeScript types.

Files:

  • apps/api/src/core/services/job-queue/job-queue.service.spec.ts
  • apps/api/src/core/services/job-queue/job-queue.service.ts
**/*.{js,ts,tsx}

📄 CodeRabbit inference engine (.cursor/rules/general.mdc)

**/*.{js,ts,tsx}: Never use deprecated methods from libraries.
Don't add unnecessary comments to the code

Files:

  • apps/api/src/core/services/job-queue/job-queue.service.spec.ts
  • apps/api/src/core/services/job-queue/job-queue.service.ts
🧬 Code graph analysis (1)
apps/api/src/core/services/job-queue/job-queue.service.ts (1)
packages/logging/src/services/logger/logger.service.ts (1)
  • error (107-109)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
  • GitHub Check: validate / validate-app
  • GitHub Check: test-build
🔇 Additional comments (8)
apps/api/src/core/services/job-queue/job-queue.service.spec.ts (3)

6-6: LGTM!

The ExecutionContextService type import is correctly added to support the updated constructor signature.


20-22: LGTM!

The retry configuration expectations are correctly updated to reflect the production changes: retryBackoff: true, retryDelayMax: 300 (5 minutes in seconds), and retryLimit: 5.

Also applies to: 52-54, 58-60, 131-133


268-273: LGTM!

The constructor call correctly includes executionContextService as the third parameter, matching the updated production signature.

apps/api/src/core/services/job-queue/job-queue.service.ts (5)

1-1: LGTM!

CASL ability imports are correctly added to support the execution context seeding with permissions.


7-7: LGTM!

ExecutionContextService import is correctly added.


19-19: LGTM!

The ExecutionContextService dependency is correctly injected as a constructor parameter.


41-43: LGTM!

Retry configuration is appropriately tuned: retryLimit: 5 (reduced from 10), retryBackoff: true (exponential backoff enabled), and retryDelayMax: 5 * 60 (5-minute cap).


131-144: LGTM!

The error handling correctly logs job failures and re-throws to trigger PgBoss retry logic.

@stalniy stalniy merged commit 0a25bb5 into main Oct 3, 2025
62 checks passed
@stalniy stalniy deleted the feat/bg-jobs-exec-context branch October 3, 2025 13:32
stalniy added a commit that referenced this pull request Nov 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

Comments