QueueCTL – A CLI-Based Background Job Queue System

queuectl is a lightweight, production-style background job processing system built using Node.js.
It provides a CLI to enqueue jobs, start workers, manage retries with exponential backoff, and handle permanently failed jobs through a Dead Letter Queue (DLQ).

This project is designed as part of a Backend Developer Internship Assignment and implements all required features with clean architecture, persistence, and test coverage.

Project demonstration

video walkthrough:

🔗 Access the Google Drive Folder

Features

• Enqueue background jobs from CLI
• Parallel worker processes
• Exponential retry with base^attempts
• Dead Letter Queue (DLQ)
• Persistent storage using SQLite
• Graceful worker shutdown
• Prevents duplicate job execution
• Configurable system
• Job logging (stdout & stderr)
• Full job lifecycle tracking
• Windows-friendly
• End-to-end test script included

Tech Stack

• Node.js (ESM modules)
• SQLite (better-sqlite3)
• Commander.js
• Child processes
• UUID

Project Structure

queuectl/
│
├── bin/
│   └── queuectl.js          # CLI entry point
│
├── src/
│   ├── cli/                 # CLI command handlers
│   ├── core/                # Database, repos, utils
│   ├── manager/             # Worker start/stop manager
│   └── worker/              # Worker process logic
│
├── scripts/
│   ├── test-e2e.js          # End-to-end test (Node version)
│   └── demo.sh              # Demo script
│
├── .queuectl/               # Runtime data: DB + logs
├── package.json
└── README.md

Job Lifecycle

PENDING → PROCESSING → (COMPLETED or FAILED → RETRY → DEAD)

PENDING
   ↓ worker claims
PROCESSING
   ├── success → COMPLETED
   └── failure → FAILED → RETRY (backoff) → DEAD (DLQ)

---

---

Getting Started

1. Clone the repository

git clone https://github.com/krishnactive/CLI.git
cd CLI

2. Install dependencies

npm install

3. Run database migration

node src/core/db.js --migrate

CLI Usage (Windows Git Bash – Option 1)

Enqueue a job

node bin/queuectl.js enqueue --cmd "echo 'Hello World'"

Start workers

node bin/queuectl.js worker start --count 3

Stop all workers

node bin/queuectl.js worker stop

System status

node bin/queuectl.js status

List jobs

node bin/queuectl.js list --state pending
node bin/queuectl.js list --state completed
node bin/queuectl.js list --state failed

Dead Letter Queue

node bin/queuectl.js dlq list
node bin/queuectl.js dlq retry <JOB_ID>

Configuration

node bin/queuectl.js config set max_retries 3
node bin/queuectl.js config get

---

Testing

npm run test:e2e

---

Architecture Overview

CLI Layer

Parses commands and interacts with manager & DB.

Worker Manager

Forks worker processes and tracks them in SQLite.

Worker Processes

Pick jobs, execute commands, log output, retry or complete jobs.

Persistence

SQLite with WAL.

Retry + Backoff

delay = backoff_base ^ attempts

---